• Re: 504 gateway time-outs (was: FYI: The elapsed time shown for beefy24 [main-amd64] and beefy23 [150amd64-default] stopped over 2hrs ago)

    From Kevin Oberman@rkoberman@gmail.com to muc.lists.freebsd.stable on Sat Feb 28 17:57:43 2026
    From Newsgroup: muc.lists.freebsd.stable

    --000000000000c08eda064beccca4
    Content-Type: text/plain; charset="UTF-8"
    Content-Transfer-Encoding: quoted-printable

    On Sat, Feb 28, 2026 at 5:48=E2=80=AFPM Kevin Oberman <rkoberman@gmail.com>=
    wrote:

    On Sat, Feb 28, 2026 at 3:33=E2=80=AFAM Graham Perrin <grahamperrin@gmail=
    .com>
    wrote:

    On 28/02/2026 10:32, Mark Millard wrote:
    =E2=80=A6

    The following got "504 Gateway Time-out" when I tried them:

    <
    https://pkg-status.freebsd.org/beefy24/build.html?mastername=3Dmain-amd6= 4-default&build=3Dpdf4f957ea181_s178d0b5b8d


    <
    https://pkg-status.freebsd.org/beefy23/build.html?mastername=3D150amd64-= default&build=3Ddf4f957ea181



    Both somewhat slow to load, however they do load for me.

    Thanks


    beefy22 is showing hte same issues. Something is tying up the builders
    with builds hanging in "build-depends" and "lib-depends" for very long periods of time, as long as over 4 hours) with high load averages. This h=
    as
    been going on for months and seems to be getting worse. Chromium builds
    which used to take around 20 hours are now running around 40 and I really don't think that this is just code bloat. At times, about half of the
    active builds are in some state other than "build"; mostly one of the
    depends states which should never take long as all dependencies are built before a build is started.

    After some time passes (often hours) things clear up. Dependencies are suddenly loaded in a few minutes or less. Note that things do continue to complete, but the 10 minute averages are in single digits and frequently =
    go
    to 0. During this time, my connection to beefy22 will timeout and sometim=
    es
    I can't establish a connection.

    I don't have any special access to the machines... just via pkg-status, s=
    o
    I'm fairly limited in any analysis.


    Just before I started my last message, all was well, but I last saw were a dozen builds in one of the "depends" states and "impulse" at 74 when I lost
    my connection. I can reconnect to the beefy22 status page, but it has not updated for several minutes.
    --=20
    Kevin Oberman, Part time kid herder and retired Network Engineer
    E-mail: rkoberman@gmail.com
    PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683

    --000000000000c08eda064beccca4
    Content-Type: text/html; charset="UTF-8"
    Content-Transfer-Encoding: quoted-printable

    <div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-family:tahoma,sans-serif;font-size:small">On Sat, Feb 28, 2026 at 5:48=E2= =80=AFPM Kevin Oberman &lt;<a href=3D"mailto:rkoberman@gmail.com">rkoberman= @gmail.com</a>&gt; wrote:</div></div><div class=3D"gmail_quote gmail_quote_= container"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.= 8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"lt= r"><div dir=3D"ltr"><div style=3D"font-family:tahoma,sans-serif;font-size:s= mall">On Sat, Feb 28, 2026 at 3:33=E2=80=AFAM Graham Perrin &lt;<a href=3D"= mailto:grahamperrin@gmail.com" target=3D"_blank">grahamperrin@gmail.com</a>= &gt; wrote:</div></div><div class=3D"gmail_quote"><blockquote class=3D"gmai= l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20= 4,204);padding-left:1ex">On 28/02/2026 10:32, Mark Millard wrote:<br>
    &gt; =E2=80=A6<br>
    &gt;<br>
    &gt; The following got &quot;504 Gateway Time-out&quot; when I tried them:<=

    &gt;<br>
    &gt; &lt;<a href=3D"https://pkg-status.freebsd.org/beefy24/build.html?maste= rname=3Dmain-amd64-default&amp;build=3Dpdf4f957ea181_s178d0b5b8d" rel=3D"no= referrer" target=3D"_blank">https://pkg-status.freebsd.org/beefy24/build.ht= ml?mastername=3Dmain-amd64-default&amp;build=3Dpdf4f957ea181_s178d0b5b8d</a= >&gt;<br>
    &gt;<br>
    &gt; &lt;<a href=3D"https://pkg-status.freebsd.org/beefy23/build.html?maste= rname=3D150amd64-default&amp;build=3Ddf4f957ea181" rel=3D"noreferrer" targe= t=3D"_blank">https://pkg-status.freebsd.org/beefy23/build.html?mastername= =3D150amd64-default&amp;build=3Ddf4f957ea181</a>&gt;<br>


    Both somewhat slow to load, however they do load for me.<br>

    Thanks</blockquote><div>=C2=A0</div></div><div><div style=3D"font-family:ta= homa,sans-serif;font-size:small">beefy22 is showing hte same issues. Someth= ing is tying up the builders with builds hanging in &quot;build-depends&quo=
    t; and &quot;lib-depends&quot; for very long periods of time, as long as ov=
    er 4 hours) with high load averages. This has been going on for months and = seems to be getting worse. Chromium builds which used to take around 20 hou=
    rs are now running around 40 and I really don&#39;t think=C2=A0that this is=
    just code bloat. At times, about half of the active builds=C2=A0are in som=
    e state=C2=A0other than &quot;build&quot;; mostly one of the depends states=
    which should never take long as all dependencies are built before a build =
    is started.</div><div style=3D"font-family:tahoma,sans-serif;font-size:smal= l"><br></div><div style=3D"font-family:tahoma,sans-serif;font-size:small">A= fter some time passes (often hours) things clear up. Dependencies are sudde= nly loaded in a few minutes or less. Note that things do continue to comple= te, but the 10 minute averages are in single digits and frequently go to 0.=
    During this time, my connection to beefy22 will timeout and sometimes I ca= n&#39;t establish a connection.</div><div style=3D"font-family:tahoma,sans-= serif;font-size:small"><br></div><div style=3D"font-family:tahoma,sans-seri= f;font-size:small">I don&#39;t have any special access to the machines... j= ust via pkg-status, so I&#39;m fairly limited in any analysis.</div></div><= /div>
    </blockquote></div><div><br></div><div><div style=3D"font-family:tahoma,san= s-serif;font-size:small" class=3D"gmail_default">Just before I started my l= ast message,=C2=A0 all was well, but I last=C2=A0saw were a dozen builds in=
    one of the=C2=A0&quot;depends&quot; states and &quot;impulse&quot; at 74 w= hen I lost my connection. I can reconnect to the beefy22 status page, but i=
    t has not updated for several=C2=A0minutes.</div></div><span class=3D"gmail= _signature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"= ><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir= =3D"ltr">Kevin Oberman, Part time kid herder and retired Network Engineer<b= r>E-mail: <a href=3D"mailto:rkoberman@gmail.com" target=3D"_blank">rkoberma= n@gmail.com</a><br></div><div>PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB= 39EF1B055683</div></div></div></div></div></div></div></div></div>

    --000000000000c08eda064beccca4--


    --
    Posted automagically by a mail2news gateway at muc.de e.V.
    Please direct questions, flames, donations, etc. to news-admin@muc.de
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From Philip Paeps@philip@freebsd.org to muc.lists.freebsd.stable on Tue Mar 31 21:25:44 2026
    From Newsgroup: muc.lists.freebsd.stable

    On 2026-02-28 19:32:35 (+0800), Graham Perrin wrote:
    On 28/02/2026 10:32, Mark Millard wrote:
    The following got "504 Gateway Time-out" when I tried them:

    <https://pkg-status.freebsd.org/beefy24/build.html?mastername=main-amd64-default&build=pdf4f957ea181_s178d0b5b8d>

    <https://pkg-status.freebsd.org/beefy23/build.html?mastername=150amd64-default&build=df4f957ea181>

    Both somewhat slow to load, however they do load for me.
    I only just noticed this thread, sorry for resurrecting it.
    I've noticed that beefy23 and beefy24 (2x EPYC 9254, 512G RAM) sometimes get too busy building to schedule nginx (or sshd). They eventually manage to plough through. Usually.
    That causes the 504 timeouts if you're going through pkg-status.f.o. If you're going directly to beefyX.chi.freebsd.org you'll just get a timeout.
    They're running exactly the same poudriere.conf as the other builders. I wonder if our calc_builders() function that tries to assign about 12G per builder isn't quite right for this particular configuration of cores and RAM.
    I haven't had a chance to look closely. As far as I can tell the builds do eventually succeed. If the only problem is "I can't obsessively poll pkg-status in real time", it's not a very high priority. :)
    This is on my list. It's just a VERY long list. :)
    Philip
    --
    Posted automagically by a mail2news gateway at muc.de e.V.
    Please direct questions, flames, donations, etc. to news-admin@muc.de
    --- Synchronet 3.21f-Linux NewsLink 1.2