From Newsgroup: muc.lists.freebsd.stable
--000000000000bfce64064c194f08
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Sat, Feb 28, 2026 at 8:55=E2=80=AFPM Mark Millard <
marklmi@yahoo.com> wr= ote:
On 2/28/26 17:57, Kevin Oberman wrote:
On Sat, Feb 28, 2026 at 5:48=E2=80=AFPM Kevin Oberman <rkoberman@gmail.=
com
<mailto:rkoberman@gmail.com>> wrote:
On Sat, Feb 28, 2026 at 3:33=E2=80=AFAM Graham Perrin
<grahamperrin@gmail.com <mailto:grahamperrin@gmail.com>> wrote:
On 28/02/2026 10:32, Mark Millard wrote:
> =E2=80=A6
>
> The following got "504 Gateway Time-out" when I tried them:
>
> <https://pkg-status.freebsd.org/beefy24/build.html?
mastername=3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b=
8d
<https://pkg-status.freebsd.org/beefy24/build.html?
mastername=3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b=
>
> <https://pkg-status.freebsd.org/beefy23/build.html?
mastername=3D150amd64-default&build=3Ddf4f957ea181 <https://pkg=
-
status.freebsd.org/beefy23/build.html?mastername=3D150amd64-
default&build=3Ddf4f957ea181>>
Both somewhat slow to load, however they do load for me.
Thanks
beefy22 is showing hte same issues. Something is tying up the
builders with builds hanging in "build-depends" and "lib-depends"
for very long periods of time, as long as over 4 hours) with high
load averages. This has been going on for months and seems to be
getting worse. Chromium builds which used to take around 20 hours
are now running around 40 and I really don't think that this is jus=
t
code bloat. At times, about half of the active builds are in some
state other than "build"; mostly one of the depends states which
should never take long as all dependencies are built before a build
is started.
After some time passes (often hours) things clear up. Dependencies
are suddenly loaded in a few minutes or less. Note that things do
continue to complete, but the 10 minute averages are in single
digits and frequently go to 0. During this time, my connection to
beefy22 will timeout and sometimes I can't establish a connection.
I don't have any special access to the machines... just via pkg-
status, so I'm fairly limited in any analysis.
Just before I started my last message, all was well, but I last saw
were a dozen builds in one of the "depends" states and "impulse" at 74
when I lost my connection. I can reconnect to the beefy22 status page,
but it has not updated for several minutes.
[Do not expect that I edited my all notes in presentation-text-order.]
Via <https://pkg-status.freebsd.org/builds?type=3Dpackage&all=3D1> :
beefy22: 143amd64-default 17:21:38
Accurate would be over 27 hrs (see below).
Note: beefy22 has 64 FreeBSD cpus. I do not know the RAM or SWAP space.
<
https://pkg-status.freebsd.org/beefy22/build.html?mastername=3D143amd64-d=
efault&build=3Ddf4f957ea181
:
Load Averages Swapinfo Elapsed
(100%) 63.94 69.19 70.56 0.24% 27:37:33
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com <mailto:rkoberman@gmail.com>
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
I see such on beefy24 (main-amd64-default) currently but also got lucky
for an detailed page display. Notably:
<
https://pkg-status.freebsd.org/beefy24/build.html?mastername=3Dmain-amd64=
-default&build=3Dpdf4f957ea181_s178d0b5b8d
shows:
Load Averages Swapinfo Elapsed
(107%) 102.71 106.93 107.91 55.15% 26:35:05
but <https://pkg-status.freebsd.org/builds?type=3Dpackage&all=3D1> shows 23:10:02 for Elapsed.
(Note: There are 96 FreeBSD cpus in beefy24 and in beefy23. The
poudriere configuration is allowing 45 builders at once. Each builder is likely to be configured to allow, say, up to 3 make processes, MAKE_JOBS_NUMBER=3D3 . It is difficult to look at logs now to check on th=
e
figure. I've no information about the amount of RAM or the size of the
SWAP space for beefy24 or beefy23.)
The longest-still-active port-package builders elapsed-time-so-far
figures shown were:
qt6-webengine-6.10.2 23:33:08
chromium-145.0.7632.109 20:43:03
electron39-39.7.0 20:41:59
apache-openoffice-devel-4.2.1768900765_1,4 11:20:05
Those are likely the major users of tmpfs space that competes for
RAM+SWAP for USE_TMPFS=3Dall (or other large USE_TMPFS settings).
I'd guess that the paging is thrashing for the above but have no way of checking. It could be that some use of TMPFS_BLACKLIST to avoid the use
of TMPFS for the port-packages that have huge TMPFS involved could help (without otherwise disabling USE_TMPFS=3Dall to still speed up most builders).
(I will note that TMPFS_BLACKLIST entries do not necessarily end up with
no tmpfs use, just less. But that less can help avoid paging that ends
up thrashing.)
Note during my editing/research and other activities, the web page had
not updated at all after its initial display. It is now significantly
later than when I started editing this note.
Later: Hmm.
https://pkg-status.freebsd.org/builds?type=3Dpackage&all=3D1
05:48:27 beefy23 (150amd64-default)
but . . .
<
https://pkg-status.freebsd.org/beefy23/build.html?mastername=3D150amd64-d=
efault&build=3Ddf4f957ea181
Load Averages Swapinfo Elapsed
(112%) 107.81 100.28 97.16 0.04% 27:42:2
--
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
beefy22 started a full ports build Saturday at 00:10 UTC. Most of last
Saturday and early Sunday had 'impulse' of no more than double digits and
over a 20 hour period the number of completed builds was just over 1000. Chromium and electron were killed when they reached 48 hours. The build now exceeds 72 hours. Last month's fill build was about 64.5 hours.
It really looks like things are getting worse. I suspect some resource is reaching exhaustion. One suggestion I saw that looks possible is tmp which
I believe is a tempfs, but I really can't tell too much with what is
visible on pkg-status.
On to wild speculation. Maybe reduce the number of concurrent builds. If a small amount of DRAM could help, that would be great, but DRAM is getting
very expensive and reducing the number of builds might speed things up at
no cost.
--=20
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail:
rkoberman@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
--000000000000bfce64064c194f08
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-family:tahoma,sans-serif;font-size:small">On Sat, Feb 28, 2026 at 8:55=E2= =80=AFPM Mark Millard <<a href=3D"mailto:
marklmi@yahoo.com">marklmi@yaho= o.com</a>> wrote:</div></div><div class=3D"gmail_quote gmail_quote_conta= iner"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b= order-left:1px solid rgb(204,204,204);padding-left:1ex">On 2/28/26 17:57, K= evin Oberman wrote:<br>
> On Sat, Feb 28, 2026 at 5:48=E2=80=AFPM Kevin Oberman <<a href=3D"m= ailto:
rkoberman@gmail.com" target=3D"_blank">
rkoberman@gmail.com</a><br>
> <mailto:<a href=3D"mailto:
rkoberman@gmail.com" target=3D"_blank">rk=
oberman@gmail.com</a>>> wrote:<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0On Sat, Feb 28, 2026 at 3:33=E2=80=AFAM Graham Perr= in<br>
>=C2=A0 =C2=A0 =C2=A0<<a href=3D"mailto:
grahamperrin@gmail.com" targe= t=3D"_blank">
grahamperrin@gmail.com</a> <mailto:<a href=3D"mailto:graham=
perrin@gmail.com" target=3D"_blank">
grahamperrin@gmail.com</a>>> wrot= e:<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0On 28/02/2026 10:32, Mark Millard wro= te:<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> =E2=80=A6<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> The following got "504 Gate= way Time-out" when I tried them:<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> <<a href=3D"
https://pkg-statu= s.freebsd.org/beefy24/build.html" rel=3D"noreferrer" target=3D"_blank">http= s://pkg-status.freebsd.org/beefy24/build.html</a>?<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mastername=3Dmain-amd64-default&b= uild=3Dpdf4f957ea181_s178d0b5b8d<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<<a href=3D"
https://pkg-status.fre= ebsd.org/beefy24/build.html" rel=3D"noreferrer" target=3D"_blank">
https://p= kg-status.freebsd.org/beefy24/build.html</a>?<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mastername=3Dmain-amd64-default&b= uild=3Dpdf4f957ea181_s178d0b5b8d>><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> <<a href=3D"
https://pkg-statu= s.freebsd.org/beefy23/build.html" rel=3D"noreferrer" target=3D"_blank">http= s://pkg-status.freebsd.org/beefy23/build.html</a>?<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mastername=3D150amd64-default&bui= ld=3Ddf4f957ea181 <<a href=3D"
https://pkg-" rel=3D"noreferrer" target=3D= "_blank">
https://pkg-</a><br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"
http://status.freebsd.org/= beefy23/build.html?mastername=3D150amd64-" rel=3D"noreferrer" target=3D"_bl= ank">status.freebsd.org/beefy23/build.html?mastername=3D150amd64-</a><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0default&build=3Ddf4f957ea181>&= gt;<br>
> <br>
> <br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Both somewhat slow to load, however t= hey do load for me.<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Thanks<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0=C2=A0<br>
>=C2=A0 =C2=A0 =C2=A0beefy22 is showing hte same issues. Something is ty= ing up the<br>
>=C2=A0 =C2=A0 =C2=A0builders with builds hanging in "build-depends= " and "lib-depends"<br>
>=C2=A0 =C2=A0 =C2=A0for very long periods of time, as long as over 4 ho= urs) with high<br>
>=C2=A0 =C2=A0 =C2=A0load averages. This has been going on for months an=
d seems to be<br>
>=C2=A0 =C2=A0 =C2=A0getting worse. Chromium builds which used to take a= round 20 hours<br>
>=C2=A0 =C2=A0 =C2=A0are now running around 40 and I really don't th= ink=C2=A0that this is just<br>
>=C2=A0 =C2=A0 =C2=A0code bloat. At times, about half of the active buil= ds=C2=A0are in some<br>
>=C2=A0 =C2=A0 =C2=A0state=C2=A0other than "build"; mostly one=
of the depends states which<br>
>=C2=A0 =C2=A0 =C2=A0should never take long as all dependencies are buil=
t before a build<br>
>=C2=A0 =C2=A0 =C2=A0is started.<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0After some time passes (often hours) things clear u=
p. Dependencies<br>
>=C2=A0 =C2=A0 =C2=A0are suddenly loaded in a few minutes or less. Note = that things do<br>
>=C2=A0 =C2=A0 =C2=A0continue to complete, but the 10 minute averages ar=
e in single<br>
>=C2=A0 =C2=A0 =C2=A0digits and frequently go to 0. During this time, my=
connection to<br>
>=C2=A0 =C2=A0 =C2=A0beefy22 will timeout and sometimes I can't esta= blish a connection.<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0I don't have any special access to the machines= ... just via pkg-<br>
>=C2=A0 =C2=A0 =C2=A0status, so I'm fairly limited in any analysis.<=
> <br>
> <br>
> Just before I started my last message,=C2=A0 all was well, but I last= =C2=A0saw<br>
> were a dozen builds in one of the=C2=A0"depends" states and = "impulse" at 74<br>
> when I lost my connection. I can reconnect to the beefy22 status page,=
> but it has not updated for several=C2=A0minutes.<br>
[Do not expect that I edited my all notes in presentation-text-order.]<br>
Via <<a href=3D"
https://pkg-status.freebsd.org/builds?type=3Dpackage&= ;all=3D1" rel=3D"noreferrer" target=3D"_blank">
https://pkg-status.freebsd.o= rg/builds?type=3Dpackage&all=3D1</a>> :<br>
beefy22: 143amd64-default 17:21:38<br>
Accurate would be over 27 hrs (see below).<br>
Note: beefy22 has 64 FreeBSD cpus. I do not know the RAM or SWAP space.<br>
<<a href=3D"
https://pkg-status.freebsd.org/beefy22/build.html?mastername= =3D143amd64-default&build=3Ddf4f957ea181" rel=3D"noreferrer" target=3D"= _blank">
https://pkg-status.freebsd.org/beefy22/build.html?mastername=3D143a= md64-default&build=3Ddf4f957ea181</a>><br>
:<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0Load Averages=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0S= wapinfo=C2=A0 Elapsed<br>
(100%) 63.94 69.19 70.56=C2=A0 =C2=A0 =C2=A00.24%=C2=A0 =C2=A0 =C2=A027:37:=
33 <br>
> -- <br>
> Kevin Oberman, Part time kid herder and retired Network Engineer<br>
> E-mail: <a href=3D"mailto:
rkoberman@gmail.com" target=3D"_blank">rkobe=
rman@gmail.com</a> <mailto:<a href=3D"mailto:
rkoberman@gmail.com" target= =3D"_blank">
rkoberman@gmail.com</a>><br>
> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683<br>
I see such on beefy24 (main-amd64-default) currently but also got lucky<br>
for an detailed page display. Notably:<br>
<<a href=3D"
https://pkg-status.freebsd.org/beefy24/build.html?mastername= =3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b8d" rel=3D"norefer= rer" target=3D"_blank">
https://pkg-status.freebsd.org/beefy24/build.html?ma= stername=3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b8d</a>>=
shows:<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0Load Averages=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0S= wapinfo=C2=A0 Elapsed<br>
(107%) 102.71 106.93 107.91=C2=A0 55.15%=C2=A0 =C2=A0 26:35:05<br>
but <<a href=3D"
https://pkg-status.freebsd.org/builds?type=3Dpackage&= ;all=3D1" rel=3D"noreferrer" target=3D"_blank">
https://pkg-status.freebsd.o= rg/builds?type=3Dpackage&all=3D1</a>> shows<br>
23:10:02 for Elapsed.<br>
(Note: There are 96 FreeBSD cpus in beefy24 and in beefy23. The<br>
poudriere configuration is allowing 45 builders at once. Each builder is<br=
likely to be configured to allow, say, up to 3 make processes,<br> MAKE_JOBS_NUMBER=3D3 . It is difficult to look at logs now to check on the<=
figure. I've no information about the amount of RAM or the size of the<=
SWAP space for beefy24 or beefy23.)<br>
The longest-still-active port-package builders elapsed-time-so-far<br>
figures shown were:<br>
qt6-webengine-6.10.2=C2=A0 =C2=A0 23:33:08<br>
chromium-145.0.7632.109 20:43:03<br>
electron39-39.7.0=C2=A0 =C2=A0 =C2=A0 =C2=A020:41:59<br> apache-openoffice-devel-4.2.1768900765_1,4 11:20:05<br>
Those are likely the major users of tmpfs space that competes for<br>
RAM+SWAP for USE_TMPFS=3Dall (or other large USE_TMPFS settings).<br>
I'd guess that the paging is thrashing for the above but have no way of=
checking. It could be that some use of TMPFS_BLACKLIST to avoid the use<br>
of TMPFS for the port-packages that have huge TMPFS involved could help<br> (without otherwise disabling USE_TMPFS=3Dall to still speed up most builder= s).<br>
(I will note that TMPFS_BLACKLIST entries do not necessarily end up with<br=
no tmpfs use, just less. But that less can help avoid paging that ends<br>
up thrashing.)<br>
Note during my editing/research and other activities, the web page had<br>
not updated at all after its initial display. It is now significantly<br>
later than when I started editing this note.<br>
Later: Hmm.<br>
<a href=3D"
https://pkg-status.freebsd.org/builds?type=3Dpackage&all=3D1=
" rel=3D"noreferrer" target=3D"_blank">
https://pkg-status.freebsd.org/build= s?type=3Dpackage&all=3D1</a><br>
05:48:27 beefy23 (150amd64-default)<br>
but . . .<br>
<<a href=3D"
https://pkg-status.freebsd.org/beefy23/build.html?mastername= =3D150amd64-default&build=3Ddf4f957ea181" rel=3D"noreferrer" target=3D"= _blank">
https://pkg-status.freebsd.org/beefy23/build.html?mastername=3D150a= md64-default&build=3Ddf4f957ea181</a>><br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0Load Averages=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0S= wapinfo=C2=A0 Elapsed<br>
(112%) 107.81 100.28 97.16=C2=A0 =C2=A00.04%=C2=A0 =C2=A0 =C2=A027:42:2<br>
-- <br>
=3D=3D=3D<br>
Mark Millard<br>
marklmi at <a href=3D"
http://yahoo.com" rel=3D"noreferrer" target=3D"_blank= ">yahoo.com</a><br>
</blockquote></div><div><br clear=3D"all"></div><div style=3D"font-family:t= ahoma,sans-serif;font-size:small" class=3D"gmail_default">beefy22 started a=
full ports build Saturday at 00:10 UTC. Most of last Saturday and early Su= nday had 'impulse' of no more than double digits and=C2=A0</div><di=
v style=3D"font-family:tahoma,sans-serif;font-size:small" class=3D"gmail_de= fault">over a 20 hour period the number of completed builds was just over 1= 000. Chromium and electron=C2=A0were killed when they reached 48 hours. The=
build now exceeds=C2=A072 hours. Last month's fill build was about 64.=
5 hours.</div><div style=3D"font-family:tahoma,sans-serif;font-size:small" = class=3D"gmail_default"><br></div><div style=3D"font-family:tahoma,sans-ser= if;font-size:small" class=3D"gmail_default">It really looks like things are=
getting worse. I suspect some resource is reaching exhaustion. One suggest= ion I saw that looks possible is tmp which I believe is a tempfs, but I rea= lly can't tell too much with what is visible on pkg-status.=C2=A0</div>= <div style=3D"font-family:tahoma,sans-serif;font-size:small" class=3D"gmail= _default"><br></div><div style=3D"font-family:tahoma,sans-serif;font-size:s= mall" class=3D"gmail_default">On to wild speculation. Maybe reduce the numb=
er of concurrent builds. If a small amount of DRAM could help, that would b=
e great, but DRAM is getting very expensive and reducing the=C2=A0number of=
builds might speed things up at no cost.=C2=A0</div><span class=3D"gmail_s= ignature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><= div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir= =3D"ltr">Kevin Oberman, Part time kid herder and retired Network Engineer<b= r>E-mail: <a href=3D"mailto:
rkoberman@gmail.com" target=3D"_blank">rkoberma=
n@gmail.com</a><br></div><div>PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB= 39EF1B055683</div></div></div></div></div></div></div></div></div>
--000000000000bfce64064c194f08--
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to
news-admin@muc.de
--- Synchronet 3.21d-Linux NewsLink 1.2