From Newsgroup: muc.lists.freebsd.stable

Hi folks,

Recently, I've been upgrading from releng/14.3 to releng/14.4 and as
usual, the first thing I did after booting into new BE was 'poudriere
bulk' (also, in releng/14.4 jail). For my set of packages, full
rebuild with cold ccache takes about a day, however this time it took
_twice_ as long. I did not notice any difference in CPU frequency or utilization during the build, everything looked completely normal.

After the build finished, compared the cold build times for ports I maintain:

/usr/local/poudriere/data/logs/bulk/releng143-default/2026-01-18_15h59m51s/logs/tremotesf-qt6-2.9.1.log

<==
build time: 00:09:38

/usr/local/poudriere/data/logs/bulk/releng144-default/2026-03-08_17h24m59s/logs/tremotesf-qt6-2.9.1.log

<==
build time: 00:23:00

/usr/local/poudriere/data/logs/bulk/releng135-default/2026-01-24_16h37m38s/logs/xscreensaver-6.14.log

<==
build time: 00:03:24

/usr/local/poudriere/data/logs/bulk/releng143-default/2026-01-24_16h41m22s/logs/xscreensaver-6.14.log

<==
build time: 00:03:20

/usr/local/poudriere/data/logs/bulk/releng144-default/2026-03-08_17h24m59s/logs/xscreensaver-6.14.log

<==
build time: 00:07:33

Also, unjailed 'poudriere options' for my set went up from about 16 to
22 minutes.
Quick and dirty ad-hoc benchmarks using 'openssl speed', 'lzbench',
and 'diskinfo' didn't show significant difference, just regular small fluctuations.

I don't believe issue may be in any of the build, system, or any other configurations, or in ccache/poudriere packages -- that didn't change.

The only things changed are: kernel, and world. Thus, I've rebooted
into 14.3 BE and observed regular 16 minutes for poudriere options,
then combined 14.3 kernel with 14.4 world and got slowdown to 22
minutes again. It seems that the problem is in the world rather than
in kernel, something like 'cc suddenly started to spend twice to
thrice more time on same code' ('poudriere options' runs 'make config-recursive', which in turn runs 'cc' at times).

Not adding dmesg or hardware configuration yet deliberately as it
doesn't seem to matter, what matters is same hardware with the same
config became twice as slow in tasks related to building code. Oddly
enough, while maintaining performance in other tasks (for example, my Transmission instance seeds ~300 Mbps and average CPU utilization is
~12% before and after).

Any thoughts, ideas, etc would be greatly appreciated.

--
wbr,
Anton


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21d-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

On 13 Mar 2026, at 14:53, Olivier Certner <olce@freebsd.org> wrote:

You are likely hitting a known problem, see thread "performance regressions in 15.0". E.g., just picked one mail saying what I also personally think about it:
https://lists.freebsd.org/archives/freebsd-current/2025-December/009623.html (arguably, some RAM factories have burnt out for retail consumers since then...)

AFAIK, the default was never switched back, but if you're building from source, a new knob was added to recover the previous behavior, see:
https://cgit.freebsd.org/src/commit/?id=8d5a11cd0137d3ad

It looks like the performance hit is particularly noticeable when you fork a lot of cc processes with short lifetimes, i.e. when running configure scripts that check features with small .c files.
Apparently loading a relatively large static executable is much faster than loading a small dynamic executable linked to a large dynamic library. Maybe this is exacerbated by PIE, and/or having to look through a large set of symbols.
I think it would be interesting if someone could figure out where the bottleneck lies in the dynamic linker, and if any improvements are possible.
-Dimitry
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21d-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

Perhaps a naive question, but is debugging enabled?

On 3/13/26 10:17, Dimitry Andric wrote:

On 13 Mar 2026, at 14:53, Olivier Certner <olce@freebsd.org> wrote:

You are likely hitting a known problem, see thread "performance regressions in 15.0". E.g., just picked one mail saying what I also personally think about it:
https://lists.freebsd.org/archives/freebsd-current/2025-December/009623.html >> (arguably, some RAM factories have burnt out for retail consumers since then...)

AFAIK, the default was never switched back, but if you're building from source, a new knob was added to recover the previous behavior, see:
https://cgit.freebsd.org/src/commit/?id=8d5a11cd0137d3ad

It looks like the performance hit is particularly noticeable when you fork a lot of cc processes with short lifetimes, i.e. when running configure scripts that check features with small .c files.

Apparently loading a relatively large static executable is much faster than loading a small dynamic executable linked to a large dynamic library. Maybe this is exacerbated by PIE, and/or having to look through a large set of symbols.

I think it would be interesting if someone could figure out where the bottleneck lies in the dynamic linker, and if any improvements are possible.

-Dimitry

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21d-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

On Fri, Mar 13, 2026 at 3:54rC>PM Olivier Certner <olce@freebsd.org> wrote:

Hi Anton,

You are likely hitting a known problem, see thread "performance regressions in 15.0". E.g., just picked one mail saying what I also personally think about it:
https://lists.freebsd.org/archives/freebsd-current/2025-December/009623.html (arguably, some RAM factories have burnt out for retail consumers since then...)

AFAIK, the default was never switched back, but if you're building from source, a new knob was added to recover the previous behavior, see:
https://cgit.freebsd.org/src/commit/?id=8d5a11cd0137d3ad

Thanks and regards.

Olivier, thanks a lot for your suggestion, it's indeed exactly the
problem I suffered. I've rebuilt world with
WITH_LLVM_LINK_STATIC_LIBRARIES= and 'poudriere options' instantly
went back to 16-17 minutes. Meanwhile I also noted world build time
before and after (with ccache running, but CCACHE_DISABLED=true):

World build completed on Fri Mar 13 22:39:17 EET 2026
World built in 6202 seconds, ncpu: 8, make -j6
World build completed on Sat Mar 14 03:06:36 EET 2026
World built in 3730 seconds, ncpu: 8, make -j6

I'm pretty sure ports will build in reasonable time again, just didn't
rebuild poudriere jail yet.
--
wbr,
Anton
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21d-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

On 2026-03-13 21:53:48 (+0800), Olivier Certner wrote:

You are likely hitting a known problem, see thread "performance
regressions in 15.0". E.g., just picked one mail saying what I also personally think about it: https://lists.freebsd.org/archives/freebsd-current/2025-December/009623.html (arguably, some RAM factories have burnt out for retail consumers
since then...)

AFAIK, the default was never switched back, but if you're building
from source, a new knob was added to recover the previous behavior,
see:
https://cgit.freebsd.org/src/commit/?id=8d5a11cd0137d3ad

Does this affect the package builders? Does that build fix go in the
host or in the build jails?

Thanks.

Philip [hat: cluster babysitter]


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21d-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

On Tue, Mar 17, 2026 at 12:40rC>AM Philip Paeps <philip@freebsd.org> wrote:

On 2026-03-16 19:18:45 (+0800), Mark Millard wrote:

On 3/15/26 21:18, Philip Paeps wrote:

On 2026-03-13 21:53:48 (+0800), Olivier Certner wrote:

You are likely hitting a known problem, see thread "performance
regressions in 15.0". E.g., just picked one mail saying what I
also
personally think about it:
https://lists.freebsd.org/archives/freebsd-current/2025-
December/009623.html
(arguably, some RAM factories have burnt out for retail consumers
since then...)

AFAIK, the default was never switched back, but if you're building
from source, a new knob was added to recover the previous behavior,
see:
https://cgit.freebsd.org/src/commit/?id=8d5a11cd0137d3ad

Does this affect the package builders?

Yes for the 15.* jails, the main jails [so: 16.0], and the future
14.4+
jails.

(The 13.5 jails and the 14.3 jails do not have the problem.)

Does that build fix go in the
host or in the build jails?

What primarily matters is how the world put in the build-jail was
built.

pkgmgr controls the build jails. I don't think they use clusteradm
images in the jails. We only supply the host operating system.

We use default releases (or security updates of default releases)
Antoine
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21d-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

On Tue, Mar 17, 2026 at 7:20rC>PM Mark Millard <marklmi@yahoo.com> wrote:

What about for main's builder jails (so: no releases, ampere2 and
beefy24 builder systems as stands)?

We use default (empty src.conf, empty make.conf)
Antoine
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21e-Linux NewsLink 1.2