From Newsgroup: muc.lists.freebsd.stable

An example is during the cpdup activities when multiple happen
in overlappingtime frames:
. . .
29502 0 root 0 0 20976Ki 4832Ki biowr 2 0:00 0.06% cpdup -i0 -s0 -f -x ref 01
29479 0 root 0 0 20976Ki 4836Ki biord 7 0:01 0.75% cpdup -i0 -o ref 32
29467 0 root 0 0 20976Ki 4824Ki getblk 26 0:01 0.63% cpdup -i0 -o ref 31
29429 0 root 1 0 20976Ki 4768Ki flswai 23 0:01 0.61% cpdup -i0 -o ref 30
29414 0 root 0 0 20976Ki 4780Ki flswai 28 0:01 0.70% cpdup -i0 -o ref 29
29396 0 root 59 0 16872Ki 4948Ki wait 9 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29382 0 root 0 0 20976Ki 4876Ki flswai 15 0:01 0.95% cpdup -i0 -o ref 28
29353 0 root 59 0 16872Ki 4948Ki wait 7 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29326 0 root 0 0 20976Ki 4804Ki biord 5 0:01 0.44% cpdup -i0 -o ref 27
29308 0 root 0 0 20976Ki 4816Ki biord 2 0:01 0.77% cpdup -i0 -o ref 26
29307 0 root 59 0 16872Ki 4948Ki wait 5 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29273 0 root 59 0 16872Ki 4944Ki wait 4 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29262 0 root 0 0 20976Ki 4768Ki flswai 20 0:01 0.76% cpdup -i0 -o ref 25
29233 0 root 0 0 20976Ki 4784Ki flswai 3 0:01 0.71% cpdup -i0 -o ref 24
29232 0 root 59 0 16872Ki 4944Ki wait 6 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
. . .
It took over 6 minutes for the 32 builders to finish. (Normal is
under 2 seconds.)
Also, later activity gets extensive "rename" STATE time, again, at least
some of it during cpdup activity:
. . .
80055 0 root 59 0 16872Ki 4968Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][06]: build_pkg (boost-jam-1.88.0) (sh)
78347 0 root 0 0 20976Ki 4624Ki rename 0 0:00 1.05% cpdup -i0 -s0 -f -x ref 31
78331 0 root 59 0 16872Ki 4960Ki wait 18 0:00 0.00% sh: poudriere[official-amd64-default][31]: build_pkg (lua53-5.3.6_1) (sh)
77930 0 root 0 0 20976Ki 4636Ki rename 29 0:00 1.02% cpdup -i0 -s0 -f -x ref 30
77912 0 root 59 0 16872Ki 4960Ki wait 25 0:00 0.00% sh: poudriere[official-amd64-default][30]: build_pkg (libsunacl-1.0.1_1) (sh)
75398 0 root 0 0 20976Ki 4628Ki rename 2 0:00 0.99% cpdup -i0 -s0 -f -x ref 16
75335 0 root 59 0 16872Ki 4960Ki wait 20 0:00 0.00% sh: poudriere[official-amd64-default][16]: build_pkg (iwmbt-firmware-20250410) (sh)
75324 0 root 0 0 20976Ki 4628Ki rename 24 0:00 1.04% cpdup -i0 -s0 -f -x ref 19
75308 0 root 59 0 16872Ki 4960Ki wait 24 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (lua54-5.4.8) (sh)
74963 0 root 0 0 20976Ki 4624Ki rename 4 0:00 1.05% cpdup -i0 -s0 -f -x ref 32
74922 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][32]: build_pkg (lzo2-2.10_1) (sh)
72722 0 root 0 0 20976Ki 4776Ki rename 1 0:00 1.28% cpdup -i0 -s0 -f -x ref 15
72647 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][15]: build_pkg (indexinfo-0.3.1_1) (sh)
71967 0 root 0 0 20976Ki 4856Ki rename 30 0:00 1.48% cpdup -i0 -s0 -f -x ref 25
71917 0 root 59 0 16872Ki 4960Ki wait 1 0:00 0.00% sh: poudriere[official-amd64-default][25]: build_pkg (mpg123-1.33.2) (sh)
71629 0 root 0 0 20976Ki 4828Ki rename 31 0:00 1.61% cpdup -i0 -s0 -f
. . .
But I'll also see such on c compiles, ld commands, etc. I've
not seen rename for pkg-static but I have seen flswai for it.
The system spends lots of time 95%+ idle from the wait
activities.
I see such directly booted from the USB3 media (a 15.0-ALPHA4
context on UFS media) and when using that media via chroot
from both ZFS and UFS boots that are not USB based. The ZFS
and UFS boots do not show the behavior with the normal
non-USB3 media used instead.
The system in use is an AMD 7950X3D with 32 FreeBSD cpus,
192 GiBytes of RAM. main 16 booting for non-USB boots
and 15.0-ALPHA4 boots for the USB3 boots. kernel and
world are via official pkgbase distribution installs:
it is not a personal build of the kernel or world.
More rename and flswai examples:
. . .
39320 183 nobody 59 0 171676Ki 160576Ki rename 29 0:00 2.17% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3372Ki wait 25 0:00 0.04% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 10 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39327 179 nobody 59 0 105680Ki 93264Ki biowr 2 0:00 0.61% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o test /usr/lib/crt
39325 179 nobody 59 0 165928Ki 155088Ki wait 17 0:00 1.00% /usr/bin/cc -o test ../test.o
39294 179 nobody 59 0 14744Ki 3448Ki wait 10 0:00 0.01% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki rename 22 0:03 0.47% cpdup -i0 -s0 -f -x ref 26
32420 0 root 0 0 20976Ki 5024Ki rename 13 0:03 0.50% cpdup -i0 -s0 -f -x ref 11
32358 0 root 0 0 20976Ki 5068Ki rename 26 0:03 0.53% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki 4960Ki wait 9 0:00 0.00% sh: poudriere[official-amd64-default][20]: build_pkg (linux-c7-libaio-0.3.109) (sh)
32301 0 root 59 0 16872Ki 4960Ki wait 19 0:00 0.00% sh: poudriere[official-amd64-default][11]: build_pkg (linux-c7-dosfstools-3.0.20) (sh)
13808 0 root 0 0 20976Ki 4984Ki rename 4 0:03 0.57% cpdup -i0 -s0 -f -x ref 19
12316 0 root 0 0 20976Ki 5072Ki biowr 27 0:03 0.56% cpdup -i0 -s0 -f -x ref 13
12283 0 root 0 0 20976Ki 5068Ki rename 5 0:03 0.53% cpdup -i0 -s0 -f -x ref 01
12277 0 root 0 0 20976Ki 5068Ki rename 28 0:03 0.54% cpdup -i0 -s0 -f -x ref 14
12096 0 root 59 0 16872Ki 4960Ki wait 27 0:00 0.00% sh: poudriere[official-amd64-def
. . .
. . .
39927 183 nobody 59 0 171676Ki 160396Ki flswai 13 0:00 2.03% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3400Ki wait 3 0:00 0.06% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 28 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39924 179 nobody 59 0 125140Ki 95572Ki flswai 20 0:00 0.60% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 0 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 13 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 16 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 27 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39923 179 nobody 59 0 165928Ki 155092Ki wait 0 0:00 1.00% /usr/bin/cc -o dynamic.exe -dynamic dynamic.o
39294 179 nobody 59 0 14744Ki 3464Ki wait 9 0:00 0.02% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki flswai 17 0:03 2.82% cpdup -i0 -s0 -f -x ref 26
32420 0 root 1 0 20976Ki 5024Ki flswai 25 0:03 2.81% cpdup -i0 -s0 -f -x ref 11
32358 0 root 1 0 20976Ki 5068Ki flswai 14 0:03 2.79% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki
. . .
. . .
53486 185 root 0 0 66428Ki 34196Ki getblk 25 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53485 185 root 59 0 23308Ki 12024Ki wait 0 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53446 185 root 59 0 14744Ki 3504Ki wait 11 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
53445 185 root 59 0 14284Ki 3828Ki wait 1 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/p5-YAML-Tiny build-depends
96359 183 root 0 0 66428Ki 34196Ki flswai 31 0:00 1.43% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96358 183 root 59 0 23308Ki 12020Ki wait 29 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96331 183 root 59 0 14744Ki 3504Ki wait 5 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
96330 183 root 59 0 14284Ki 3860Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/security/cyrus-sasl2 patch-depends
9658 170 nobody 59 0 14224Ki 2852Ki flswai 3 0:00 0.02% install -m 0644 ./libcdio.info /wrkdirs/usr/ports/sysutils/libcdio/work/stage/usr/local/share/info
9652 170 nobody 59 0 14744Ki 3420Ki wait 2 0:00 0.00% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9643 170 nobody 59 0 14744Ki 3420Ki wait 8 0:00 0.03% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9642 170 nobody 59 0 14776Ki 3652Ki wait 21 0:00 0.04% /usr/local/bin/gmake install-exec-am install-data-am
9641 170 nobody 59 0 14776Ki 3632Ki wait 3 0:00 0.04% /usr/local/bin/gmake install
9636 170 nobody 59 0 14744Ki 3424Ki wait 15 0:00 0.03% /bin/sh -c fail=; \\\nif (target_option=k; case ${target_option-} in ?) ;; *) echo "am__make_running_with_option: interna
9635 170 nobody 59 0 14776Ki 3708Ki wait 0 0:00 0.06% /usr/local/bin/gmake INSTALL_PROGRAM=/bin/sh /wrkdirs/usr/ports/sysutils/libcdio/work/libcdio-2.2.0/install-sh -c -s inst
9634 170 nobody 59 0 14776Ki 3680Ki wait 29 0:00 0.08% /usr/local/bin/gmake -f Makefile DESTDIR=/wrkdirs/usr/ports/sysutils/libcdio/work/stage install-strip
9633 170 nobody 59 0 14744Ki 3428Ki wait 13 0:00 0.02% /bin/sh -e /tmp/makeOENiKj
9611 170 nobody 59 0 16848Ki 3876Ki wait 21 0:00 0.22% /usr/bin/make -C /usr/ports/sysutils/libcdio stage
51574 158 root 0 0 74620Ki 34184Ki flswai 20 0:01 0.28% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51573 158 root 59 0 23308Ki 12020Ki wait 20 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51547 158 root 59 0 14744Ki 3512Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
51545 158 root 59 0 14284Ki 3852Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/hyphen build-depends
56591 157 root 0 0 66428Ki 34184Ki getblk 17 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56590 157 root 59 0 23308Ki 12024Ki wait 12 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56528 157 root 59 0 14744Ki 3504Ki wait 28 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56469 157 root 59 0 14284Ki 3832Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-MIME-Base32 build-depends
56593 155 root 0 0 66428Ki 34188Ki getblk 8 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56592 155 root 59 0 23308Ki 12024Ki wait 4 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56559 155 root 59 0 14744Ki 3508Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56532 155 root 59 0 14284Ki 3828Ki wait 13 0:00 0.00% /usr/bin/make -C /usr/ports/devel/p5-Parse-Yapp build-depends
52781 145 root 0 0 74620Ki 34244Ki flswai 12 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52780 145 root 59 0 23308Ki 12024Ki wait 22 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52753 145 root 59 0 14744Ki 3508Ki wait 24 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
52749 145 root 59 0 14284Ki 3824Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-JSON build-depends
9610 0 root 59 0 16872Ki 4972Ki wait 14 0:00 0.01% sh: poudriere[official-amd64-default][14]: build_pkg (libcdio-2.2.0) (sh)
99560 0 root 1 0 20976Ki 4860Ki flswai 15 0:00 2.82% cpdup -i0 -s0 -f -x ref 01
99539 0 root 59 0 16872Ki 4960Ki wait 10 0:00 0.00% sh: poudriere[official-amd64-default][01]: build_pkg (texi2html-5.0_5,1) (sh)
98112 0 root 1 0 20976Ki 4828Ki flswai 19 0:00 3.47% cpdup -i0 -s0 -f -x ref 28
98096 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][28]: build_pkg (gdbm-1.26) (sh)
97694 0 root 1 0 20976Ki 4844Ki flswai 7 0:01 2.51% cpdup -i0 -s0 -f -x ref 27
97665 0 root 59 0 16872Ki 4960Ki wait 0 0:00 0.00% sh: poudriere[official-amd64-default][27]: build_pkg (sudo-1.9.17p2) (sh)
96108 0 root 0 0 20976Ki 4800Ki flswai 29 0:01 3.06% cpdup -i0 -s0 -f -x ref 19
96091 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (libgpg-error-1.55) (sh)
93884 0 root 0 0 20976Ki 5064Ki flswai 19 0:01 2.43% cpdup -i0 -s0 -f -x ref 26
93857 0 root 59 0 16872Ki 4960Ki wait 16 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (libxml2-2.14.5) (sh)
93271 0 root 0 0 20976Ki 5224Ki flswai 5 0:01 2.38% cpdup -i0 -s0 -f -x ref 12
93212 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][12]: build_pkg (lame-3.100_5) (sh)
88119 0 root 0 0 20976Ki 5212Ki flswai 0 0:01 2.51% cpdup -i0 -s0 -f -x ref 32
88077 0 root 0 0 20976Ki 5176Ki flswai 0 0:01 2.38% cpdup -i0 -s0 -f -x ref 31
. . .
===
Mark Millard
marklmi at yahoo.com
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

On Sep 29, 2025, at 13:01, Mark Millard <marklmi@yahoo.com> wrote:

An example is during the cpdup activities when multiple happen
in overlappingtime frames:

I'll note that I see this on the amd64 32-FreeBSD-cpu system
but not on the aarch64 8-FreeBSD-cpu Windows Dev Kit 2023
system. May be at some point I'll try the older 16-FreeBSD-cpu
aarch64 (Cortex-A72) system.
Also, on the 7950X3D amd74 system, I see the behavior with
14.3-Stable. Apparently, this is not new with 15+. It has
been a long time since I'd tried using an amd64 system for
such activity based on using USB3 media. But it has been
common for me for aarch64 over that time frame.

. . .
29502 0 root 0 0 20976Ki 4832Ki biowr 2 0:00 0.06% cpdup -i0 -s0 -f -x ref 01
29479 0 root 0 0 20976Ki 4836Ki biord 7 0:01 0.75% cpdup -i0 -o ref 32
29467 0 root 0 0 20976Ki 4824Ki getblk 26 0:01 0.63% cpdup -i0 -o ref 31
29429 0 root 1 0 20976Ki 4768Ki flswai 23 0:01 0.61% cpdup -i0 -o ref 30
29414 0 root 0 0 20976Ki 4780Ki flswai 28 0:01 0.70% cpdup -i0 -o ref 29
29396 0 root 59 0 16872Ki 4948Ki wait 9 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29382 0 root 0 0 20976Ki 4876Ki flswai 15 0:01 0.95% cpdup -i0 -o ref 28
29353 0 root 59 0 16872Ki 4948Ki wait 7 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29326 0 root 0 0 20976Ki 4804Ki biord 5 0:01 0.44% cpdup -i0 -o ref 27
29308 0 root 0 0 20976Ki 4816Ki biord 2 0:01 0.77% cpdup -i0 -o ref 26
29307 0 root 59 0 16872Ki 4948Ki wait 5 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29273 0 root 59 0 16872Ki 4944Ki wait 4 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29262 0 root 0 0 20976Ki 4768Ki flswai 20 0:01 0.76% cpdup -i0 -o ref 25
29233 0 root 0 0 20976Ki 4784Ki flswai 3 0:01 0.71% cpdup -i0 -o ref 24
29232 0 root 59 0 16872Ki 4944Ki wait 6 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
. . .

It took over 6 minutes for the 32 builders to finish. (Normal is
under 2 seconds.)

Also, later activity gets extensive "rename" STATE time, again, at least
some of it during cpdup activity:

. . .
80055 0 root 59 0 16872Ki 4968Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][06]: build_pkg (boost-jam-1.88.0) (sh)
78347 0 root 0 0 20976Ki 4624Ki rename 0 0:00 1.05% cpdup -i0 -s0 -f -x ref 31
78331 0 root 59 0 16872Ki 4960Ki wait 18 0:00 0.00% sh: poudriere[official-amd64-default][31]: build_pkg (lua53-5.3.6_1) (sh)
77930 0 root 0 0 20976Ki 4636Ki rename 29 0:00 1.02% cpdup -i0 -s0 -f -x ref 30
77912 0 root 59 0 16872Ki 4960Ki wait 25 0:00 0.00% sh: poudriere[official-amd64-default][30]: build_pkg (libsunacl-1.0.1_1) (sh)
75398 0 root 0 0 20976Ki 4628Ki rename 2 0:00 0.99% cpdup -i0 -s0 -f -x ref 16
75335 0 root 59 0 16872Ki 4960Ki wait 20 0:00 0.00% sh: poudriere[official-amd64-default][16]: build_pkg (iwmbt-firmware-20250410) (sh)
75324 0 root 0 0 20976Ki 4628Ki rename 24 0:00 1.04% cpdup -i0 -s0 -f -x ref 19
75308 0 root 59 0 16872Ki 4960Ki wait 24 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (lua54-5.4.8) (sh)
74963 0 root 0 0 20976Ki 4624Ki rename 4 0:00 1.05% cpdup -i0 -s0 -f -x ref 32
74922 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][32]: build_pkg (lzo2-2.10_1) (sh)
72722 0 root 0 0 20976Ki 4776Ki rename 1 0:00 1.28% cpdup -i0 -s0 -f -x ref 15
72647 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][15]: build_pkg (indexinfo-0.3.1_1) (sh)
71967 0 root 0 0 20976Ki 4856Ki rename 30 0:00 1.48% cpdup -i0 -s0 -f -x ref 25
71917 0 root 59 0 16872Ki 4960Ki wait 1 0:00 0.00% sh: poudriere[official-amd64-default][25]: build_pkg (mpg123-1.33.2) (sh)
71629 0 root 0 0 20976Ki 4828Ki rename 31 0:00 1.61% cpdup -i0 -s0 -f
. . .

But I'll also see such on c compiles, ld commands, etc. I've
not seen rename for pkg-static but I have seen flswai for it.

The system spends lots of time 95%+ idle from the wait
activities.

I see such directly booted from the USB3 media (a 15.0-ALPHA4
context on UFS media) and when using that media via chroot
from both ZFS and UFS boots that are not USB based. The ZFS
and UFS boots do not show the behavior with the normal
non-USB3 media used instead.

The system in use is an AMD 7950X3D with 32 FreeBSD cpus,
192 GiBytes of RAM. main 16 booting for non-USB boots
and 15.0-ALPHA4 boots for the USB3 boots. kernel and
world are via official pkgbase distribution installs:
it is not a personal build of the kernel or world.

More rename and flswai examples:

. . .
39320 183 nobody 59 0 171676Ki 160576Ki rename 29 0:00 2.17% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3372Ki wait 25 0:00 0.04% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 10 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39327 179 nobody 59 0 105680Ki 93264Ki biowr 2 0:00 0.61% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o test /usr/lib/crt
39325 179 nobody 59 0 165928Ki 155088Ki wait 17 0:00 1.00% /usr/bin/cc -o test ../test.o
39294 179 nobody 59 0 14744Ki 3448Ki wait 10 0:00 0.01% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki rename 22 0:03 0.47% cpdup -i0 -s0 -f -x ref 26
32420 0 root 0 0 20976Ki 5024Ki rename 13 0:03 0.50% cpdup -i0 -s0 -f -x ref 11
32358 0 root 0 0 20976Ki 5068Ki rename 26 0:03 0.53% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki 4960Ki wait 9 0:00 0.00% sh: poudriere[official-amd64-default][20]: build_pkg (linux-c7-libaio-0.3.109) (sh)
32301 0 root 59 0 16872Ki 4960Ki wait 19 0:00 0.00% sh: poudriere[official-amd64-default][11]: build_pkg (linux-c7-dosfstools-3.0.20) (sh)
13808 0 root 0 0 20976Ki 4984Ki rename 4 0:03 0.57% cpdup -i0 -s0 -f -x ref 19
12316 0 root 0 0 20976Ki 5072Ki biowr 27 0:03 0.56% cpdup -i0 -s0 -f -x ref 13
12283 0 root 0 0 20976Ki 5068Ki rename 5 0:03 0.53% cpdup -i0 -s0 -f -x ref 01
12277 0 root 0 0 20976Ki 5068Ki rename 28 0:03 0.54% cpdup -i0 -s0 -f -x ref 14
12096 0 root 59 0 16872Ki 4960Ki wait 27 0:00 0.00% sh: poudriere[official-amd64-def
. . .

. . .
39927 183 nobody 59 0 171676Ki 160396Ki flswai 13 0:00 2.03% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3400Ki wait 3 0:00 0.06% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 28 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39924 179 nobody 59 0 125140Ki 95572Ki flswai 20 0:00 0.60% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 0 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 13 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 16 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 27 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39923 179 nobody 59 0 165928Ki 155092Ki wait 0 0:00 1.00% /usr/bin/cc -o dynamic.exe -dynamic dynamic.o
39294 179 nobody 59 0 14744Ki 3464Ki wait 9 0:00 0.02% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki flswai 17 0:03 2.82% cpdup -i0 -s0 -f -x ref 26
32420 0 root 1 0 20976Ki 5024Ki flswai 25 0:03 2.81% cpdup -i0 -s0 -f -x ref 11
32358 0 root 1 0 20976Ki 5068Ki flswai 14 0:03 2.79% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki
. . .

. . .
53486 185 root 0 0 66428Ki 34196Ki getblk 25 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53485 185 root 59 0 23308Ki 12024Ki wait 0 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53446 185 root 59 0 14744Ki 3504Ki wait 11 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
53445 185 root 59 0 14284Ki 3828Ki wait 1 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/p5-YAML-Tiny build-depends
96359 183 root 0 0 66428Ki 34196Ki flswai 31 0:00 1.43% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96358 183 root 59 0 23308Ki 12020Ki wait 29 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96331 183 root 59 0 14744Ki 3504Ki wait 5 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
96330 183 root 59 0 14284Ki 3860Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/security/cyrus-sasl2 patch-depends
9658 170 nobody 59 0 14224Ki 2852Ki flswai 3 0:00 0.02% install -m 0644 ./libcdio.info /wrkdirs/usr/ports/sysutils/libcdio/work/stage/usr/local/share/info
9652 170 nobody 59 0 14744Ki 3420Ki wait 2 0:00 0.00% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9643 170 nobody 59 0 14744Ki 3420Ki wait 8 0:00 0.03% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9642 170 nobody 59 0 14776Ki 3652Ki wait 21 0:00 0.04% /usr/local/bin/gmake install-exec-am install-data-am
9641 170 nobody 59 0 14776Ki 3632Ki wait 3 0:00 0.04% /usr/local/bin/gmake install
9636 170 nobody 59 0 14744Ki 3424Ki wait 15 0:00 0.03% /bin/sh -c fail=; \\\nif (target_option=k; case ${target_option-} in ?) ;; *) echo "am__make_running_with_option: interna
9635 170 nobody 59 0 14776Ki 3708Ki wait 0 0:00 0.06% /usr/local/bin/gmake INSTALL_PROGRAM=/bin/sh /wrkdirs/usr/ports/sysutils/libcdio/work/libcdio-2.2.0/install-sh -c -s inst
9634 170 nobody 59 0 14776Ki 3680Ki wait 29 0:00 0.08% /usr/local/bin/gmake -f Makefile DESTDIR=/wrkdirs/usr/ports/sysutils/libcdio/work/stage install-strip
9633 170 nobody 59 0 14744Ki 3428Ki wait 13 0:00 0.02% /bin/sh -e /tmp/makeOENiKj
9611 170 nobody 59 0 16848Ki 3876Ki wait 21 0:00 0.22% /usr/bin/make -C /usr/ports/sysutils/libcdio stage
51574 158 root 0 0 74620Ki 34184Ki flswai 20 0:01 0.28% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51573 158 root 59 0 23308Ki 12020Ki wait 20 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51547 158 root 59 0 14744Ki 3512Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
51545 158 root 59 0 14284Ki 3852Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/hyphen build-depends
56591 157 root 0 0 66428Ki 34184Ki getblk 17 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56590 157 root 59 0 23308Ki 12024Ki wait 12 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56528 157 root 59 0 14744Ki 3504Ki wait 28 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56469 157 root 59 0 14284Ki 3832Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-MIME-Base32 build-depends
56593 155 root 0 0 66428Ki 34188Ki getblk 8 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56592 155 root 59 0 23308Ki 12024Ki wait 4 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56559 155 root 59 0 14744Ki 3508Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56532 155 root 59 0 14284Ki 3828Ki wait 13 0:00 0.00% /usr/bin/make -C /usr/ports/devel/p5-Parse-Yapp build-depends
52781 145 root 0 0 74620Ki 34244Ki flswai 12 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52780 145 root 59 0 23308Ki 12024Ki wait 22 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52753 145 root 59 0 14744Ki 3508Ki wait 24 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
52749 145 root 59 0 14284Ki 3824Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-JSON build-depends
9610 0 root 59 0 16872Ki 4972Ki wait 14 0:00 0.01% sh: poudriere[official-amd64-default][14]: build_pkg (libcdio-2.2.0) (sh)
99560 0 root 1 0 20976Ki 4860Ki flswai 15 0:00 2.82% cpdup -i0 -s0 -f -x ref 01
99539 0 root 59 0 16872Ki 4960Ki wait 10 0:00 0.00% sh: poudriere[official-amd64-default][01]: build_pkg (texi2html-5.0_5,1) (sh)
98112 0 root 1 0 20976Ki 4828Ki flswai 19 0:00 3.47% cpdup -i0 -s0 -f -x ref 28
98096 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][28]: build_pkg (gdbm-1.26) (sh)
97694 0 root 1 0 20976Ki 4844Ki flswai 7 0:01 2.51% cpdup -i0 -s0 -f -x ref 27
97665 0 root 59 0 16872Ki 4960Ki wait 0 0:00 0.00% sh: poudriere[official-amd64-default][27]: build_pkg (sudo-1.9.17p2) (sh)
96108 0 root 0 0 20976Ki 4800Ki flswai 29 0:01 3.06% cpdup -i0 -s0 -f -x ref 19
96091 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (libgpg-error-1.55) (sh)
93884 0 root 0 0 20976Ki 5064Ki flswai 19 0:01 2.43% cpdup -i0 -s0 -f -x ref 26
93857 0 root 59 0 16872Ki 4960Ki wait 16 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (libxml2-2.14.5) (sh)
93271 0 root 0 0 20976Ki 5224Ki flswai 5 0:01 2.38% cpdup -i0 -s0 -f -x ref 12
93212 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][12]: build_pkg (lame-3.100_5) (sh)
88119 0 root 0 0 20976Ki 5212Ki flswai 0 0:01 2.51% cpdup -i0 -s0 -f -x ref 32
88077 0 root 0 0 20976Ki 5176Ki flswai 0 0:01 2.38% cpdup -i0 -s0 -f -x ref 31
. . .

===
Mark Millard
marklmi at yahoo.com
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

[The new material here ends up being about nameicap_cleanup
and its exclusive use of mnt_renamelock being one potential
bottleneck involved here. I make no claim it has anything to
do with the flswai activity reported. The possible
bottleneck is an observation, not something that I claim
there is any alternative to. I do not know if this is of any
interest or not.]
On Sep 29, 2025, at 16:06, Mark Millard <marklmi@yahoo.com> wrote:

On Sep 29, 2025, at 13:01, Mark Millard <marklmi@yahoo.com> wrote:

An example is during the cpdup activities when multiple happen
in overlappingtime frames:

I'll note that I see this on the amd64 32-FreeBSD-cpu system
but not on the aarch64 8-FreeBSD-cpu Windows Dev Kit 2023
system. May be at some point I'll try the older 16-FreeBSD-cpu
aarch64 (Cortex-A72) system.

Also, on the 7950X3D amd74 system, I see the behavior with
14.3-Stable. Apparently, this is not new with 15+. It has
been a long time since I'd tried using an amd64 system for
such activity based on using USB3 media. But it has been
common for me for aarch64 over that time frame.

. . .
29502 0 root 0 0 20976Ki 4832Ki biowr 2 0:00 0.06% cpdup -i0 -s0 -f -x ref 01
29479 0 root 0 0 20976Ki 4836Ki biord 7 0:01 0.75% cpdup -i0 -o ref 32
29467 0 root 0 0 20976Ki 4824Ki getblk 26 0:01 0.63% cpdup -i0 -o ref 31
29429 0 root 1 0 20976Ki 4768Ki flswai 23 0:01 0.61% cpdup -i0 -o ref 30
29414 0 root 0 0 20976Ki 4780Ki flswai 28 0:01 0.70% cpdup -i0 -o ref 29
29396 0 root 59 0 16872Ki 4948Ki wait 9 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29382 0 root 0 0 20976Ki 4876Ki flswai 15 0:01 0.95% cpdup -i0 -o ref 28
29353 0 root 59 0 16872Ki 4948Ki wait 7 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29326 0 root 0 0 20976Ki 4804Ki biord 5 0:01 0.44% cpdup -i0 -o ref 27
29308 0 root 0 0 20976Ki 4816Ki biord 2 0:01 0.77% cpdup -i0 -o ref 26
29307 0 root 59 0 16872Ki 4948Ki wait 5 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29273 0 root 59 0 16872Ki 4944Ki wait 4 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29262 0 root 0 0 20976Ki 4768Ki flswai 20 0:01 0.76% cpdup -i0 -o ref 25
29233 0 root 0 0 20976Ki 4784Ki flswai 3 0:01 0.71% cpdup -i0 -o ref 24
29232 0 root 59 0 16872Ki 4944Ki wait 6 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
. . .

It took over 6 minutes for the 32 builders to finish. (Normal is
under 2 seconds.)


Also, later activity gets extensive "rename" STATE time, again, at least
some of it during cpdup activity:

Looks like the "rename" state is new (and I've only
seen in for 15+):
/usr/src/sys/kern/vfs_mount.c has as the source
of the name used for the "rname" STATE:
static int
mount_init(void *mem, int size, int flags)
{
struct mount *mp;

mp = (struct mount *)mem;
mtx_init(&mp->mnt_mtx, "struct mount mtx", NULL, MTX_DEF);
mtx_init(&mp->mnt_listmtx, "struct mount vlist mtx", NULL, MTX_DEF);
lockinit(&mp->mnt_explock, PVFS, "explock", 0, 0);
lockinit(&mp->mnt_renamelock, PVFS, "rename", 0, 0);
mp->mnt_pcpu = uma_zalloc_pcpu(pcpu_zone_16, M_WAITOK | M_ZERO);
mp->mnt_ref = 0;
mp->mnt_vfs_ops = 1;
mp->mnt_rootvnode = NULL;
return (0);
}
author Konstantin Belousov <kib@FreeBSD.org> 2025-06-02 07:05:06 +0000 committer Konstantin Belousov <kib@FreeBSD.org> 2025-07-04 15:23:42 +0000 commit ef6ea91593ebff73e2fc201efd9f848b71c5a125 (patch)
tree 5f90f06e4b190a3a8fe9b5f3a016b69415d62a6d /sys/kern/vfs_mount.c
parent f9cf745a5084dc0a38733db8e774252e4ed2f444 (diff)
VOP_RENAME: add mp-global lock
It is before all vnode locks, but after vn_start_write().
The lock prevents parallel rename operations on the same mount point,
which should in (near future) simplify a lot of code in VFS/fs that
otherwise need to code with either the changing hierarchy, or with the
lock order for vnodes due to changed hierarchy.
On renames, the lock is taken on the lowest stacked filesystem.
Otherwise rename could still occur in parallel, by performing one of op
on the lower fs.
Proposed by: mjg (long time ago)
Reviewed by: markj, olce
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D50648
. . .
The context here is not renaming activity, so my
guess here is that the following may be in use
in a wider range of contexts.
/usr/src/sys/kern/vfs_lookup.c has:
static void
nameicap_cleanup(struct nameidata *ndp, int error)
{
struct nameicap_tracker *nt, *nt1;
struct mount *mp;
KASSERT((ndp->ni_nctrack_mnt == NULL &&
TAILQ_EMPTY(&ndp->ni_cap_tracker)) ||
(ndp->ni_lcf & NI_LCF_CAP_DOTDOT) != 0,
("tracker active and not strictrelative"));
TAILQ_FOREACH_SAFE(nt, &ndp->ni_cap_tracker, nm_link, nt1) {
mp = nt->mp;
lockmgr(&mp->mnt_renamelock, LK_RELEASE, 0);
vfs_rel(mp);
TAILQ_REMOVE(&ndp->ni_cap_tracker, nt, nm_link);
free(nt, M_NAMEITRACKER);
}
mp = ndp->ni_nctrack_mnt;
if (mp != NULL) {
if (error == ERESTART) {
lockmgr(&mp->mnt_renamelock, LK_EXCLUSIVE, 0);
lockmgr(&mp->mnt_renamelock, LK_RELEASE, 0);
}
vfs_rel(mp);
ndp->ni_nctrack_mnt = NULL;
}
}
and it may be the source of the large number
"rename" STATE processes that are all waiting
on "rename" below, forcing 1 of 32 at a time
for cpdup (32 poudriere builders). Later there
is an example of cc also waiting on it when
there is also cpdup activity waiting on it.

. . .
80055 0 root 59 0 16872Ki 4968Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][06]: build_pkg (boost-jam-1.88.0) (sh)
78347 0 root 0 0 20976Ki 4624Ki rename 0 0:00 1.05% cpdup -i0 -s0 -f -x ref 31
78331 0 root 59 0 16872Ki 4960Ki wait 18 0:00 0.00% sh: poudriere[official-amd64-default][31]: build_pkg (lua53-5.3.6_1) (sh)
77930 0 root 0 0 20976Ki 4636Ki rename 29 0:00 1.02% cpdup -i0 -s0 -f -x ref 30
77912 0 root 59 0 16872Ki 4960Ki wait 25 0:00 0.00% sh: poudriere[official-amd64-default][30]: build_pkg (libsunacl-1.0.1_1) (sh)
75398 0 root 0 0 20976Ki 4628Ki rename 2 0:00 0.99% cpdup -i0 -s0 -f -x ref 16
75335 0 root 59 0 16872Ki 4960Ki wait 20 0:00 0.00% sh: poudriere[official-amd64-default][16]: build_pkg (iwmbt-firmware-20250410) (sh)
75324 0 root 0 0 20976Ki 4628Ki rename 24 0:00 1.04% cpdup -i0 -s0 -f -x ref 19
75308 0 root 59 0 16872Ki 4960Ki wait 24 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (lua54-5.4.8) (sh)
74963 0 root 0 0 20976Ki 4624Ki rename 4 0:00 1.05% cpdup -i0 -s0 -f -x ref 32
74922 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][32]: build_pkg (lzo2-2.10_1) (sh)
72722 0 root 0 0 20976Ki 4776Ki rename 1 0:00 1.28% cpdup -i0 -s0 -f -x ref 15
72647 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][15]: build_pkg (indexinfo-0.3.1_1) (sh)
71967 0 root 0 0 20976Ki 4856Ki rename 30 0:00 1.48% cpdup -i0 -s0 -f -x ref 25
71917 0 root 59 0 16872Ki 4960Ki wait 1 0:00 0.00% sh: poudriere[official-amd64-default][25]: build_pkg (mpg123-1.33.2) (sh)
71629 0 root 0 0 20976Ki 4828Ki rename 31 0:00 1.61% cpdup -i0 -s0 -f
. . .

But I'll also see such on c compiles, ld commands, etc. I've
not seen rename for pkg-static but I have seen flswai for it.

The system spends lots of time 95%+ idle from the wait
activities.

I see such directly booted from the USB3 media (a 15.0-ALPHA4
context on UFS media) and when using that media via chroot
from both ZFS and UFS boots that are not USB based. The ZFS
and UFS boots do not show the behavior with the normal
non-USB3 media used instead.

The system in use is an AMD 7950X3D with 32 FreeBSD cpus,
192 GiBytes of RAM. main 16 booting for non-USB boots
and 15.0-ALPHA4 boots for the USB3 boots. kernel and
world are via official pkgbase distribution installs:
it is not a personal build of the kernel or world.


More rename and flswai examples:

. . .
39320 183 nobody 59 0 171676Ki 160576Ki rename 29 0:00 2.17% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3372Ki wait 25 0:00 0.04% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 10 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39327 179 nobody 59 0 105680Ki 93264Ki biowr 2 0:00 0.61% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o test /usr/lib/crt
39325 179 nobody 59 0 165928Ki 155088Ki wait 17 0:00 1.00% /usr/bin/cc -o test ../test.o
39294 179 nobody 59 0 14744Ki 3448Ki wait 10 0:00 0.01% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki rename 22 0:03 0.47% cpdup -i0 -s0 -f -x ref 26
32420 0 root 0 0 20976Ki 5024Ki rename 13 0:03 0.50% cpdup -i0 -s0 -f -x ref 11
32358 0 root 0 0 20976Ki 5068Ki rename 26 0:03 0.53% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki 4960Ki wait 9 0:00 0.00% sh: poudriere[official-amd64-default][20]: build_pkg (linux-c7-libaio-0.3.109) (sh)
32301 0 root 59 0 16872Ki 4960Ki wait 19 0:00 0.00% sh: poudriere[official-amd64-default][11]: build_pkg (linux-c7-dosfstools-3.0.20) (sh)
13808 0 root 0 0 20976Ki 4984Ki rename 4 0:03 0.57% cpdup -i0 -s0 -f -x ref 19
12316 0 root 0 0 20976Ki 5072Ki biowr 27 0:03 0.56% cpdup -i0 -s0 -f -x ref 13
12283 0 root 0 0 20976Ki 5068Ki rename 5 0:03 0.53% cpdup -i0 -s0 -f -x ref 01
12277 0 root 0 0 20976Ki 5068Ki rename 28 0:03 0.54% cpdup -i0 -s0 -f -x ref 14
12096 0 root 59 0 16872Ki 4960Ki wait 27 0:00 0.00% sh: poudriere[official-amd64-def
. . .


. . .
39927 183 nobody 59 0 171676Ki 160396Ki flswai 13 0:00 2.03% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3400Ki wait 3 0:00 0.06% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 28 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39924 179 nobody 59 0 125140Ki 95572Ki flswai 20 0:00 0.60% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 0 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 13 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 16 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 27 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39923 179 nobody 59 0 165928Ki 155092Ki wait 0 0:00 1.00% /usr/bin/cc -o dynamic.exe -dynamic dynamic.o
39294 179 nobody 59 0 14744Ki 3464Ki wait 9 0:00 0.02% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki flswai 17 0:03 2.82% cpdup -i0 -s0 -f -x ref 26
32420 0 root 1 0 20976Ki 5024Ki flswai 25 0:03 2.81% cpdup -i0 -s0 -f -x ref 11
32358 0 root 1 0 20976Ki 5068Ki flswai 14 0:03 2.79% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki
. . .


. . .
53486 185 root 0 0 66428Ki 34196Ki getblk 25 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53485 185 root 59 0 23308Ki 12024Ki wait 0 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53446 185 root 59 0 14744Ki 3504Ki wait 11 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
53445 185 root 59 0 14284Ki 3828Ki wait 1 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/p5-YAML-Tiny build-depends
96359 183 root 0 0 66428Ki 34196Ki flswai 31 0:00 1.43% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96358 183 root 59 0 23308Ki 12020Ki wait 29 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96331 183 root 59 0 14744Ki 3504Ki wait 5 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
96330 183 root 59 0 14284Ki 3860Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/security/cyrus-sasl2 patch-depends
9658 170 nobody 59 0 14224Ki 2852Ki flswai 3 0:00 0.02% install -m 0644 ./libcdio.info /wrkdirs/usr/ports/sysutils/libcdio/work/stage/usr/local/share/info
9652 170 nobody 59 0 14744Ki 3420Ki wait 2 0:00 0.00% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9643 170 nobody 59 0 14744Ki 3420Ki wait 8 0:00 0.03% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9642 170 nobody 59 0 14776Ki 3652Ki wait 21 0:00 0.04% /usr/local/bin/gmake install-exec-am install-data-am
9641 170 nobody 59 0 14776Ki 3632Ki wait 3 0:00 0.04% /usr/local/bin/gmake install
9636 170 nobody 59 0 14744Ki 3424Ki wait 15 0:00 0.03% /bin/sh -c fail=; \\\nif (target_option=k; case ${target_option-} in ?) ;; *) echo "am__make_running_with_option: interna
9635 170 nobody 59 0 14776Ki 3708Ki wait 0 0:00 0.06% /usr/local/bin/gmake INSTALL_PROGRAM=/bin/sh /wrkdirs/usr/ports/sysutils/libcdio/work/libcdio-2.2.0/install-sh -c -s inst
9634 170 nobody 59 0 14776Ki 3680Ki wait 29 0:00 0.08% /usr/local/bin/gmake -f Makefile DESTDIR=/wrkdirs/usr/ports/sysutils/libcdio/work/stage install-strip
9633 170 nobody 59 0 14744Ki 3428Ki wait 13 0:00 0.02% /bin/sh -e /tmp/makeOENiKj
9611 170 nobody 59 0 16848Ki 3876Ki wait 21 0:00 0.22% /usr/bin/make -C /usr/ports/sysutils/libcdio stage
51574 158 root 0 0 74620Ki 34184Ki flswai 20 0:01 0.28% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51573 158 root 59 0 23308Ki 12020Ki wait 20 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51547 158 root 59 0 14744Ki 3512Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
51545 158 root 59 0 14284Ki 3852Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/hyphen build-depends
56591 157 root 0 0 66428Ki 34184Ki getblk 17 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56590 157 root 59 0 23308Ki 12024Ki wait 12 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56528 157 root 59 0 14744Ki 3504Ki wait 28 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56469 157 root 59 0 14284Ki 3832Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-MIME-Base32 build-depends
56593 155 root 0 0 66428Ki 34188Ki getblk 8 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56592 155 root 59 0 23308Ki 12024Ki wait 4 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56559 155 root 59 0 14744Ki 3508Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56532 155 root 59 0 14284Ki 3828Ki wait 13 0:00 0.00% /usr/bin/make -C /usr/ports/devel/p5-Parse-Yapp build-depends
52781 145 root 0 0 74620Ki 34244Ki flswai 12 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52780 145 root 59 0 23308Ki 12024Ki wait 22 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52753 145 root 59 0 14744Ki 3508Ki wait 24 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
52749 145 root 59 0 14284Ki 3824Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-JSON build-depends
9610 0 root 59 0 16872Ki 4972Ki wait 14 0:00 0.01% sh: poudriere[official-amd64-default][14]: build_pkg (libcdio-2.2.0) (sh)
99560 0 root 1 0 20976Ki 4860Ki flswai 15 0:00 2.82% cpdup -i0 -s0 -f -x ref 01
99539 0 root 59 0 16872Ki 4960Ki wait 10 0:00 0.00% sh: poudriere[official-amd64-default][01]: build_pkg (texi2html-5.0_5,1) (sh)
98112 0 root 1 0 20976Ki 4828Ki flswai 19 0:00 3.47% cpdup -i0 -s0 -f -x ref 28
98096 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][28]: build_pkg (gdbm-1.26) (sh)
97694 0 root 1 0 20976Ki 4844Ki flswai 7 0:01 2.51% cpdup -i0 -s0 -f -x ref 27
97665 0 root 59 0 16872Ki 4960Ki wait 0 0:00 0.00% sh: poudriere[official-amd64-default][27]: build_pkg (sudo-1.9.17p2) (sh)
96108 0 root 0 0 20976Ki 4800Ki flswai 29 0:01 3.06% cpdup -i0 -s0 -f -x ref 19
96091 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (libgpg-error-1.55) (sh)
93884 0 root 0 0 20976Ki 5064Ki flswai 19 0:01 2.43% cpdup -i0 -s0 -f -x ref 26
93857 0 root 59 0 16872Ki 4960Ki wait 16 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (libxml2-2.14.5) (sh)
93271 0 root 0 0 20976Ki 5224Ki flswai 5 0:01 2.38% cpdup -i0 -s0 -f -x ref 12
93212 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][12]: build_pkg (lame-3.100_5) (sh)
88119 0 root 0 0 20976Ki 5212Ki flswai 0 0:01 2.51% cpdup -i0 -s0 -f -x ref 32
88077 0 root 0 0 20976Ki 5176Ki flswai 0 0:01 2.38% cpdup -i0 -s0 -f -x ref 31
. . .

===
Mark Millard
marklmi at yahoo.com
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

[This note is tied to "flswai" instead of to "rename".]
On Sep 30, 2025, at 20:43, Mark Millard <marklmi@yahoo.com> wrote:

[The new material here ends up being about nameicap_cleanup
and its exclusive use of mnt_renamelock being one potential
bottleneck involved here. I make no claim it has anything to
do with the flswai activity reported. The possible
bottleneck is an observation, not something that I claim
there is any alternative to. I do not know if this is of any
interest or not.]

On Sep 29, 2025, at 16:06, Mark Millard <marklmi@yahoo.com> wrote:

On Sep 29, 2025, at 13:01, Mark Millard <marklmi@yahoo.com> wrote:

An example is during the cpdup activities when multiple happen
in overlappingtime frames:

I'll note that I see this on the amd64 32-FreeBSD-cpu system
but not on the aarch64 8-FreeBSD-cpu Windows Dev Kit 2023
system. May be at some point I'll try the older 16-FreeBSD-cpu
aarch64 (Cortex-A72) system.

Also, on the 7950X3D amd74 system, I see the behavior with
14.3-Stable. Apparently, this is not new with 15+. It has
been a long time since I'd tried using an amd64 system for
such activity based on using USB3 media. But it has been
common for me for aarch64 over that time frame.

previous to modern:
@@ -2659,8 +2659,7 @@ bwillwrite(void)
mtx_lock(&bdirtylock);
while (buf_dirty_count_severe()) {
bdirtywait = 1;
- msleep(&bdirtywait, &bdirtylock, (PRIBIO + 4),
- "flswai", 0);
+ msleep(&bdirtywait, &bdirtylock, PVFS, "flswai", 0);
}
mtx_unlock(&bdirtylock);
where in modern code:
/usr/src/sys/sys/priority.h:#define PRIBIO (PRI_MIN_KERN + 3)
and:
/usr/src/sys/sys/priority.h:#define PVFS (PRI_MIN_KERN + 4)
Presuming slpflags == 0 (see later reference code)
as an example context:
previous vs. now for . . .
bufspace_wait: (PRIBIO + 4) vs. (PRI_MIN_KERN + 3 + 4)
bwillwrite: (PRIBIO + 4) vs. (PRI_MIN_KERN + 4) // now a smallest, previously a largest (of the 3)
bufobj_wwait: (PRIBIO + 1) vs. (PRI_MIN_KERN + 3 + 1)
Suggestive summary, using routinenames to indicate values
(for slpflags == 0 contexts):
bufobj_wwait < bwillwrite == bufspace_wait (previous)
vs.
bufobj_wwait == bwillwrite < bufspace_wait (now)
QUOTE
Since RQ_PPQ is 4, differences of less than 4 are insignificant, so just
remove them. These small differences have not been changed for years,
so it is likely they have no real meaning (besides having no practical
effect). One can still consult the changes history to recover them if
ever needed.
END QUOTE
The above code does not "just remove them", but creates
new small differences and causes a change in the
relative order for the 3 contexts.
Any chance that is a potential performance problem for
some types of use now?
For reference:
diff --git a/sys/kern/vfs_bio.c b/sys/kern/vfs_bio.c
index ea66a81b5903..fa655c43d155 100644
--- a/sys/kern/vfs_bio.c
+++ b/sys/kern/vfs_bio.c
@@ -756,7 +756,7 @@ bufspace_wait(struct bufdomain *bd, struct vnode *vp, int gbflags,
break;
}
error = msleep(&bd->bd_wanted, BD_LOCKPTR(bd),
- (PRIBIO + 4) | slpflag, "newbuf", slptimeo);
+ PVFS | slpflag, "newbuf", slptimeo);
if (error != 0)
break;
}
@@ -2659,8 +2659,7 @@ bwillwrite(void)
mtx_lock(&bdirtylock);
while (buf_dirty_count_severe()) {
bdirtywait = 1;
- msleep(&bdirtywait, &bdirtylock, (PRIBIO + 4),
- "flswai", 0);
+ msleep(&bdirtywait, &bdirtylock, PVFS, "flswai", 0);
}
mtx_unlock(&bdirtylock);
}
@@ -5239,7 +5238,7 @@ bufobj_wwait(struct bufobj *bo, int slpflag, int timeo)
while (bo->bo_numoutput) {
bo->bo_flag |= BO_WWAIT;
error = msleep(&bo->bo_numoutput, BO_LOCKPTR(bo),
- slpflag | (PRIBIO + 1), "bo_wwait", timeo);
+ slpflag | PRIBIO, "bo_wwait", timeo);
if (error)
break;
}

. . .
29502 0 root 0 0 20976Ki 4832Ki biowr 2 0:00 0.06% cpdup -i0 -s0 -f -x ref 01
29479 0 root 0 0 20976Ki 4836Ki biord 7 0:01 0.75% cpdup -i0 -o ref 32
29467 0 root 0 0 20976Ki 4824Ki getblk 26 0:01 0.63% cpdup -i0 -o ref 31
29429 0 root 1 0 20976Ki 4768Ki flswai 23 0:01 0.61% cpdup -i0 -o ref 30
29414 0 root 0 0 20976Ki 4780Ki flswai 28 0:01 0.70% cpdup -i0 -o ref 29
29396 0 root 59 0 16872Ki 4948Ki wait 9 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29382 0 root 0 0 20976Ki 4876Ki flswai 15 0:01 0.95% cpdup -i0 -o ref 28
29353 0 root 59 0 16872Ki 4948Ki wait 7 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29326 0 root 0 0 20976Ki 4804Ki biord 5 0:01 0.44% cpdup -i0 -o ref 27
29308 0 root 0 0 20976Ki 4816Ki biord 2 0:01 0.77% cpdup -i0 -o ref 26
29307 0 root 59 0 16872Ki 4948Ki wait 5 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29273 0 root 59 0 16872Ki 4944Ki wait 4 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29262 0 root 0 0 20976Ki 4768Ki flswai 20 0:01 0.76% cpdup -i0 -o ref 25
29233 0 root 0 0 20976Ki 4784Ki flswai 3 0:01 0.71% cpdup -i0 -o ref 24
29232 0 root 59 0 16872Ki 4944Ki wait 6 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
. . .

It took over 6 minutes for the 32 builders to finish. (Normal is
under 2 seconds.)


Also, later activity gets extensive "rename" STATE time, again, at least >>> some of it during cpdup activity:

Looks like the "rename" state is new (and I've only
seen in for 15+):

/usr/src/sys/kern/vfs_mount.c has as the source
of the name used for the "rname" STATE:

static int
mount_init(void *mem, int size, int flags)
{
struct mount *mp;

mp = (struct mount *)mem;
mtx_init(&mp->mnt_mtx, "struct mount mtx", NULL, MTX_DEF);
mtx_init(&mp->mnt_listmtx, "struct mount vlist mtx", NULL, MTX_DEF);
lockinit(&mp->mnt_explock, PVFS, "explock", 0, 0);
lockinit(&mp->mnt_renamelock, PVFS, "rename", 0, 0);
mp->mnt_pcpu = uma_zalloc_pcpu(pcpu_zone_16, M_WAITOK | M_ZERO);
mp->mnt_ref = 0;
mp->mnt_vfs_ops = 1;
mp->mnt_rootvnode = NULL;
return (0);
}

author Konstantin Belousov <kib@FreeBSD.org> 2025-06-02 07:05:06 +0000

committer Konstantin Belousov <kib@FreeBSD.org> 2025-07-04 15:23:42 +0000 commit ef6ea91593ebff73e2fc201efd9f848b71c5a125 (patch)
tree 5f90f06e4b190a3a8fe9b5f3a016b69415d62a6d /sys/kern/vfs_mount.c
parent f9cf745a5084dc0a38733db8e774252e4ed2f444 (diff)

VOP_RENAME: add mp-global lock

It is before all vnode locks, but after vn_start_write().

The lock prevents parallel rename operations on the same mount point,
which should in (near future) simplify a lot of code in VFS/fs that
otherwise need to code with either the changing hierarchy, or with the
lock order for vnodes due to changed hierarchy.

On renames, the lock is taken on the lowest stacked filesystem.
Otherwise rename could still occur in parallel, by performing one of op
on the lower fs.

Proposed by: mjg (long time ago)
Reviewed by: markj, olce
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D50648
. . .

The context here is not renaming activity, so my
guess here is that the following may be in use
in a wider range of contexts.

/usr/src/sys/kern/vfs_lookup.c has:

static void
nameicap_cleanup(struct nameidata *ndp, int error)
{
struct nameicap_tracker *nt, *nt1;
struct mount *mp;

KASSERT((ndp->ni_nctrack_mnt == NULL &&
TAILQ_EMPTY(&ndp->ni_cap_tracker)) ||
(ndp->ni_lcf & NI_LCF_CAP_DOTDOT) != 0,
("tracker active and not strictrelative"));

TAILQ_FOREACH_SAFE(nt, &ndp->ni_cap_tracker, nm_link, nt1) {
mp = nt->mp;
lockmgr(&mp->mnt_renamelock, LK_RELEASE, 0);
vfs_rel(mp);
TAILQ_REMOVE(&ndp->ni_cap_tracker, nt, nm_link);
free(nt, M_NAMEITRACKER);
}

mp = ndp->ni_nctrack_mnt;
if (mp != NULL) {
if (error == ERESTART) {
lockmgr(&mp->mnt_renamelock, LK_EXCLUSIVE, 0);
lockmgr(&mp->mnt_renamelock, LK_RELEASE, 0);
}
vfs_rel(mp);
ndp->ni_nctrack_mnt = NULL;
}
}

and it may be the source of the large number
"rename" STATE processes that are all waiting
on "rename" below, forcing 1 of 32 at a time
for cpdup (32 poudriere builders). Later there
is an example of cc also waiting on it when
there is also cpdup activity waiting on it.

. . .
80055 0 root 59 0 16872Ki 4968Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][06]: build_pkg (boost-jam-1.88.0) (sh)
78347 0 root 0 0 20976Ki 4624Ki rename 0 0:00 1.05% cpdup -i0 -s0 -f -x ref 31
78331 0 root 59 0 16872Ki 4960Ki wait 18 0:00 0.00% sh: poudriere[official-amd64-default][31]: build_pkg (lua53-5.3.6_1) (sh)
77930 0 root 0 0 20976Ki 4636Ki rename 29 0:00 1.02% cpdup -i0 -s0 -f -x ref 30
77912 0 root 59 0 16872Ki 4960Ki wait 25 0:00 0.00% sh: poudriere[official-amd64-default][30]: build_pkg (libsunacl-1.0.1_1) (sh)
75398 0 root 0 0 20976Ki 4628Ki rename 2 0:00 0.99% cpdup -i0 -s0 -f -x ref 16
75335 0 root 59 0 16872Ki 4960Ki wait 20 0:00 0.00% sh: poudriere[official-amd64-default][16]: build_pkg (iwmbt-firmware-20250410) (sh)
75324 0 root 0 0 20976Ki 4628Ki rename 24 0:00 1.04% cpdup -i0 -s0 -f -x ref 19
75308 0 root 59 0 16872Ki 4960Ki wait 24 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (lua54-5.4.8) (sh)
74963 0 root 0 0 20976Ki 4624Ki rename 4 0:00 1.05% cpdup -i0 -s0 -f -x ref 32
74922 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][32]: build_pkg (lzo2-2.10_1) (sh)
72722 0 root 0 0 20976Ki 4776Ki rename 1 0:00 1.28% cpdup -i0 -s0 -f -x ref 15
72647 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][15]: build_pkg (indexinfo-0.3.1_1) (sh)
71967 0 root 0 0 20976Ki 4856Ki rename 30 0:00 1.48% cpdup -i0 -s0 -f -x ref 25
71917 0 root 59 0 16872Ki 4960Ki wait 1 0:00 0.00% sh: poudriere[official-amd64-default][25]: build_pkg (mpg123-1.33.2) (sh)
71629 0 root 0 0 20976Ki 4828Ki rename 31 0:00 1.61% cpdup -i0 -s0 -f
. . .

But I'll also see such on c compiles, ld commands, etc. I've
not seen rename for pkg-static but I have seen flswai for it.

The system spends lots of time 95%+ idle from the wait
activities.

I see such directly booted from the USB3 media (a 15.0-ALPHA4
context on UFS media) and when using that media via chroot
from both ZFS and UFS boots that are not USB based. The ZFS
and UFS boots do not show the behavior with the normal
non-USB3 media used instead.

The system in use is an AMD 7950X3D with 32 FreeBSD cpus,
192 GiBytes of RAM. main 16 booting for non-USB boots
and 15.0-ALPHA4 boots for the USB3 boots. kernel and
world are via official pkgbase distribution installs:
it is not a personal build of the kernel or world.


More rename and flswai examples:

. . .
39320 183 nobody 59 0 171676Ki 160576Ki rename 29 0:00 2.17% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3372Ki wait 25 0:00 0.04% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 10 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39327 179 nobody 59 0 105680Ki 93264Ki biowr 2 0:00 0.61% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o test /usr/lib/crt
39325 179 nobody 59 0 165928Ki 155088Ki wait 17 0:00 1.00% /usr/bin/cc -o test ../test.o
39294 179 nobody 59 0 14744Ki 3448Ki wait 10 0:00 0.01% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki rename 22 0:03 0.47% cpdup -i0 -s0 -f -x ref 26
32420 0 root 0 0 20976Ki 5024Ki rename 13 0:03 0.50% cpdup -i0 -s0 -f -x ref 11
32358 0 root 0 0 20976Ki 5068Ki rename 26 0:03 0.53% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki 4960Ki wait 9 0:00 0.00% sh: poudriere[official-amd64-default][20]: build_pkg (linux-c7-libaio-0.3.109) (sh)
32301 0 root 59 0 16872Ki 4960Ki wait 19 0:00 0.00% sh: poudriere[official-amd64-default][11]: build_pkg (linux-c7-dosfstools-3.0.20) (sh)
13808 0 root 0 0 20976Ki 4984Ki rename 4 0:03 0.57% cpdup -i0 -s0 -f -x ref 19
12316 0 root 0 0 20976Ki 5072Ki biowr 27 0:03 0.56% cpdup -i0 -s0 -f -x ref 13
12283 0 root 0 0 20976Ki 5068Ki rename 5 0:03 0.53% cpdup -i0 -s0 -f -x ref 01
12277 0 root 0 0 20976Ki 5068Ki rename 28 0:03 0.54% cpdup -i0 -s0 -f -x ref 14
12096 0 root 59 0 16872Ki 4960Ki wait 27 0:00 0.00% sh: poudriere[official-amd64-def
. . .


. . .
39927 183 nobody 59 0 171676Ki 160396Ki flswai 13 0:00 2.03% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3400Ki wait 3 0:00 0.06% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 28 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39924 179 nobody 59 0 125140Ki 95572Ki flswai 20 0:00 0.60% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 0 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 13 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 16 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 27 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39923 179 nobody 59 0 165928Ki 155092Ki wait 0 0:00 1.00% /usr/bin/cc -o dynamic.exe -dynamic dynamic.o
39294 179 nobody 59 0 14744Ki 3464Ki wait 9 0:00 0.02% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki flswai 17 0:03 2.82% cpdup -i0 -s0 -f -x ref 26
32420 0 root 1 0 20976Ki 5024Ki flswai 25 0:03 2.81% cpdup -i0 -s0 -f -x ref 11
32358 0 root 1 0 20976Ki 5068Ki flswai 14 0:03 2.79% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki
. . .


. . .
53486 185 root 0 0 66428Ki 34196Ki getblk 25 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53485 185 root 59 0 23308Ki 12024Ki wait 0 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53446 185 root 59 0 14744Ki 3504Ki wait 11 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
53445 185 root 59 0 14284Ki 3828Ki wait 1 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/p5-YAML-Tiny build-depends
96359 183 root 0 0 66428Ki 34196Ki flswai 31 0:00 1.43% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96358 183 root 59 0 23308Ki 12020Ki wait 29 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96331 183 root 59 0 14744Ki 3504Ki wait 5 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
96330 183 root 59 0 14284Ki 3860Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/security/cyrus-sasl2 patch-depends
9658 170 nobody 59 0 14224Ki 2852Ki flswai 3 0:00 0.02% install -m 0644 ./libcdio.info /wrkdirs/usr/ports/sysutils/libcdio/work/stage/usr/local/share/info
9652 170 nobody 59 0 14744Ki 3420Ki wait 2 0:00 0.00% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9643 170 nobody 59 0 14744Ki 3420Ki wait 8 0:00 0.03% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9642 170 nobody 59 0 14776Ki 3652Ki wait 21 0:00 0.04% /usr/local/bin/gmake install-exec-am install-data-am
9641 170 nobody 59 0 14776Ki 3632Ki wait 3 0:00 0.04% /usr/local/bin/gmake install
9636 170 nobody 59 0 14744Ki 3424Ki wait 15 0:00 0.03% /bin/sh -c fail=; \\\nif (target_option=k; case ${target_option-} in ?) ;; *) echo "am__make_running_with_option: interna
9635 170 nobody 59 0 14776Ki 3708Ki wait 0 0:00 0.06% /usr/local/bin/gmake INSTALL_PROGRAM=/bin/sh /wrkdirs/usr/ports/sysutils/libcdio/work/libcdio-2.2.0/install-sh -c -s inst
9634 170 nobody 59 0 14776Ki 3680Ki wait 29 0:00 0.08% /usr/local/bin/gmake -f Makefile DESTDIR=/wrkdirs/usr/ports/sysutils/libcdio/work/stage install-strip
9633 170 nobody 59 0 14744Ki 3428Ki wait 13 0:00 0.02% /bin/sh -e /tmp/makeOENiKj
9611 170 nobody 59 0 16848Ki 3876Ki wait 21 0:00 0.22% /usr/bin/make -C /usr/ports/sysutils/libcdio stage
51574 158 root 0 0 74620Ki 34184Ki flswai 20 0:01 0.28% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51573 158 root 59 0 23308Ki 12020Ki wait 20 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51547 158 root 59 0 14744Ki 3512Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
51545 158 root 59 0 14284Ki 3852Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/hyphen build-depends
56591 157 root 0 0 66428Ki 34184Ki getblk 17 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56590 157 root 59 0 23308Ki 12024Ki wait 12 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56528 157 root 59 0 14744Ki 3504Ki wait 28 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56469 157 root 59 0 14284Ki 3832Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-MIME-Base32 build-depends
56593 155 root 0 0 66428Ki 34188Ki getblk 8 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56592 155 root 59 0 23308Ki 12024Ki wait 4 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56559 155 root 59 0 14744Ki 3508Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56532 155 root 59 0 14284Ki 3828Ki wait 13 0:00 0.00% /usr/bin/make -C /usr/ports/devel/p5-Parse-Yapp build-depends
52781 145 root 0 0 74620Ki 34244Ki flswai 12 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52780 145 root 59 0 23308Ki 12024Ki wait 22 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52753 145 root 59 0 14744Ki 3508Ki wait 24 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
52749 145 root 59 0 14284Ki 3824Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-JSON build-depends
9610 0 root 59 0 16872Ki 4972Ki wait 14 0:00 0.01% sh: poudriere[official-amd64-default][14]: build_pkg (libcdio-2.2.0) (sh)
99560 0 root 1 0 20976Ki 4860Ki flswai 15 0:00 2.82% cpdup -i0 -s0 -f -x ref 01
99539 0 root 59 0 16872Ki 4960Ki wait 10 0:00 0.00% sh: poudriere[official-amd64-default][01]: build_pkg (texi2html-5.0_5,1) (sh)
98112 0 root 1 0 20976Ki 4828Ki flswai 19 0:00 3.47% cpdup -i0 -s0 -f -x ref 28
98096 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][28]: build_pkg (gdbm-1.26) (sh)
97694 0 root 1 0 20976Ki 4844Ki flswai 7 0:01 2.51% cpdup -i0 -s0 -f -x ref 27
97665 0 root 59 0 16872Ki 4960Ki wait 0 0:00 0.00% sh: poudriere[official-amd64-default][27]: build_pkg (sudo-1.9.17p2) (sh)
96108 0 root 0 0 20976Ki 4800Ki flswai 29 0:01 3.06% cpdup -i0 -s0 -f -x ref 19
96091 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (libgpg-error-1.55) (sh)
93884 0 root 0 0 20976Ki 5064Ki flswai 19 0:01 2.43% cpdup -i0 -s0 -f -x ref 26
93857 0 root 59 0 16872Ki 4960Ki wait 16 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (libxml2-2.14.5) (sh)
93271 0 root 0 0 20976Ki 5224Ki flswai 5 0:01 2.38% cpdup -i0 -s0 -f -x ref 12
93212 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][12]: build_pkg (lame-3.100_5) (sh)
88119 0 root 0 0 20976Ki 5212Ki flswai 0 0:01 2.51% cpdup -i0 -s0 -f -x ref 32
88077 0 root 0 0 20976Ki 5176Ki flswai 0 0:01 2.38% cpdup -i0 -s0 -f -x ref 31
. . .

===
Mark Millard
marklmi at yahoo.com
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

[Resending: Correcting an Email address copy/paste error
and adding adjusting a little wording.]
[This note's additions are tied to "flswai" STATE instead
of to "rename" STATE.]
On Sep 30, 2025, at 20:43, Mark Millard <marklmi@yahoo.com> wrote:

[The new material here ends up being about nameicap_cleanup
and its exclusive use of mnt_renamelock being one potential
bottleneck involved here. I make no claim it has anything to
do with the flswai activity reported. The possible
bottleneck is an observation, not something that I claim
there is any alternative to. I do not know if this is of any
interest or not.]

On Sep 29, 2025, at 16:06, Mark Millard <marklmi@yahoo.com> wrote:

On Sep 29, 2025, at 13:01, Mark Millard <marklmi@yahoo.com> wrote:

An example is during the cpdup activities when multiple happen
in overlappingtime frames:

I'll note that I see this on the amd64 32-FreeBSD-cpu system
but not on the aarch64 8-FreeBSD-cpu Windows Dev Kit 2023
system. May be at some point I'll try the older 16-FreeBSD-cpu
aarch64 (Cortex-A72) system.

Also, on the 7950X3D amd74 system, I see the behavior with
14.3-Stable. Apparently, this is not new with 15+. It has
been a long time since I'd tried using an amd64 system for
such activity based on using USB3 media. But it has been
common for me for aarch64 over that time frame.

previous to modern:
@@ -2659,8 +2659,7 @@ bwillwrite(void)
mtx_lock(&bdirtylock);
while (buf_dirty_count_severe()) {
bdirtywait = 1;
- msleep(&bdirtywait, &bdirtylock, (PRIBIO + 4),
- "flswai", 0);
+ msleep(&bdirtywait, &bdirtylock, PVFS, "flswai", 0);
}
mtx_unlock(&bdirtylock);
where in modern code:
/usr/src/sys/sys/priority.h:#define PRIBIO (PRI_MIN_KERN + 3)
and:
/usr/src/sys/sys/priority.h:#define PVFS (PRI_MIN_KERN + 4)
Presuming slpflags == 0 (see later reference code)
as an example context:
previous vs. now for . . .
bufspace_wait: (PRIBIO + 4) vs. (PRI_MIN_KERN + 3 + 4)
bwillwrite: (PRIBIO + 4) vs. (PRI_MIN_KERN + 4) // now a smallest, previously a largest (of the 3)
bufobj_wwait: (PRIBIO + 1) vs. (PRI_MIN_KERN + 3 + 1)
Suggestive summary, using routine names to indicate values
(for slpflags == 0 contexts):
bufobj_wwait < bwillwrite == bufspace_wait (previous)
vs.
bufobj_wwait == bwillwrite < bufspace_wait (now)
QUOTE
Since RQ_PPQ is 4, differences of less than 4 are insignificant, so just
remove them. These small differences have not been changed for years,
so it is likely they have no real meaning (besides having no practical
effect). One can still consult the changes history to recover them if
ever needed.
END QUOTE
The above code does not "just remove them", but creates
new small differences and causes a change in the
relative order for the 3 contexts.
Any chance that is a potential performance problem for
some types of use now?
For reference:
diff --git a/sys/kern/vfs_bio.c b/sys/kern/vfs_bio.c
index ea66a81b5903..fa655c43d155 100644
--- a/sys/kern/vfs_bio.c
+++ b/sys/kern/vfs_bio.c
@@ -756,7 +756,7 @@ bufspace_wait(struct bufdomain *bd, struct vnode *vp, int gbflags,
break;
}
error = msleep(&bd->bd_wanted, BD_LOCKPTR(bd),
- (PRIBIO + 4) | slpflag, "newbuf", slptimeo);
+ PVFS | slpflag, "newbuf", slptimeo);
if (error != 0)
break;
}
@@ -2659,8 +2659,7 @@ bwillwrite(void)
mtx_lock(&bdirtylock);
while (buf_dirty_count_severe()) {
bdirtywait = 1;
- msleep(&bdirtywait, &bdirtylock, (PRIBIO + 4),
- "flswai", 0);
+ msleep(&bdirtywait, &bdirtylock, PVFS, "flswai", 0);
}
mtx_unlock(&bdirtylock);
}
@@ -5239,7 +5238,7 @@ bufobj_wwait(struct bufobj *bo, int slpflag, int timeo)
while (bo->bo_numoutput) {
bo->bo_flag |= BO_WWAIT;
error = msleep(&bo->bo_numoutput, BO_LOCKPTR(bo),
- slpflag | (PRIBIO + 1), "bo_wwait", timeo);
+ slpflag | PRIBIO, "bo_wwait", timeo);
if (error)
break;
}

. . .
29502 0 root 0 0 20976Ki 4832Ki biowr 2 0:00 0.06% cpdup -i0 -s0 -f -x ref 01
29479 0 root 0 0 20976Ki 4836Ki biord 7 0:01 0.75% cpdup -i0 -o ref 32
29467 0 root 0 0 20976Ki 4824Ki getblk 26 0:01 0.63% cpdup -i0 -o ref 31
29429 0 root 1 0 20976Ki 4768Ki flswai 23 0:01 0.61% cpdup -i0 -o ref 30
29414 0 root 0 0 20976Ki 4780Ki flswai 28 0:01 0.70% cpdup -i0 -o ref 29
29396 0 root 59 0 16872Ki 4948Ki wait 9 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29382 0 root 0 0 20976Ki 4876Ki flswai 15 0:01 0.95% cpdup -i0 -o ref 28
29353 0 root 59 0 16872Ki 4948Ki wait 7 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29326 0 root 0 0 20976Ki 4804Ki biord 5 0:01 0.44% cpdup -i0 -o ref 27
29308 0 root 0 0 20976Ki 4816Ki biord 2 0:01 0.77% cpdup -i0 -o ref 26
29307 0 root 59 0 16872Ki 4948Ki wait 5 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29273 0 root 59 0 16872Ki 4944Ki wait 4 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
29262 0 root 0 0 20976Ki 4768Ki flswai 20 0:01 0.76% cpdup -i0 -o ref 25
29233 0 root 0 0 20976Ki 4784Ki flswai 3 0:01 0.71% cpdup -i0 -o ref 24
29232 0 root 59 0 16872Ki 4944Ki wait 6 0:00 0.00% /usr/local/libexec/poudriere/sh -e -o pipefail /usr/local/share/poudriere/bulk.sh -jofficial-amd64 -c -f /usr/home/root/o
. . .

It took over 6 minutes for the 32 builders to finish. (Normal is
under 2 seconds.)


Also, later activity gets extensive "rename" STATE time, again, at least >>> some of it during cpdup activity:

Looks like the "rename" state is new (and I've only
seen in for 15+):

/usr/src/sys/kern/vfs_mount.c has as the source
of the name used for the "rname" STATE:

static int
mount_init(void *mem, int size, int flags)
{
struct mount *mp;

mp = (struct mount *)mem;
mtx_init(&mp->mnt_mtx, "struct mount mtx", NULL, MTX_DEF);
mtx_init(&mp->mnt_listmtx, "struct mount vlist mtx", NULL, MTX_DEF);
lockinit(&mp->mnt_explock, PVFS, "explock", 0, 0);
lockinit(&mp->mnt_renamelock, PVFS, "rename", 0, 0);
mp->mnt_pcpu = uma_zalloc_pcpu(pcpu_zone_16, M_WAITOK | M_ZERO);
mp->mnt_ref = 0;
mp->mnt_vfs_ops = 1;
mp->mnt_rootvnode = NULL;
return (0);
}

author Konstantin Belousov <kib@FreeBSD.org> 2025-06-02 07:05:06 +0000

committer Konstantin Belousov <kib@FreeBSD.org> 2025-07-04 15:23:42 +0000 commit ef6ea91593ebff73e2fc201efd9f848b71c5a125 (patch)
tree 5f90f06e4b190a3a8fe9b5f3a016b69415d62a6d /sys/kern/vfs_mount.c
parent f9cf745a5084dc0a38733db8e774252e4ed2f444 (diff)

VOP_RENAME: add mp-global lock

It is before all vnode locks, but after vn_start_write().

The lock prevents parallel rename operations on the same mount point,
which should in (near future) simplify a lot of code in VFS/fs that
otherwise need to code with either the changing hierarchy, or with the
lock order for vnodes due to changed hierarchy.

On renames, the lock is taken on the lowest stacked filesystem.
Otherwise rename could still occur in parallel, by performing one of op
on the lower fs.

Proposed by: mjg (long time ago)
Reviewed by: markj, olce
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D50648
. . .

The context here is not renaming activity, so my
guess here is that the following may be in use
in a wider range of contexts.

/usr/src/sys/kern/vfs_lookup.c has:

static void
nameicap_cleanup(struct nameidata *ndp, int error)
{
struct nameicap_tracker *nt, *nt1;
struct mount *mp;

KASSERT((ndp->ni_nctrack_mnt == NULL &&
TAILQ_EMPTY(&ndp->ni_cap_tracker)) ||
(ndp->ni_lcf & NI_LCF_CAP_DOTDOT) != 0,
("tracker active and not strictrelative"));

TAILQ_FOREACH_SAFE(nt, &ndp->ni_cap_tracker, nm_link, nt1) {
mp = nt->mp;
lockmgr(&mp->mnt_renamelock, LK_RELEASE, 0);
vfs_rel(mp);
TAILQ_REMOVE(&ndp->ni_cap_tracker, nt, nm_link);
free(nt, M_NAMEITRACKER);
}

mp = ndp->ni_nctrack_mnt;
if (mp != NULL) {
if (error == ERESTART) {
lockmgr(&mp->mnt_renamelock, LK_EXCLUSIVE, 0);
lockmgr(&mp->mnt_renamelock, LK_RELEASE, 0);
}
vfs_rel(mp);
ndp->ni_nctrack_mnt = NULL;
}
}

and it may be the source of the large number
"rename" STATE processes that are all waiting
on "rename" below, forcing 1 of 32 at a time
for cpdup (32 poudriere builders). Later there
is an example of cc also waiting on it when
there is also cpdup activity waiting on it.

. . .
80055 0 root 59 0 16872Ki 4968Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][06]: build_pkg (boost-jam-1.88.0) (sh)
78347 0 root 0 0 20976Ki 4624Ki rename 0 0:00 1.05% cpdup -i0 -s0 -f -x ref 31
78331 0 root 59 0 16872Ki 4960Ki wait 18 0:00 0.00% sh: poudriere[official-amd64-default][31]: build_pkg (lua53-5.3.6_1) (sh)
77930 0 root 0 0 20976Ki 4636Ki rename 29 0:00 1.02% cpdup -i0 -s0 -f -x ref 30
77912 0 root 59 0 16872Ki 4960Ki wait 25 0:00 0.00% sh: poudriere[official-amd64-default][30]: build_pkg (libsunacl-1.0.1_1) (sh)
75398 0 root 0 0 20976Ki 4628Ki rename 2 0:00 0.99% cpdup -i0 -s0 -f -x ref 16
75335 0 root 59 0 16872Ki 4960Ki wait 20 0:00 0.00% sh: poudriere[official-amd64-default][16]: build_pkg (iwmbt-firmware-20250410) (sh)
75324 0 root 0 0 20976Ki 4628Ki rename 24 0:00 1.04% cpdup -i0 -s0 -f -x ref 19
75308 0 root 59 0 16872Ki 4960Ki wait 24 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (lua54-5.4.8) (sh)
74963 0 root 0 0 20976Ki 4624Ki rename 4 0:00 1.05% cpdup -i0 -s0 -f -x ref 32
74922 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][32]: build_pkg (lzo2-2.10_1) (sh)
72722 0 root 0 0 20976Ki 4776Ki rename 1 0:00 1.28% cpdup -i0 -s0 -f -x ref 15
72647 0 root 59 0 16872Ki 4960Ki wait 26 0:00 0.00% sh: poudriere[official-amd64-default][15]: build_pkg (indexinfo-0.3.1_1) (sh)
71967 0 root 0 0 20976Ki 4856Ki rename 30 0:00 1.48% cpdup -i0 -s0 -f -x ref 25
71917 0 root 59 0 16872Ki 4960Ki wait 1 0:00 0.00% sh: poudriere[official-amd64-default][25]: build_pkg (mpg123-1.33.2) (sh)
71629 0 root 0 0 20976Ki 4828Ki rename 31 0:00 1.61% cpdup -i0 -s0 -f
. . .

But I'll also see such on c compiles, ld commands, etc. I've
not seen rename for pkg-static but I have seen flswai for it.

The system spends lots of time 95%+ idle from the wait
activities.

I see such directly booted from the USB3 media (a 15.0-ALPHA4
context on UFS media) and when using that media via chroot
from both ZFS and UFS boots that are not USB based. The ZFS
and UFS boots do not show the behavior with the normal
non-USB3 media used instead.

The system in use is an AMD 7950X3D with 32 FreeBSD cpus,
192 GiBytes of RAM. main 16 booting for non-USB boots
and 15.0-ALPHA4 boots for the USB3 boots. kernel and
world are via official pkgbase distribution installs:
it is not a personal build of the kernel or world.


More rename and flswai examples:

. . .
39320 183 nobody 59 0 171676Ki 160576Ki rename 29 0:00 2.17% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3372Ki wait 25 0:00 0.04% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 10 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39327 179 nobody 59 0 105680Ki 93264Ki biowr 2 0:00 0.61% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o test /usr/lib/crt
39325 179 nobody 59 0 165928Ki 155088Ki wait 17 0:00 1.00% /usr/bin/cc -o test ../test.o
39294 179 nobody 59 0 14744Ki 3448Ki wait 10 0:00 0.01% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki rename 22 0:03 0.47% cpdup -i0 -s0 -f -x ref 26
32420 0 root 0 0 20976Ki 5024Ki rename 13 0:03 0.50% cpdup -i0 -s0 -f -x ref 11
32358 0 root 0 0 20976Ki 5068Ki rename 26 0:03 0.53% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki 4960Ki wait 9 0:00 0.00% sh: poudriere[official-amd64-default][20]: build_pkg (linux-c7-libaio-0.3.109) (sh)
32301 0 root 59 0 16872Ki 4960Ki wait 19 0:00 0.00% sh: poudriere[official-amd64-default][11]: build_pkg (linux-c7-dosfstools-3.0.20) (sh)
13808 0 root 0 0 20976Ki 4984Ki rename 4 0:03 0.57% cpdup -i0 -s0 -f -x ref 19
12316 0 root 0 0 20976Ki 5072Ki biowr 27 0:03 0.56% cpdup -i0 -s0 -f -x ref 13
12283 0 root 0 0 20976Ki 5068Ki rename 5 0:03 0.53% cpdup -i0 -s0 -f -x ref 01
12277 0 root 0 0 20976Ki 5068Ki rename 28 0:03 0.54% cpdup -i0 -s0 -f -x ref 14
12096 0 root 59 0 16872Ki 4960Ki wait 27 0:00 0.00% sh: poudriere[official-amd64-def
. . .


. . .
39927 183 nobody 59 0 171676Ki 160396Ki flswai 13 0:00 2.03% cc -D_BLD_DLL -fPIC -D_BLD_ast -Os -Wno-unused-value -Wno-parentheses -Wno-logical-op-parentheses -fno-strict-aliasing -I
62772 183 nobody 59 0 14376Ki 3400Ki wait 3 0:00 0.06% mamake -C lib/libast install
12602 183 nobody 59 0 14376Ki 3072Ki wait 17 0:00 0.00% mamake -r */* install
12600 183 nobody 59 0 14376Ki 3076Ki wait 22 0:00 0.00% mamake install
12596 183 nobody 0 0 14068Ki 2688Ki piperd 28 0:00 0.01% tee -a /wrkdirs/usr/ports/shells/ksh/work/ksh-1.0.10/arch/freebsd15.i386-64/lib/package/gen/make.out
8980 183 nobody 59 0 14744Ki 3732Ki wait 6 0:00 0.00% /bin/sh bin/package flat make
8977 183 nobody 59 0 14744Ki 3436Ki wait 3 0:00 0.00% /bin/sh -e /tmp/makelMk4B8
8929 183 nobody 59 0 14288Ki 3872Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh build
39924 179 nobody 59 0 125140Ki 95572Ki flswai 20 0:00 0.60% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 0 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 13 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 16 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39924 179 nobody 59 0 125140Ki 95572Ki uwait 27 0:00 0.01% /usr/bin/ld --eh-frame-hdr -dynamic-linker /libexec/ld-elf.so.1 --hash-style=both --enable-new-dtags -o dynamic.exe /usr/
39923 179 nobody 59 0 165928Ki 155092Ki wait 0 0:00 1.00% /usr/bin/cc -o dynamic.exe -dynamic dynamic.o
39294 179 nobody 59 0 14744Ki 3464Ki wait 9 0:00 0.02% sh /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/C/make/probe /usr/bin/cc PREROOT='' UNIVER
39293 179 nobody 59 0 15264Ki 3704Ki piperd 19 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/probe/probe C make /usr/bin/cc
39291 179 nobody 59 0 16344Ki 4960Ki wait 10 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake --ignorelock --keepgoing --errorid=lib/libast
39290 179 nobody 59 0 14744Ki 3444Ki wait 12 0:00 0.00% sh /dev/fd/3
39258 179 nobody 59 0 14744Ki 3452Ki piperd 0 0:00 0.00% sh /dev/fd/3
37014 179 nobody 0 0 14068Ki 2692Ki piperd 30 0:00 0.00% tee -a /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/lib/package/gen/make.out
37013 179 nobody 0 0 16728Ki 5320Ki select 14 0:00 0.00% /wrkdirs/usr/ports/shells/ksh93/work/ast-93u/arch/freebsd16.amd64/bin/nmake -k -K recurse believe nmake
32949 179 nobody 13 0 14744Ki 4136Ki wait 31 0:00 0.00% /bin/sh bin/package flat make ksh93
32948 179 nobody 59 0 14744Ki 3444Ki wait 19 0:00 0.00% /bin/sh -e /tmp/makeW36tmB
32940 179 nobody 59 0 14288Ki 3892Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/shells/ksh93 build
32453 0 root 0 0 20976Ki 5020Ki flswai 17 0:03 2.82% cpdup -i0 -s0 -f -x ref 26
32420 0 root 1 0 20976Ki 5024Ki flswai 25 0:03 2.81% cpdup -i0 -s0 -f -x ref 11
32358 0 root 1 0 20976Ki 5068Ki flswai 14 0:03 2.79% cpdup -i0 -s0 -f -x ref 20
32333 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (linux-c7-numactl-libs-2.0.12_1) (sh)
32313 0 root 59 0 16872Ki
. . .


. . .
53486 185 root 0 0 66428Ki 34196Ki getblk 25 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53485 185 root 59 0 23308Ki 12024Ki wait 0 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
53446 185 root 59 0 14744Ki 3504Ki wait 11 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
53445 185 root 59 0 14284Ki 3828Ki wait 1 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/p5-YAML-Tiny build-depends
96359 183 root 0 0 66428Ki 34196Ki flswai 31 0:00 1.43% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96358 183 root 59 0 23308Ki 12020Ki wait 29 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
96331 183 root 59 0 14744Ki 3504Ki wait 5 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
96330 183 root 59 0 14284Ki 3860Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/security/cyrus-sasl2 patch-depends
9658 170 nobody 59 0 14224Ki 2852Ki flswai 3 0:00 0.02% install -m 0644 ./libcdio.info /wrkdirs/usr/ports/sysutils/libcdio/work/stage/usr/local/share/info
9652 170 nobody 59 0 14744Ki 3420Ki wait 2 0:00 0.00% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9643 170 nobody 59 0 14744Ki 3420Ki wait 8 0:00 0.03% /bin/sh -c srcdirstrip=`echo "." | sed 's|.|.|g'`; \\\nlist='./libcdio.info'; test -n "/usr/local/share/info" || list=; \
9642 170 nobody 59 0 14776Ki 3652Ki wait 21 0:00 0.04% /usr/local/bin/gmake install-exec-am install-data-am
9641 170 nobody 59 0 14776Ki 3632Ki wait 3 0:00 0.04% /usr/local/bin/gmake install
9636 170 nobody 59 0 14744Ki 3424Ki wait 15 0:00 0.03% /bin/sh -c fail=; \\\nif (target_option=k; case ${target_option-} in ?) ;; *) echo "am__make_running_with_option: interna
9635 170 nobody 59 0 14776Ki 3708Ki wait 0 0:00 0.06% /usr/local/bin/gmake INSTALL_PROGRAM=/bin/sh /wrkdirs/usr/ports/sysutils/libcdio/work/libcdio-2.2.0/install-sh -c -s inst
9634 170 nobody 59 0 14776Ki 3680Ki wait 29 0:00 0.08% /usr/local/bin/gmake -f Makefile DESTDIR=/wrkdirs/usr/ports/sysutils/libcdio/work/stage install-strip
9633 170 nobody 59 0 14744Ki 3428Ki wait 13 0:00 0.02% /bin/sh -e /tmp/makeOENiKj
9611 170 nobody 59 0 16848Ki 3876Ki wait 21 0:00 0.22% /usr/bin/make -C /usr/ports/sysutils/libcdio stage
51574 158 root 0 0 74620Ki 34184Ki flswai 20 0:01 0.28% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51573 158 root 59 0 23308Ki 12020Ki wait 20 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
51547 158 root 59 0 14744Ki 3512Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
51545 158 root 59 0 14284Ki 3852Ki wait 27 0:00 0.00% /usr/bin/make -C /usr/ports/textproc/hyphen build-depends
56591 157 root 0 0 66428Ki 34184Ki getblk 17 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56590 157 root 59 0 23308Ki 12024Ki wait 12 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56528 157 root 59 0 14744Ki 3504Ki wait 28 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56469 157 root 59 0 14284Ki 3832Ki wait 4 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-MIME-Base32 build-depends
56593 155 root 0 0 66428Ki 34188Ki getblk 8 0:01 0.26% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56592 155 root 59 0 23308Ki 12024Ki wait 4 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
56559 155 root 59 0 14744Ki 3508Ki wait 18 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
56532 155 root 59 0 14284Ki 3828Ki wait 13 0:00 0.00% /usr/bin/make -C /usr/ports/devel/p5-Parse-Yapp build-depends
52781 145 root 0 0 74620Ki 34244Ki flswai 12 0:01 0.25% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52780 145 root 59 0 23308Ki 12024Ki wait 22 0:00 0.00% /usr/local/sbin/pkg-static add -A /packages/All/perl5-5.42.0_1.pkg
52753 145 root 59 0 14744Ki 3508Ki wait 24 0:00 0.00% /bin/sh /usr/ports/Mk/Scripts/do-depends.sh
52749 145 root 59 0 14284Ki 3824Ki wait 21 0:00 0.00% /usr/bin/make -C /usr/ports/converters/p5-JSON build-depends
9610 0 root 59 0 16872Ki 4972Ki wait 14 0:00 0.01% sh: poudriere[official-amd64-default][14]: build_pkg (libcdio-2.2.0) (sh)
99560 0 root 1 0 20976Ki 4860Ki flswai 15 0:00 2.82% cpdup -i0 -s0 -f -x ref 01
99539 0 root 59 0 16872Ki 4960Ki wait 10 0:00 0.00% sh: poudriere[official-amd64-default][01]: build_pkg (texi2html-5.0_5,1) (sh)
98112 0 root 1 0 20976Ki 4828Ki flswai 19 0:00 3.47% cpdup -i0 -s0 -f -x ref 28
98096 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][28]: build_pkg (gdbm-1.26) (sh)
97694 0 root 1 0 20976Ki 4844Ki flswai 7 0:01 2.51% cpdup -i0 -s0 -f -x ref 27
97665 0 root 59 0 16872Ki 4960Ki wait 0 0:00 0.00% sh: poudriere[official-amd64-default][27]: build_pkg (sudo-1.9.17p2) (sh)
96108 0 root 0 0 20976Ki 4800Ki flswai 29 0:01 3.06% cpdup -i0 -s0 -f -x ref 19
96091 0 root 59 0 16872Ki 4960Ki wait 31 0:00 0.00% sh: poudriere[official-amd64-default][19]: build_pkg (libgpg-error-1.55) (sh)
93884 0 root 0 0 20976Ki 5064Ki flswai 19 0:01 2.43% cpdup -i0 -s0 -f -x ref 26
93857 0 root 59 0 16872Ki 4960Ki wait 16 0:00 0.00% sh: poudriere[official-amd64-default][26]: build_pkg (libxml2-2.14.5) (sh)
93271 0 root 0 0 20976Ki 5224Ki flswai 5 0:01 2.38% cpdup -i0 -s0 -f -x ref 12
93212 0 root 59 0 16872Ki 4960Ki wait 11 0:00 0.00% sh: poudriere[official-amd64-default][12]: build_pkg (lame-3.100_5) (sh)
88119 0 root 0 0 20976Ki 5212Ki flswai 0 0:01 2.51% cpdup -i0 -s0 -f -x ref 32
88077 0 root 0 0 20976Ki 5176Ki flswai 0 0:01 2.38% cpdup -i0 -s0 -f -x ref 31
. . .

===
Mark Millard
marklmi at yahoo.com
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

On Sep 30, 2025, at 20:43, Mark Millard <marklmi@yahoo.com> wrote:

[The new material here ends up being about nameicap_cleanup
and its exclusive use of mnt_renamelock being one potential
bottleneck involved here. I make no claim it has anything to
do with the flswai activity reported. The possible
bottleneck is an observation, not something that I claim
there is any alternative to. I do not know if this is of any
interest or not.]

On Sep 29, 2025, at 16:06, Mark Millard <marklmi@yahoo.com> wrote:

On Sep 29, 2025, at 13:01, Mark Millard <marklmi@yahoo.com> wrote:

An example is during the cpdup activities when multiple happen
in overlappingtime frames:

I'll note that I see this on the amd64 32-FreeBSD-cpu system
but not on the aarch64 8-FreeBSD-cpu Windows Dev Kit 2023
system. May be at some point I'll try the older 16-FreeBSD-cpu
aarch64 (Cortex-A72) system.

Also, on the 7950X3D amd74 system, I see the behavior with
14.3-Stable. Apparently, this is not new with 15+. It has
been a long time since I'd tried using an amd64 system for
such activity based on using USB3 media. But it has been
common for me for aarch64 over that time frame.

. . .
. . .

. . .

. . .
. . .

But I'll also see such on c compiles, ld commands, etc. I've
not seen rename for pkg-static but I have seen flswai for it.

The system spends lots of time 95%+ idle from the wait
activities.

I see such directly booted from the USB3 media (a 15.0-ALPHA4
context on UFS media) and when using that media via chroot
from both ZFS and UFS boots that are not USB based. The ZFS
and UFS boots do not show the behavior with the normal
non-USB3 media used instead.

The system in use is an AMD 7950X3D with 32 FreeBSD cpus,
192 GiBytes of RAM. main 16 booting for non-USB boots
and 15.0-ALPHA4 boots for the USB3 boots. kernel and
world are via official pkgbase distribution installs:
it is not a personal build of the kernel or world.


. . .
. . .

I got a test context were I could compare the same
media used on the same USB4 port on a laptop
(Dell Precision 5490, 22 FreeBSD cpus, 32 GiBytes
RAM, 4 USB4 ports), where, on boot, the media ends
up being handled as either:
) "nda0 at nvme0" (via involving a Thunderbolt 3 hub)
) "da0" (via a direct connection)
(The UEFI/ACPI does enough to make basic operation
work, presenting some view to FreeBSD for media on
USB4 ports.)
Both have the bottlenecks visible when monitored with
top, both "flswai" and "rename" examples occur in
both contexts.
But "nda0 at nvme0" bottleneck periods do not last
nearly as long as "da0" bottleneck periods do, making
"nda0 at nvme0" use much more reasonable for the
type of activity.
Still, this eliminates the possibility that the issue
was limited to USB. It also eliminates it being
specific to the prior AMD (7950X3D) test context.
For reference:
# uname -apKU
FreeBSD USB4sys 16.0-CURRENT FreeBSD 16.0-CURRENT main-n280801-213170eb956f GENERIC-NODEBUG amd64 amd64 1600001 1600001
(It is from official pkgbase distribution use, a
copy of another boot media with some parameters
replaced afterwards.)
QUOTE
CPU: Intel(R) Core(TM) Ultra 7 165H (3072.00-MHz K8-class CPU)
Origin="GenuineIntel" Id=0xa06a4 Family=0x6 Model=0xaa Stepping=4
. . .
WARNING: L3 data cache covers more APIC IDs than a package (6 > 3)
FreeBSD/SMP: Multiprocessor System Detected: 22 CPUs
FreeBSD/SMP: Non-uniform topology
END QUOTE
(The internal NVMe media has Dell's ubuntu on it.)
Note:
Ignoring very old Intel MacBook Pro's and Mac Mini
2018's, none of which I've ever native-booted FreeBSD
on, the Dell P. 5490 is the only Thunderbolt or USB4
based system that I've access to.
===
Mark Millard
marklmi at yahoo.com
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

On Oct 2, 2025, at 20:45, Mark Millard <marklmi@yahoo.com> wrote:

On Sep 30, 2025, at 20:43, Mark Millard <marklmi@yahoo.com> wrote:

[The new material here ends up being about nameicap_cleanup
and its exclusive use of mnt_renamelock being one potential
bottleneck involved here. I make no claim it has anything to
do with the flswai activity reported. The possible
bottleneck is an observation, not something that I claim
there is any alternative to. I do not know if this is of any
interest or not.]

On Sep 29, 2025, at 16:06, Mark Millard <marklmi@yahoo.com> wrote:

On Sep 29, 2025, at 13:01, Mark Millard <marklmi@yahoo.com> wrote:

An example is during the cpdup activities when multiple happen
in overlappingtime frames:

I'll note that I see this on the amd64 32-FreeBSD-cpu system
but not on the aarch64 8-FreeBSD-cpu Windows Dev Kit 2023
system. May be at some point I'll try the older 16-FreeBSD-cpu
aarch64 (Cortex-A72) system.

Also, on the 7950X3D amd74 system, I see the behavior with
14.3-Stable. Apparently, this is not new with 15+. It has
been a long time since I'd tried using an amd64 system for
such activity based on using USB3 media. But it has been
common for me for aarch64 over that time frame.

. . .
. . .

. . .

. . .
. . .

But I'll also see such on c compiles, ld commands, etc. I've
not seen rename for pkg-static but I have seen flswai for it.

The system spends lots of time 95%+ idle from the wait
activities.

I see such directly booted from the USB3 media (a 15.0-ALPHA4
context on UFS media) and when using that media via chroot
from both ZFS and UFS boots that are not USB based. The ZFS
and UFS boots do not show the behavior with the normal
non-USB3 media used instead.

The system in use is an AMD 7950X3D with 32 FreeBSD cpus,
192 GiBytes of RAM. main 16 booting for non-USB boots
and 15.0-ALPHA4 boots for the USB3 boots. kernel and
world are via official pkgbase distribution installs:
it is not a personal build of the kernel or world.


. . .
. . .

I got a test context were I could compare the same
media used on the same USB4 port on a laptop
(Dell Precision 5490, 22 FreeBSD cpus, 32 GiBytes
RAM, 4 USB4 ports), where, on boot, the media ends
up being handled as either:

) "nda0 at nvme0" (via involving a Thunderbolt 3 hub)
) "da0" (via a direct connection)

(The UEFI/ACPI does enough to make basic operation
work, presenting some view to FreeBSD for media on
USB4 ports.)

Both have the bottlenecks visible when monitored with
top, both "flswai" and "rename" examples occur in
both contexts.

But "nda0 at nvme0" bottleneck periods do not last
nearly as long as "da0" bottleneck periods do, making
"nda0 at nvme0" use much more reasonable for the
type of activity.

Still, this eliminates the possibility that the issue
was limited to USB. It also eliminates it being
specific to the prior AMD (7950X3D) test context.

For reference:

# uname -apKU
FreeBSD USB4sys 16.0-CURRENT FreeBSD 16.0-CURRENT main-n280801-213170eb956f GENERIC-NODEBUG amd64 amd64 1600001 1600001

(It is from official pkgbase distribution use, a
copy of another boot media with some parameters
replaced afterwards.)

QUOTE
CPU: Intel(R) Core(TM) Ultra 7 165H (3072.00-MHz K8-class CPU) Origin="GenuineIntel" Id=0xa06a4 Family=0x6 Model=0xaa Stepping=4
. . .
WARNING: L3 data cache covers more APIC IDs than a package (6 > 3) FreeBSD/SMP: Multiprocessor System Detected: 22 CPUs
FreeBSD/SMP: Non-uniform topology
END QUOTE

(The internal NVMe media has Dell's ubuntu on it.)

Note:
Ignoring very old Intel MacBook Pro's and Mac Mini
2018's, none of which I've ever native-booted FreeBSD
on, the Dell P. 5490 is the only Thunderbolt or USB4
based system that I've access to.

I've discovered what is tied to the flswai/rename:
Soft Updates had been enabled. Disabling Soft Updates
leads to biowr and getblk as what shows for the
bottleneck. (There is still a bottleneck.)
===
Mark Millard
marklmi at yahoo.com
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

On Oct 2, 2025, at 21:43, Mark Millard <marklmi@yahoo.com> wrote:

On Oct 2, 2025, at 20:45, Mark Millard <marklmi@yahoo.com> wrote:

On Sep 30, 2025, at 20:43, Mark Millard <marklmi@yahoo.com> wrote:

[The new material here ends up being about nameicap_cleanup
and its exclusive use of mnt_renamelock being one potential
bottleneck involved here. I make no claim it has anything to
do with the flswai activity reported. The possible
bottleneck is an observation, not something that I claim
there is any alternative to. I do not know if this is of any
interest or not.]

On Sep 29, 2025, at 16:06, Mark Millard <marklmi@yahoo.com> wrote:

On Sep 29, 2025, at 13:01, Mark Millard <marklmi@yahoo.com> wrote:

An example is during the cpdup activities when multiple happen
in overlappingtime frames:

I'll note that I see this on the amd64 32-FreeBSD-cpu system
but not on the aarch64 8-FreeBSD-cpu Windows Dev Kit 2023
system. May be at some point I'll try the older 16-FreeBSD-cpu
aarch64 (Cortex-A72) system.

Also, on the 7950X3D amd74 system, I see the behavior with
14.3-Stable. Apparently, this is not new with 15+. It has
been a long time since I'd tried using an amd64 system for
such activity based on using USB3 media. But it has been
common for me for aarch64 over that time frame.

. . .
. . .

. . .

. . .
. . .

But I'll also see such on c compiles, ld commands, etc. I've
not seen rename for pkg-static but I have seen flswai for it.

The system spends lots of time 95%+ idle from the wait
activities.

I see such directly booted from the USB3 media (a 15.0-ALPHA4
context on UFS media) and when using that media via chroot
from both ZFS and UFS boots that are not USB based. The ZFS
and UFS boots do not show the behavior with the normal
non-USB3 media used instead.

The system in use is an AMD 7950X3D with 32 FreeBSD cpus,
192 GiBytes of RAM. main 16 booting for non-USB boots
and 15.0-ALPHA4 boots for the USB3 boots. kernel and
world are via official pkgbase distribution installs:
it is not a personal build of the kernel or world.


. . .
. . .

I got a test context were I could compare the same
media used on the same USB4 port on a laptop
(Dell Precision 5490, 22 FreeBSD cpus, 32 GiBytes
RAM, 4 USB4 ports), where, on boot, the media ends
up being handled as either:

) "nda0 at nvme0" (via involving a Thunderbolt 3 hub)
) "da0" (via a direct connection)

(The UEFI/ACPI does enough to make basic operation
work, presenting some view to FreeBSD for media on
USB4 ports.)

Both have the bottlenecks visible when monitored with
top, both "flswai" and "rename" examples occur in
both contexts.

But "nda0 at nvme0" bottleneck periods do not last
nearly as long as "da0" bottleneck periods do, making
"nda0 at nvme0" use much more reasonable for the
type of activity.

Still, this eliminates the possibility that the issue
was limited to USB. It also eliminates it being
specific to the prior AMD (7950X3D) test context.


For reference:

# uname -apKU
FreeBSD USB4sys 16.0-CURRENT FreeBSD 16.0-CURRENT main-n280801-213170eb956f GENERIC-NODEBUG amd64 amd64 1600001 1600001

(It is from official pkgbase distribution use, a
copy of another boot media with some parameters
replaced afterwards.)

QUOTE
CPU: Intel(R) Core(TM) Ultra 7 165H (3072.00-MHz K8-class CPU)
Origin="GenuineIntel" Id=0xa06a4 Family=0x6 Model=0xaa Stepping=4
. . .
WARNING: L3 data cache covers more APIC IDs than a package (6 > 3)
FreeBSD/SMP: Multiprocessor System Detected: 22 CPUs
FreeBSD/SMP: Non-uniform topology
END QUOTE

(The internal NVMe media has Dell's ubuntu on it.)


Note:
Ignoring very old Intel MacBook Pro's and Mac Mini
2018's, none of which I've ever native-booted FreeBSD
on, the Dell P. 5490 is the only Thunderbolt or USB4
based system that I've access to.

I've discovered what is tied to the flswai/rename:
Soft Updates had been enabled. Disabling Soft Updates
leads to biowr and getblk as what shows for the
bottleneck. (There is still a bottleneck.)

An FYI showing what can happen for the 32 FreeBSD cpu
USB3-based test context. The below occurred later,
after the initial bottleneck sequence when the jails
are being filling in via cpdup of the ref (for UFS
context via chrooting into it to then start the
build):
# uname -apKU
FreeBSD 7950X3D-ZFS 16.0-CURRENT FreeBSD 16.0-CURRENT main-n280910-85531add2844 GENERIC-NODEBUG amd64 amd64 1600001 1500066
[00:18:54] [official-amd64-default] [2025-10-04_21h23m47s] [parallel_build] Time: 00:17:58
Queued: 693 Inspected: 0 Ignored: 0 Built: 33 Failed: 0 Skipped: 0 Fetched: 0 Remaining: 660
ID TOTAL ORIGIN PKGNAME PHASE TIME TMPFS CPU% MEM%
[29] 00:08:55 databases/lmdb | lmdb-0.9.33,1 starting 00:08:55
[01] 00:02:53 graphics/libpotrace | libpotrace-1.16 starting 00:02:53
[15] 00:07:37 archivers/lzo2 | lzo2-2.10_1 starting 00:07:37
[30] 00:07:22 lang/lua53 | lua53-5.3.6_1 starting 00:07:22
[02] 00:07:51 devel/libmtdev | libmtdev-1.1.7 starting 00:07:51
[16] 00:07:35 lang/lua54 | lua54-5.4.8 starting 00:07:35
[31] 00:07:27 sysutils/libsunacl | libsunacl-1.0.1_1 starting 00:07:27
[03] 00:08:07 sysutils/dmidecode | dmidecode-3.6 starting 00:08:07
[17] 00:08:33 x11/xorgproto | xorgproto-2024.1 starting 00:08:33
[32] 00:07:38 print/indexinfo | indexinfo-0.3.1_1 starting 00:07:38
[04] 00:06:18 devel/libdatrie | libdatrie-0.2.13_2 starting 00:06:18
[18] 00:07:52 shells/bash-completion-freebsd | bash-completion-freebsd-1.4.0 starting 00:07:52
[05] 00:08:22 audio/libvorbis | libvorbis-1.3.7_2,3 starting 00:08:22
[19] 00:07:34 comms/iwmbt-firmware | iwmbt-firmware-20250410 starting 00:07:34
[06] 00:04:56 sysutils/sdparm | sdparm-1.12_1 starting 00:04:56
[20] 00:08:01 shells/ksh93 | ksh93-93.u_4,2 starting 00:08:01
[07] 00:08:02 textproc/sdocbook-xml | sdocbook-xml-1.1_2,2 starting 00:08:02
[21] 00:08:49 archivers/libmspack | libmspack-0.11alpha starting 00:08:49
[08] 00:08:23 benchmarks/iperf3 | iperf3-3.19.1 starting 00:08:23
[22] 00:08:01 textproc/xmlcharent | xmlcharent-0.3_2 starting 00:08:01
[09] 00:08:03 devel/pkgconf | pkgconf-2.4.3,1 starting 00:08:03
[23] 00:08:15 lang/perl5.42 | perl5-5.42.0_1 starting 00:08:15
[10] 00:08:16 devel/opencl | opencl-3.0.19 starting 00:08:16
[24] 00:08:11 benchmarks/bonnie | bonnie-2.0.6_2 starting 00:08:11
[11] 00:08:45 security/easy-rsa | easy-rsa-3.2.4,1 starting 00:08:45
[25] 00:07:48 audio/mpg123 | mpg123-1.33.2 starting 00:07:48
[12] 00:03:34 databases/sqlite3@default | sqlite3-3.50.2_1,1 starting 00:03:34
[26] 00:08:45 ports-mgmt/portconfig | portconfig-0.6.2_2 starting 00:08:45
[13] 00:07:54 textproc/expat2 | expat-2.7.3 starting 00:07:54
[27] 00:07:49 multimedia/v4l_compat | v4l_compat-1.23.0_7 starting 00:07:49
[14] 00:08:27 dns/public_suffix_list | public_suffix_list-20250828 starting 00:08:27
[28] 00:07:54 devel/libdwarf | libdwarf-20161124 starting 00:07:54
It does eventually make progress again after this.
The laptop 22 FreeBSD cpu context with external USB4
media being handled as "nda0 at nvme0" (via using a
Thunderbolt 3 hub between the laptop and the media)
has much better performance for builds. (No chroot
is involved here.)
# uname -apKU
FreeBSD USB4sys 16.0-CURRENT FreeBSD 16.0-CURRENT main-n280910-85531add2844 GENERIC-NODEBUG amd64 amd64 1600001 1600001
Shorter duration bottlenecking is still visible but
use wold be much more reasonable in this type of
context.
I''ll note that I did not notice any "rename" STATE
bottleneck time during this activity but did see
"flswai" STATE bottlenecking. (Soft Updates was
enabled.)
===
Mark Millard
marklmi at yahoo.com
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21a-Linux NewsLink 1.2