• i386 on amd64 can fail to return from cond_wait_user, using basically100% of a FreeBSD cpu

    From Mark Millard@marklmi@yahoo.com to muc.lists.freebsd.stable on Fri Jul 4 23:01:22 2025
    From Newsgroup: muc.lists.freebsd.stable

    Some package builds are failing on the port-packages build cluster
    machines that do i386 builds during the following code. The analysis
    is from replication in a personal context (using poudriere bulk with
    -i), using poudriere-devel instead. I'll note that the personal
    context is from using PkgBase 14.3-RELEASE in the poudriere jail.
    I also installed most of the realted *-dbg* PkgBase packages in order
    to get the nicer backtracing.
    (gdb) bt
    #0 _umtx_op_err () at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/arch/i386/i386/_umtx_op_err.S:37
    #1 0x2499f897 in _thr_umtx_timedwait_uint (mtx=0x249a365c, id=0, clockid=4, abstime=0x0, shared=0) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_umtx.c:233
    #2 0x24995b26 in _thr_sleep (curthread=0x24d36004, clockid=4, abstime=0x0) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_kern.c:197
    #3 0x24990beb in cond_wait_user (cvp=0x24dfa8a0, mp=0x24d38d04, abstime=<optimized out>, cancel=<optimized out>) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_cond.c:317
    NOTE: cond_wait_user never returns but #2..#0 repeat (observed by
    repeated ^c and bt usage).
    (i386 is the oddball with 32-bit time_t but I do not know
    if that is involved here.)
    #4 cond_wait_common (cond=<optimized out>, mutex=<optimized out>, abstime=0x0, cancel=1) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_cond.c:377
    #5 0x24990e8f in __thr_cond_wait (cond=0x23b9b4f4, mutex=0x23b9b4ec) at /home/pkgbuild/worktrees/releng/14.3/lib/libthr/thread/thr_cond.c:392
    #6 0x23be1e4b in uv_cond_wait () from /usr/local/lib/libuv.so.1
    #7 0x024bd497 in node::NodePlatform::DrainTasks(v8::Isolate*) ()
    #8 0x0232f5b6 in node::SpinEventLoopInternal(node::Environment*) ()
    #9 0x02485bf0 in node::NodeMainInstance::Run() ()
    #10 0x023eaba1 in node::Start(int, char**) ()
    #11 0x24a1da85 in __libc_start1 (argc=5, argv=0xffffda3c, env=0xffffda54, cleanup=0x23b73020 <rtld_nop_exit>, mainX=0x314a720 <main>)
    at /home/pkgbuild/worktrees/releng/14.3/lib/libc/csu/libc_start1.c:157
    #12 0x0232d0a8 in _start ()
    www/librewolf and other firefox related package builds can do
    this until a 7200 sec timeout by poudriere occurs:
    Killing runaway build after 7200 seconds with no output
    I'll note that truss did not generate any output when used to
    watch the process that was stuck. It appears to be a world-internal
    problem.
    For reference (note the thread with "35:49.90" that was the
    one stuck looping in such a manor):
    # ps -Halxdww
    . . .
    0 4221 93681 9 15 0 16868 5432 wait I 0 0:00.00 | | `-- sh: poudriere[release-i386-alt][20]: build_pkg (librewolf-139.0.4) (sh)
    0 4222 4221 26 59 0 6312 3332 wait IsJ - 0:00.01 | | `-- /usr/bin/make -C /usr/ports/www/librewolf build
    0 4246 4222 17 59 0 5064 2740 wait IJ - 0:00.00 | | `-- /bin/sh -e -c (cd /wrkdirs/usr/ports/www/librewolf/work/.build; if ! /usr/bin/env -i HOME=/wrkdirs/usr/ports/www/librewolf/work MACHINE_ARCH=i386 PWD="${PWD}" GIT_CEILING_DIRECTORIES=/wrkdirs/usr/ports/www/librewolf/work __MAKE_CONF=/nonexistent OSVERSION=1403000 PATH=/wrkdirs/usr/ports/www/librewolf/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin TMPDIR=/tmp UNAME_m=i386 UNAME_p=i386 UNAME_r=14.3-RELEASE UNAME_v=FreeBSD\\ 14.3-RELEASE LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 XDG_DATA_HOME=/wrkdirs/usr/ports/www/librewolf/work XDG_CONFIG_HOME=/wrkdirs/usr/ports/www/librewolf/work XDG_CACHE_HOME=/wrkdirs/usr/ports/www/librewolf/work/.cache HOME=/wrkdirs/usr/ports/www/librewolf/work TMPDIR="/tmp" PATH=/wrkdirs/usr/ports/www/librewolf/work/.bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin PKG_CONFIG_LIBDIR=/wrkdirs/usr/ports/www/librewolf/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES PREFIX=/usr/local LOCALBASE=/usr/local CC="/usr/local/bin/clang19" CFLAGS="-O2 -pipe -O3 -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing " CPP="/usr/local/bin/clang-cpp19" CPPFLAGS="-DLIBICONV_PLUG -isystem /usr/local/include" LDFLAGS=" -Wl,--as-needed -Wl,--undefined-version -fstack-protector-strong " LIBS="-L/usr/local/lib" CXX="/usr/local/bin/clang++19" CXXFLAGS="-O2 -pipe -O3 -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DLIBICONV_PLUG -isystem /usr/local/include " BSD_INSTALL_PROGRAM="install -m 555" BSD_INSTALL_LIB="install -m 0644" BSD_INSTALL_SCRIPT="install -m 555" BSD_INSTALL_DATA="install -m 0644" BSD_INSTALL_MAN="install -m 444" /usr/local/bin/gmake -f Makefile all; then if [ -n "" ] ; then echo "===> Compilation failed unexpectedly."; (echo "") | /usr/bin/fmt 75 79 ; fi; false; fi)
    0 4247 4246 30 29 0 7580 4980 wait IJ - 0:00.01 | | `-- /usr/local/bin/gmake -f Makefile all
    0 57541 4247 19 37 0 8020 5472 wait IJ - 0:00.03 | | `-- /usr/local/bin/gmake misc
    0 57542 57541 29 59 0 8020 5480 wait IJ - 0:00.03 | | `-- /usr/local/bin/gmake recurse_misc
    0 57756 57542 3 59 0 27412 24160 select IJ - 0:00.07 | | `-- /wrkdirs/usr/ports/www/librewolf/work/.mozbuild/srcdirs/librewolf-139.0.4-1-f0fbc434e1bd/_virtualenvs/build/bin/python -m mozbuild.action.file_generate /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/python/mozbuild/mozbuild/action/node.py generate devtools/client/debugger/src/actions/ast/node.stub devtools/client/debugger/src/actions/ast/.deps/node.stub.pp devtools/client/debugger/src/actions/ast/.deps/node.stub.stub /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast (python3.11)
    0 57757 57756 19 59 0 147928 131892 uwait IJ - 0:00.09 | | |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
    0 57757 57756 2 59 0 147928 131892 kqread IJ - 0:00.00 | | |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
    0 57757 57756 3 59 0 147928 131892 uwait IJ - 0:00.00 | | |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
    0 57757 57756 21 59 0 147928 131892 uwait IJ - 0:00.00 | | |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
    0 57757 57756 25 141 0 147928 131892 - RJ - 35:49.90 | | |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
    0 57757 57756 30 59 0 147928 131892 uwait IJ - 0:00.00 | | |-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
    0 57757 57756 12 59 0 147928 131892 usem IJ - 0:00.00 | | `-- /usr/local/bin/node /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/shared/build/build.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/index.js /wrkdirs/usr/ports/www/librewolf/work/librewolf-139.0.4-1/devtools/client/debugger/src/actions/ast/setInScopeLines.js /wrkdirs/usr/ports/www/librewolf/work/.build//dist/bin/browser/chrome/devtools/modules/devtools/client/debugger/src/actions/ast
    . . .
    # poudriere jail -jrelease-i386 -i
    Jail name: release-i386
    Jail version: 14.3-RELEASE
    Jail arch: i386
    Jail method: pkgbase
    Jail mount: /usr/local/poudriere/jails/release-i386
    Jail fs: zoptb/poudriere/jails/release-i386
    Jail updated: 2025-06-28 21:53:52
    Jail pkgbase: disabled
    # uname -apKU
    FreeBSD 7950X3D-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT main-n278320-3a33e39edd48 GENERIC-NODEBUG amd64 amd64 1500048 1500048
    # ~/fbsd-based-on-what-commit.sh -C /usr/ports-alt/
    6a6a0f11043c (HEAD -> main, freebsd/main, freebsd/HEAD) graphics/kdegraphics: install colord-kde
    Author: Max Brazhnikov <makc@FreeBSD.org>
    Commit: Max Brazhnikov <makc@FreeBSD.org>
    CommitDate: 2025-06-28 18:03:24 +0000
    branch: main
    merge-base: 6a6a0f11043cea55e789395d2f89b091c1cdf97c
    merge-base: CommitDate: 2025-06-28 18:03:24 +0000
    n711132 (--first-parent --count for merge-base) https://portsfallout.com/fallout?port=%2Flibrewolf&maintainer=&env=&category=build%2Frunaway&flavor=
    shows as having such failures for www/librewolf:
    135i386-default
    main-i386-default
    142i386-quarterly
    135i386-quarterly
    134i386-default
    134i386-quarterly
    Interestingly, 142i386-default is not showing up there, But: https://pkg-status.freebsd.org/beefy21/data/142i386-default/557ad2fd0e8a/logs/errors/librewolf-140.0.2.log
    is an example.
    (Looks like 142i386-default might not be getting fallout reports?)
    ===
    Mark Millard
    marklmi at yahoo.com
    --
    Posted automagically by a mail2news gateway at muc.de e.V.
    Please direct questions, flames, donations, etc. to news-admin@muc.de
    --- Synchronet 3.21a-Linux NewsLink 1.2