From Newsgroup: muc.lists.freebsd.stable

I upgraded all our machines to 15.0-STABLE last week. Which works fine,
but I have noticed a very curious thing where the load averages on our
main database machine have doubled.

See graph here...

https://api.ticketswitch.com/shared/mysql-load-average-14-to-15.png


The only thing which has changed there is the operating system, and
the re-install of the packages to match. All the versions are
identical, config is identical, workload is identical.

The CPU usage hasn't changed, just the load averages. If I look on
that machine, which is really only running mysql, then the processes
in mysql are all in the state "Waiting for table metadata lock"

My hope is that FreeBSD 15 somehow calculates load averages
differently, and I simply never noticed 100 processes
waiting on locks before. But I also worry that maybe mysql does
its locking using some operating system primitive, which has changed,
and is causing this.

Any ideas anyone ?

-pete.

PS: am also going to change the application to avoid 100 parallel
processes hitting that table at once, now I have seen this, but the
question as to why its changed still remains.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable



On 2/3/26 01:55, Pete French wrote:

I upgraded all our machines to 15.0-STABLE last week. Which works fine,
but I have noticed a very curious thing where the load averages on our
main database machine have doubled.

See graph here...

https://api.ticketswitch.com/shared/mysql-load-average-14-to-15.png

The only thing which has changed there is the operating system, and
the re-install of the packages to match. All the versions are
identical, config is identical, workload is identical.

The CPU usage hasn't changed, just the load averages. If I look on
that machine, which is really only running mysql, then the processes
in mysql are all in the state "Waiting for table metadata lock"

My hope is that FreeBSD 15 somehow calculates load averages
differently, and I simply never noticed 100 processes
waiting on locks before. But I also worry that maybe mysql does
its locking using some operating system primitive, which has changed,
and is causing this.

Any ideas anyone ?

which filesystem is your mysql data mounted on? my fist suspect would
be some performance change in the underlying filesystem, or how data is
cached in memory. systat may help shed light on this with maybe the
vmstat, iolat or iostat options.

do you have a similar system running 14.x available for comparison purposes?

-pete
--
Pete Wright
pete@nomadlogic.org



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable

------=_Part_577_1543205994.1770202627020
Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit

You could post more information about the system.

Something which pops up in my mind is that it might be interesting if it is a NUMA system.
https://en.wikipedia.org/wiki/Non-uniform_memory_access
Maybe locks use memory of other zones/nodes.

Is it possible to get the output of "procstat -kk <pid-of-mysqld>" when it is contending on these locks? That would show if it is waiting in the kernel or not.

Regards,
Ronald.


Van: Pete French <pete@twisted.org.uk>
Datum: woensdag, 4 februari 2026 11:19
Aan: stable@freebsd.org
Onderwerp: Re: Higher load averages running mysql on 15 instead fo 14

On 03/02/2026 18:19, Pete Wright wrote:

which filesystem is your mysql data mounted on? my fist suspect would > be some performance change in the underlying filesystem, or how data is > cached in memory. systat may help shed light on this with maybe the > vmstat, iolat or iostat options.

Its on ZFS, and that was my first thought as well, as the version of
ZFS did change between 14 and 15, and all the non running threads
seem to be in either kread or uwait state. But after some reading
around it doesn't look as if mysql is using the filesystem for these
kind of locks.

do you have a similar system running 14.x available for comparison > purposes?

Unfortunately not :-(

However, your suggestion of other things to look at are super helpful.
I often forget we have all these other tools available ;) thanks!

-pete. [ off to re-read the man page for systat etc.. ]

------=_Part_577_1543205994.1770202627020
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit

<html><head></head><body>You could post more information about the system.<br>

Something which pops up in my mind is that it might be interesting if it is a NUMA system.<br>
<a href="https://en.wikipedia.org/wiki/Non-uniform_memory_access">https://en.wikipedia.org/wiki/Non-uniform_memory_access</a><br>
Maybe locks use memory of other zones/nodes.<br>

Is it possible to get the output of "procstat -kk &lt;pid-of-mysqld&gt;" when it is contending on these locks? That would show if it is waiting in the kernel or not.<br>

Regards,<br>
Ronald.<br>

&nbsp;
<p><strong>Van:</strong> Pete French &lt;pete@twisted.org.uk&gt;<br> <strong>Datum:</strong> woensdag, 4 februari 2026 11:19<br> <strong>Aan:</strong> stable@freebsd.org<br>
<strong>Onderwerp:</strong> Re: Higher load averages running mysql on 15 instead fo 14</p>

<blockquote style="padding-right: 0px; padding-left: 5px; margin-left: 5px; border-left: #000000 2px solid; margin-right: 0px">
<div class="MessageRFC822Viewer" id="P">
<div class="TextPlainViewer" id="P.P"><br>

On 03/02/2026 18:19, Pete Wright wrote:<br>
&gt; which filesystem is your mysql data mounted on?&nbsp; my fist suspect would &gt; be some performance change in the underlying filesystem, or how data is &gt; cached in memory.&nbsp; systat may help shed light on this with maybe the &gt; vmstat, iolat or iostat options.<br>

Its on ZFS, and that was my first thought as well, as the version of<br>
ZFS did change between 14 and 15, and all the non running threads<br>
seem to be in either kread or uwait state. But after some reading<br>
around it doesn't look as if mysql is using the filesystem for these<br>
kind of locks.<br>

&gt; do you have a similar system running 14.x available for comparison &gt; purposes?<br>

Unfortunately not :-(<br>

However, your suggestion of other things to look at are super helpful.<br>
I often forget we have all these other tools available ;) thanks!<br>

-pete. [ off to re-read the man page for systat etc.. ]<br>
&nbsp;</div>

<hr></div>
</blockquote>

&nbsp;</body></html>
------=_Part_577_1543205994.1770202627020--


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21b-Linux NewsLink 1.2

From Newsgroup: muc.lists.freebsd.stable



On 04/02/2026 10:57, Ronald Klop wrote:

You could post more information about the system.

Ah, yes, sorry, should have done that!

Its an AWS c6id.8xlarge instance which is
64 gig of memory
32 cores (16 physical, with hyperthreads)

it has local SSD, on top of which is geli, and on
top of that is ZFS, compressed with ztsd, which
is where the mysql data lives. I cap the ARC at
32 gig, and it does eventually fill that over
time (or did on 14 anyway).
top of top...

CPU: 20.7% user, 0.0% nice, 31.8% system, 1.2% interrupt, 46.3% idle
Mem: 16G Active, 2551M Inact, 22G Wired, 21G Free
ARC: 18G Total, 1958M MFU, 16G MRU, 133M Anon, 231M Header, 12M Other
17G Compressed, 126G Uncompressed, 7.58:1 Ratio
Swap: 16G Total, 16G Free

Something which pops up in my mind is that it might be interesting if it
is a NUMA system.

The CPU is, apparently, a Xeon Platinum 8375C, so yes, could
well be NUMA, and though I would hope that AWS hypervisor would
put all my virtual cores onto a single real CPU, there's still
all that on-die stuff between cores... does the OS know about
that ?

Is it possible to get the output of "procstat -kk <pid-of-mysqld>" when
it is contending on these locks? That would show if it is waiting in the kernel or not.

Certainly!

https://api.ticketswitch.com/shared/mysql.procstat.kk.txt


But - check my thinking here - a load average is the runnable
processes, right ? So if they were simply stuck waiting on
some events, they would not be runnable, and thus wouldn't add
to the load average ?

[ Actually, having typed that sentance, I am now wondering if
this is due to some improvement in 15, so processes are now
waiting less, and completing faster, and thus letting another
one which was waiting on a lock become runnable, so we see more
active processes in a 'lav' timeslot ? ]

-pete.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de
--- Synchronet 3.21b-Linux NewsLink 1.2