From Newsgroup: alt.os.linux

Hello everyone,

I have a cluster with the latest GlusterFS on Debian12 with three nodes
but when I try a simple du -sh of the /var/www the node went in "N"
state and it doesn't come back to Y so I need to manually restart
glusterd daemon.

If I don't run backup, rsync, du, etc. the cluster works well, I used
also vmstat to see ram, cpu and disk; I have 4 vcpu 70% and 8GB of ram
for each node, free ram is roughly of 300MB, used about 1,5GB and
buffer/cache about 5GB.

I read on internet AWS starts from 16GB of ram for GlusterFS, other
documents said to use 12GB of ram, do you have experience about it?

^Bart
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: alt.os.linux

On Sat, 5 Jul 2025 08:41:23 +0200, ^Bart wrote:

I have a cluster with the latest GlusterFS on Debian12 with three nodes
but when I try a simple du -sh of the /var/www the node went in "N"
state and it doesn't come back to Y so I need to manually restart
glusterd daemon.

Any messages in the logs? journalctl? dmesg?
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: alt.os.linux

On 5/07/2025 4:41 pm, ^Bart wrote:

Hello everyone,

I have a cluster with the latest GlusterFS on Debian12 with three nodes
but when I try a simple du -sh of the /var/www the node went in "N"
state and it doesn't come back to Y so I need to manually restart
glusterd daemon.

If I don't run backup, rsync, du, etc. the cluster works well, I used
also vmstat to see ram, cpu and disk; I have 4 vcpu 70% and 8GB of ram
for each node, free ram is roughly of 300MB, used about 1,5GB and buffer/cache about 5GB.

Free Ram 300MB approx
Used Ram 1.5GB
Buffer/cache 5GB approx

Total 8GB approx
Available Ram 8GB approx

.... so are you all full up??

I read on internet AWS starts from 16GB of ram for GlusterFS, other documents said to use 12GB of ram, do you have experience about it?

^Bart

If you clear your Buffer/cache, might things run better??
--
Daniel70
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: alt.os.linux

On 05/07/2025 08.41, ^Bart wrote:

I have a cluster with the latest GlusterFS on Debian12 with three nodes
but when I try a simple du -sh of the /var/www the node went in "N"
state and it doesn't come back to Y so I need to manually restart
glusterd daemon.

Have you seen anything in the logs? Maybe check /var/log/glusterfs/glusterd.log
It can be lock that hasn't been released, sadly only fix is to restart glusterd.

If I don't run backup, rsync, du, etc. the cluster works well, I used
also vmstat to see ram, cpu and disk; I have 4 vcpu 70% and 8GB of ram
for each node, free ram is roughly of 300MB, used about 1,5GB and buffer/cache about 5GB.

It's quite many years since I used GlusterFS, but back then at work we
had quite large DELL servers (64GB RAM, 2 CPU with 8 cores each) with
SAN based storage as nodes, the system got degradation with having high read/write. In the end those was replaced by standard NFS servers which
gave more stability and then have replication from one SAN to another,
sure not a fully HA solution.

I think I should have pushed more for Lustre and the guys at CERN were
really helpful when I did some testing with just a simple setup which
needed a bit more nodes, sadly the requirements changed on the way that
made we needed to provide the file system to a closed source operating
system with poor file system support.

I read on internet AWS starts from 16GB of ram for GlusterFS, other documents said to use 12GB of ram, do you have experience about it?

gluster.org do write, for basic nodes: 2 CPU’s, 4GB of RAM each, 1
Gigabit network.
--
//Aho
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: alt.os.linux

On Sat, 5 Jul 2025 20:08:15 +1000, Daniel70
<daniel47@eternal-september.org> wrote in <104atij$1e1pj$1@dont-email.me>:

On 5/07/2025 4:41 pm, ^Bart wrote:

Hello everyone,

I have a cluster with the latest GlusterFS on Debian12 with three nodes
but when I try a simple du -sh of the /var/www the node went in "N"
state and it doesn't come back to Y so I need to manually restart
glusterd daemon.

If I don't run backup, rsync, du, etc. the cluster works well, I used
also vmstat to see ram, cpu and disk; I have 4 vcpu 70% and 8GB of ram
for each node, free ram is roughly of 300MB, used about 1,5GB and
buffer/cache about 5GB.

Free Ram 300MB approx
Used Ram 1.5GB
Buffer/cache 5GB approx

Total 8GB approx
Available Ram 8GB approx

.... so are you all full up??

I read on internet AWS starts from 16GB of ram for GlusterFS, other
documents said to use 12GB of ram, do you have experience about it?

^Bart

If you clear your Buffer/cache, might things run better??

Hi Daniel,

The way Linux works, free memory gets used as Buffer/cache.
As more memory is allocated, it pulls it from the B/C. It's
basically part of the "free" memory, but being "borrowed"
by the OS for better performance.
--
-v System76 Thelio Mega v1.1 x86_64 NVIDIA RTX 3090Ti 24G
OS: Linux 6.15.4 D: Mint 22.1 DE: Xfce 4.18 Mem: 258G
"Ever notice how fast Windows runs? Neither have I."
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: alt.os.linux

On Sat, 7/5/2025 10:09 PM, vallor wrote:

On Sat, 5 Jul 2025 20:08:15 +1000, Daniel70
<daniel47@eternal-september.org> wrote in <104atij$1e1pj$1@dont-email.me>:

On 5/07/2025 4:41 pm, ^Bart wrote:

Hello everyone,

I have a cluster with the latest GlusterFS on Debian12 with three nodes >>> but when I try a simple du -sh of the /var/www the node went in "N"
state and it doesn't come back to Y so I need to manually restart
glusterd daemon.

If I don't run backup, rsync, du, etc. the cluster works well, I used
also vmstat to see ram, cpu and disk; I have 4 vcpu 70% and 8GB of ram
for each node, free ram is roughly of 300MB, used about 1,5GB and
buffer/cache about 5GB.

Free Ram 300MB approx
Used Ram 1.5GB
Buffer/cache 5GB approx

Total 8GB approx
Available Ram 8GB approx

.... so are you all full up??

I read on internet AWS starts from 16GB of ram for GlusterFS, other
documents said to use 12GB of ram, do you have experience about it?

^Bart

If you clear your Buffer/cache, might things run better??

Hi Daniel,

The way Linux works, free memory gets used as Buffer/cache.
As more memory is allocated, it pulls it from the B/C. It's
basically part of the "free" memory, but being "borrowed"
by the OS for better performance.

"If you clear your Buffer/cache, might things run better"

Some people will tell you to test that for your very own
self as a Linux user, and to your surprise, it is as they
predicted, it makes no difference if you do a drop-cache.

On some of the older Linux media, I used to write stuff
as a liner note. This is from the paper wrapper on the
Knoppix 5.3.1 DVD I made a while back.

echo 1 > /proc/sys/vm/drop_caches

1=PageCache
2=Dentries,Inodes
3=Both

If you run "top" in one terminal session, then issue
the command in another terminal session, you will see
some numbers change in the "top" display, but the performance
of the machine does not change.

*******

The OPs problem is at a different scale than what a
home user can reproduce. Perhaps a person who has
a Linux based work environment, has seen such a setup
and can comment. Us home users, it would be pretty difficult
to make a convincing setup.

Imagine if the owner of Archive.org came online and
asked a question about "a problem with his setup".
Not many users here, have an Archive.org setup in
their basement :-}

Paul
--- Synchronet 3.21a-Linux NewsLink 1.2