| Sysop: | Amessyroom |
|---|---|
| Location: | Fayetteville, NC |
| Users: | 42 |
| Nodes: | 6 (0 / 6) |
| Uptime: | 01:38:14 |
| Calls: | 220 |
| Calls today: | 1 |
| Files: | 824 |
| Messages: | 121,542 |
| Posted today: | 6 |
On Fri, 22 Nov 2024 06:37:06 -0000 (UTC), Lawrence D'Oliveiro <ldo@nz.invalid> wrote in <vhp8qi$12m83$2@dont-email.me>:That
On 22 Nov 2024 06:09:05 GMT, vallor wrote:
Doesn't the named pipe connection work through the filesystem code?
incould add overhead.
No. The only thing that exists in the filesystem is the “special file” >> entry in the directory. Opening that triggers special-case processing
settingthe kernel that creates the usual pipe buffering/synchronization
structures (or links up with existing structures created by some prior
opening of the same special file, perhaps by a different process), not
dependent on any filesystem.
I just tried creating a C program to do speed tests on data transfers
through pipes and socket pairs between processes. I am currently
(usingthe counter to 10 gigabytes, and transferring that amount of data
iswhichever mechanism) only takes a couple of seconds on my system.
So the idea that pipes are somehow not suited to large data transfers
dev/nullpatently nonsense.
Can't use named pipes on just any filesystem -- won't work on NFS for
example, unless I'm mistaken.
Hard to believe NFS could stuff that up, but there you go ...
Just tested NFS, and named pipes work there.
$ time -p ( dd if=/dev/zero of=test count=$[1024*1024] ) & cat test > /
[1] 38859
1048576+0 records in
1048576+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 0.918945 s, 584 MB/s
real 0.92
user 0.16
sys 0.76
NFS vers 4.1.
However, the speed appears to be limited by dd in my examples -- setting
a block size to fill the pipe/packets seems to increase throughput:
$ nc -l -U -N /tmp/socket > /dev/null & time -p ( dd if=/dev/zero count=$[1024*1024*4] bs=1024 | nc -U -N /tmp/socket > /dev/null )
... or 4.1GB/s).
On 22 Nov 2024 07:29:16 GMT, vallor <vallor@cultnix.org> wrote in <lqaq6cF8btnU3@mid.individual.net>:
However, the speed appears to be limited by dd in my examples -- setting
a block size to fill the pipe/packets seems to increase throughput:
$ nc -l -U -N /tmp/socket > /dev/null & time -p ( dd if=/dev/zero
count=$[1024*1024*4] bs=1024 | nc -U -N /tmp/socket > /dev/null )
Realized the bottleneck would be the pipe between dd and nc, so wrote
a program to connect to /tmp/socket and spew data at it -- it sends
46950 212992-byte buffers (9999974400 bytes) in 2.41 seconds.
(4149366971 bytes/second, or 4.1GB/s).
(The default "MTU" for a Linux Unix socket connection
is 212992 bytes. Default pipe size is 8*512 bytes.)
At the moment we have to do 4.6GB per 0.5sec to keep up, and
7GB per 0.5sec to stay ahead. We've done everything we can to squeeze
out as much performance from pipes as we could but there just isn't much
we can do anymore.
On Fri, 22 Nov 2024 10:00:31 -0500, Phillip Frabott wrote:
At the moment we have to do 4.6GB per 0.5sec to keep up, and
7GB per 0.5sec to stay ahead. We've done everything we can to squeeze
out as much performance from pipes as we could but there just isn't much
we can do anymore.
Sounds like a job for a shared memory section.
Unless you can recommend a way to do shared memory across different
machines without extra hardware.
The only 'filesystem' access for named pipes is during the
open() call to look up the name from the filesystem. Once you get the
file descriptor back, it is the exact same in-memory FIFO queue as an anonymous pipe created via pipe() (at least on Linux).
On 11/22/2024 03:38, vallor wrote:
On 22 Nov 2024 07:29:16 GMT, vallor <vallor@cultnix.org> wrote in
<lqaq6cF8btnU3@mid.individual.net>:
However, the speed appears to be limited by dd in my examples -- setting >>> a block size to fill the pipe/packets seems to increase throughput:
$ nc -l -U -N /tmp/socket > /dev/null & time -p ( dd if=/dev/zero
count=$[1024*1024*4] bs=1024 | nc -U -N /tmp/socket > /dev/null )
Realized the bottleneck would be the pipe between dd and nc, so wrote
a program to connect to /tmp/socket and spew data at it -- it sends
46950 212992-byte buffers (9999974400 bytes) in 2.41 seconds.
(4149366971 bytes/second, or 4.1GB/s).
(The default "MTU" for a Linux Unix socket connection
is 212992 bytes. Default pipe size is 8*512 bytes.)
Yeah, this has been the experience of our testing as well. We can also
bump the MTU up a bit to get sockets to do better and we've also
modified some of the socket code to our environment so it's able to do
around 12GB/s. We think we can tune it further to get about 15GB/s from sockets. At the moment we have to do 4.6GB per 0.5sec to keep up, and
7GB per 0.5sec to stay ahead. We've done everything we can to squeeze
out as much performance from pipes as we could but there just isn't much
we can do anymore. But yeah, sockets are way more capable of large data intake vs pipes, and this shows as well in a lot of research that's been done. We've been testing sockets for about the last year and the results
have shown we are keeping up better then we are with pipes right now.
And for tribute to pipes, we've been using pipes in this project for the
last 30 years. That's a pretty great track record for something to have
been useful for such a long time and not be replaced with something else until now. Sadly sockets will likely bottom out in the next 10-15 years
and we'll have to switch to something else. But for now, it's working.
... whereas on Solaris, I swapped (unnamed) pipes (STREAMS based,
there) with socketpair() using an LD_PRELOADable wrapper, so no changes
on either end.
On Tue, 03 Dec 2024 05:23:15 GMT, Richard L. Hamilton wrote:
... whereas on Solaris, I swapped (unnamed) pipes (STREAMS based,
there) with socketpair() using an LD_PRELOADable wrapper, so no changes
on either end.
STREAMS made that easy to do, didn’t it. Yet the whole idea was never that popular; both Steve Jobs and Linus Torvalds were less than enthusiastic
about it. Were there performance downsides?