ovsqlite: Add direct reader mode for nnrpd with WAL
expireover: Add bloom filter for fast history existence checks
expire: skip per-article SMretrieve via cancel tombstone log
The general theme of all three is to reduce random or sync I/O.-a In the case of expireover and expire, the general problem is somewhat similar
but the root cause and fix are fairly different.
On my system this is a 1100x improvement, 2 weeks down to 18 minutes.
On my system this will net a 2800x improvement, 2 days down to sub minute.
Both of these suppose integrity of the spool; if you 'rm' articles
instead of using 'sm -r' you'll need to do a patrol expire to clean up
your spool; expireover builds from history so it would see the new
history state the next time it runs.-a The first time expire runs it wont have a tombstone so it will 'upgrade' to a known state cleaning any
issues.. a gap is only if you decide to manually botch the spool after
that.
I'd be curious to hear any feedback or reviews from other inn admins
that can test the patch set.
Hi Kevin,
ovsqlite: Add direct reader mode for nnrpd with WAL
expireover: Add bloom filter for fast history existence checks
expire: skip per-article SMretrieve via cancel tombstone log
The general theme of all three is to reduce random or sync I/O.-a In
the case of expireover and expire, the general problem is somewhat
similar but the root cause and fix are fairly different.
Many thanks for these very useful improvements to expiration and
ovsqlite concurrent reading.
expire and expireover run way faster (x1100 for expire and x2800 for expireover on your system !!) thanks to your optimizations.-a Some people have been complaining with their slowness from a long time, and
hopefully you managed to find out how to improve that a lot and
contributed a patch.-a It will be integrated to the next release.
On my system this is a 1100x improvement, 2 weeks down to 18 minutes.
Gosh!
On my system this will net a 2800x improvement, 2 days down to sub
minute.
Re-gosh!
Both of these suppose integrity of the spool; if you 'rm' articles
instead of using 'sm -r' you'll need to do a patrol expire to clean up
your spool; expireover builds from history so it would see the new
history state the next time it runs.-a The first time expire runs it
wont have a tombstone so it will 'upgrade' to a known state cleaning
any issues.. a gap is only if you decide to manually botch the spool
after that.
Worth mentioning.
I'd be curious to hear any feedback or reviews from other inn admins
that can test the patch set.
If anyone could test the provided patches, do not hesitate to do so in
order to eventually catch possible nits before the next release (in June).
I've wondered a bit about what might be next.
network programming changed a lot once kqueue and epoll entered a couple years later.
nnrpd would be somewhat easy to make more efficient across a few dimensions.-a The least code churn might be integrating poll() directly
in the per process forks and adding some eventing to deal with NNTP pipelining and I/O and DNS.-a The main benefit would be building up some
I/O parallelism, all storage devices benefit from that and you smooth
out the latency of hot vs cold I/Os through the page cache (or ARC/
L2ARC) and actual storage devices.
Similar opportunities likely present in innd/innfeeder but I am notSure, there also are improvements to do for them!
familiar with this code so far.
File formats would be another consideration I am not familiar enough to evaluate right now but the network and disk I/O stuff interests me more.
Data movement could optionally use sendfile and even KTLS sendfile on Linux and FreeBSD.
On 5/10/2026 8:59 PM, Kevin Bowling wrote:
Data movement could optionally use sendfile and even KTLS sendfile on
Linux and FreeBSD.
So, this gave me an interesting thought earlier this evening, as I
actually do use sendfile() a lot (it's not very portable, so I have
wrappers around sendfile, copy_file_range, splice, and other fun
functions).
I was reading the requirement for :bytes earlier today and noted that it specifically does not include dot-stuffing characters, and I recalled
then that in SMTP and NNTP, I strip out any dot-stuffed lines as I
thought was convention. However, when *sending* the articles to a
client, I have just been using ~sendfile, for efficiency. I realize now
this is illegal, since I'm not adding the dot-stuffing back.
Taking a look at INN, I noticed it counts the number of dot-stuffed
lines and subtracts it from the value it provides for :bytes, which is
what the RFC says to do. But it looks like INN doesn't remove the dot- stuffing when receiving an article. This actually seems like the more elegant way to do things; if you never remove the dot-stuffing, you
don't need to add it back, and then you can do efficient things like use sendfile rather than going line by line. So in my case, I think I don't
need to care about receiving dot-stuffed lines since I can store them in
the spool and just have it be transparent.
On the other hand, this wouldn't work for SMTP, since with BDAT/
CHUNKING, you may not need to do dot-stuffing.
I guess what I'm trying to say is that for SMTP, because there are
multiple ways to send/receive messages, it makes sense to store the
messages in the "queue" normalized (without dot-stuffing), but since
there's only one way in NNTP, it makes more sense to just store the
stuffed lines in the spool directly. Is this about the gist of it?
(And implicitly, I'm assuming that in the future, it would be hard to support other extensions like SMTP has since I would assume most news
server are storing the dots in the spool; of course, such extensions are probably of little value to NNTP considering most articles are small).
On 5/12/26 19:46, InterLinked wrote:
I guess what I'm trying to say is that for SMTP, because there are
multiple ways to send/receive messages, it makes sense to store the
messages in the "queue" normalized (without dot-stuffing), but since
there's only one way in NNTP, it makes more sense to just store the
stuffed lines in the spool directly. Is this about the gist of it?
(And implicitly, I'm assuming that in the future, it would be hard to
support other extensions like SMTP has since I would assume most news
server are storing the dots in the spool; of course, such extensions
are probably of little value to NNTP considering most articles are
small).
Can you post some example transactions showing your thoughts?
One thing that comes to mind is probably to synthesize the header(s) and sendfile at an offset of the body.-a The synthesized part could be stack allocated.-a Likewise any trailer if 'wireformat' is off.
That is similar to what I'm talking about, though I think wireformat in
INN is just line endings (whether to store the CR or not). I was talking about storing the leading dot in dot-stuffed lines in the spool (same idea really).
I believe that INN does this unconditionally without any way to disable
it... because why would you want to?
| Sysop: | Amessyroom |
|---|---|
| Location: | Fayetteville, NC |
| Users: | 65 |
| Nodes: | 6 (0 / 6) |
| Uptime: | 09:43:06 |
| Calls: | 862 |
| Files: | 1,311 |
| D/L today: |
3 files (7,546K bytes) |
| Messages: | 265,091 |