From Newsgroup: news.software.nntp

Hi,

I've been looking at inn as I run it: tradspool/ovsqlite on FreeBSD+ZFS
with unlimited retention and trying to fix some scaling issues.

ovsqlite: Add direct reader mode for nnrpd with WAL https://github.com/InterNetNews/inn/pull/338

expireover: Add bloom filter for fast history existence checks https://github.com/InterNetNews/inn/pull/339

expire: skip per-article SMretrieve via cancel tombstone log https://github.com/InterNetNews/inn/pull/340

The first two have been running on csiph.com for a few days without
issue. I haven't yet deployed the last one yet but will publish results
in the PR when I do.

The general theme of all three is to reduce random or sync I/O. In the
case of expireover and expire, the general problem is somewhat similar
but the root cause and fix are fairly different.

expireover checks history for every article and this turns into a very
slow and random serial I/O for every article in the history DB. The new option is to use a bloom filter with a configurable false positive rate, chosen for some acceptable RAM vs accuracy tradeoff by the admin and
their spool size. The bloom filter is created with a much more
efficient streaming read of the entire history file. In the case of a
false positive (bloom claims article in history, but expire removed it),
it gets cleaned up on the next expireover run (entropy introduced by
hash churn make this probabilistic reality). On my system this is a
1100x improvement, 2 weeks down to 18 minutes.

expire is a bit more involved. We don't want the same bloom tradeoff, exactness trades a bit of RAM (pending deletes) for totally eliminating unnecessary disk I/O for verification. If we keep a tombstone log of
articles expireover deletes, and any cancels, and we trust the state of
the spool, we can eliminate all speculative work and only deal with the
actual history cleanup needed. On my system this will net a 2800x improvement, 2 days down to sub minute.

With the expire tombstone, I can make 'nnrpdcheckart' an extremely
efficient hash lookup, so OVER and friends don't need to check every
single article's existence and only clean any pending tombstone entries.

Both of these suppose integrity of the spool; if you 'rm' articles
instead of using 'sm -r' you'll need to do a patrol expire to clean up
your spool; expireover builds from history so it would see the new
history state the next time it runs. The first time expire runs it wont
have a tombstone so it will 'upgrade' to a known state cleaning any
issues.. a gap is only if you decide to manually botch the spool after that.

You'd avoid the worst of some of this with CNFS, but the unlimited
retention case gets a little awkward, and tradspool should scale to a
text feed with over a billion articles with these changes on modern
hardware.

The ovsqlite direct reader mode allows each nnrpd to open the sqlite DB directly, in read only mode. This eliminates costly IPC and exclusive
locking to the ovsqlite-server which is now only concerned with innd.
The WAL file also helps with writes as they turn into streaming I/O and
the main DB does not create the same level of synchronous write load.
This is conceptually similar to how BDB used to work.

I have these gated on the FreeBSD news/inn-current port behind an
EXPERIMENTAL flag (expire tombstones will come once I'm satisfied).

I'd be curious to hear any feedback or reviews from other inn admins
that can test the patch set.

Regards,
Kevin
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Hi Kevin,

ovsqlite: Add direct reader mode for nnrpd with WAL
expireover: Add bloom filter for fast history existence checks
expire: skip per-article SMretrieve via cancel tombstone log

The general theme of all three is to reduce random or sync I/O.-a In the case of expireover and expire, the general problem is somewhat similar
but the root cause and fix are fairly different.

Many thanks for these very useful improvements to expiration and
ovsqlite concurrent reading.
expire and expireover run way faster (x1100 for expire and x2800 for expireover on your system !!) thanks to your optimizations. Some people
have been complaining with their slowness from a long time, and
hopefully you managed to find out how to improve that a lot and
contributed a patch. It will be integrated to the next release.

On my system this is a 1100x improvement, 2 weeks down to 18 minutes.

Gosh!

On my system this will net a 2800x improvement, 2 days down to sub minute.

Re-gosh!

Both of these suppose integrity of the spool; if you 'rm' articles
instead of using 'sm -r' you'll need to do a patrol expire to clean up
your spool; expireover builds from history so it would see the new
history state the next time it runs.-a The first time expire runs it wont have a tombstone so it will 'upgrade' to a known state cleaning any
issues.. a gap is only if you decide to manually botch the spool after
that.

Worth mentioning.

I'd be curious to hear any feedback or reviews from other inn admins
that can test the patch set.

If anyone could test the provided patches, do not hesitate to do so in
order to eventually catch possible nits before the next release (in June).
--
Julien |eLIE

-2-a|etre |+u ne p|Ns |-tre, telle est l|N questi|+nrCa-a-+ (Ker|+zen)

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/10/26 01:38, Julien |eLIE wrote:

Hi Kevin,

ovsqlite: Add direct reader mode for nnrpd with WAL
expireover: Add bloom filter for fast history existence checks
expire: skip per-article SMretrieve via cancel tombstone log

The general theme of all three is to reduce random or sync I/O.-a In
the case of expireover and expire, the general problem is somewhat
similar but the root cause and fix are fairly different.

Many thanks for these very useful improvements to expiration and
ovsqlite concurrent reading.
expire and expireover run way faster (x1100 for expire and x2800 for expireover on your system !!) thanks to your optimizations.-a Some people have been complaining with their slowness from a long time, and
hopefully you managed to find out how to improve that a lot and
contributed a patch.-a It will be integrated to the next release.

On my system this is a 1100x improvement, 2 weeks down to 18 minutes.

Gosh!

On my system this will net a 2800x improvement, 2 days down to sub
minute.

Re-gosh!

Both of these suppose integrity of the spool; if you 'rm' articles
instead of using 'sm -r' you'll need to do a patrol expire to clean up
your spool; expireover builds from history so it would see the new
history state the next time it runs.-a The first time expire runs it
wont have a tombstone so it will 'upgrade' to a known state cleaning
any issues.. a gap is only if you decide to manually botch the spool
after that.

Worth mentioning.

I'd be curious to hear any feedback or reviews from other inn admins
that can test the patch set.

If anyone could test the provided patches, do not hesitate to do so in
order to eventually catch possible nits before the next release (in June).

I'll pause to save some energy to fix any feedback and problems if they
arise.

And these fix my most notable issues for csiph.com. But I've wondered a
bit about what might be next. It seems like some sites use Diablo
primarily for transit. And commercial sites might use Tornado or some
home grown system (maybe we should do a NNTP Server and Capabilities
survey of top1000?).

I'm unfamiliar with Diablo but took a brief tour of the code to see what
it does. It's (also :)) archaic but would have been advanced in the
time it was conceived.. select() and aio, more efficient feeders, dns, readers, clever file formats mostly mmaped. aio is not a great fit for
Linux due to glibc. There's nothing in particular that would be a good
model or forklift for inn because network programming changed a lot once kqueue and epoll entered a couple years later.

nnrpd would be somewhat easy to make more efficient across a few
dimensions. The least code churn might be integrating poll() directly
in the per process forks and adding some eventing to deal with NNTP
pipelining and I/O and DNS. The main benefit would be building up some
I/O parallelism, all storage devices benefit from that and you smooth
out the latency of hot vs cold I/Os through the page cache (or
ARC/L2ARC) and actual storage devices. Data movement could optionally
use sendfile and even KTLS sendfile on Linux and FreeBSD.

A step further would be to actually multiplex clients. Instead of
forking per client, one or more nnrpds multiplex clients across an event
loop, usually kqueue/epoll/devpoll but could fall back to poll for wide compatibility. The decision tree would be somewhat influenced by what
set of OSes does modern inn still compile on, did any cleanup eject
1990s era unix?

Similar opportunities likely present in innd/innfeeder but I am not
familiar with this code so far.

File formats would be another consideration I am not familiar enough to evaluate right now but the network and disk I/O stuff interests me more.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Hi Kevin,

I've wondered a bit about what might be next.

Thanks for asking and being motivated to enhance and improve INN!

network programming changed a lot once kqueue and epoll entered a couple years later.

Yes, it would be worth using libevent with INN. It would permit to fix
the long-standing bug that innd does not honour DNS TTLs (https://github.com/InterNetNews/inn/issues/89) as epoll provides
asynchronous non-blocking calls to do that.
Also, implementing TLS and zlib (COMPRESS) support in innd and innfeed
would be very useful. People often ask for TLS (for privacy reasons)
and find it complicated to correctly make stunnel or tcpwrappers work
with innd.

nnrpd would be somewhat easy to make more efficient across a few dimensions.-a The least code churn might be integrating poll() directly
in the per process forks and adding some eventing to deal with NNTP pipelining and I/O and DNS.-a The main benefit would be building up some
I/O parallelism, all storage devices benefit from that and you smooth
out the latency of hot vs cold I/Os through the page cache (or ARC/
L2ARC) and actual storage devices.

I agree.

Similar opportunities likely present in innd/innfeeder but I am not
familiar with this code so far.

Sure, there also are improvements to do for them!

File formats would be another consideration I am not familiar enough to evaluate right now but the network and disk I/O stuff interests me more.

You can find some tickets about storage and file formats in the GitHub tracker. The most "urgent" would be the one about the Y2038 issue for timecaf:
timecaf disk format uses time_t (Y2038 issue for 64-bit time_t
transition on 32-bit archs)
https://github.com/InterNetNews/inn/issues/292

Normally, the rest of INN is not affected by the Y2038 issue (but not
100% sure - at least, I did not find other suspicious uses of time_t).


Another useful thing to work on is to make easier how to keep both the
active file and the newsgroups descriptions synchronized and up-to-date.
It is somewhat a pain for newcomers to understand what to do and how
to achieve it, and naturally also in the long-term how to keep it
up-to-date. We have Perl utilities to clean and merge newsgroups files
in https://github.com/InterNetNews/inn/issues/39 but they do not cover
all the cases the news admins usually want. The ideal would be that
actsync take care of everything, or that we have a wrapper which calls
several tools.
More generally, what eases the installation of INN for newcomers would
be helpful.

Maybe revisiting the paths where all INN stuff is installed could
interest you as a package maintainer :)
Support installation in FHS paths
https://github.com/InterNetNews/inn/issues/43

Another suggestion as you seem to like expiration/orphan stuff:
tradspool and timehash articles not deleted after rmgroup
https://github.com/InterNetNews/inn/issues/61


You will also find some interesting reworks Russ had in mind in the TODO
file at the root of the INN tree.
Naturally, feel free to pick up anything you are interested in and with
what you take fun working on it.
--
Julien |eLIE

-2-aCe vieux forban d'Asthmatix, il ne manquait pas d'air-a!-a-+ (Ast|-rix)

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/10/2026 8:59 PM, Kevin Bowling wrote:

Data movement could optionally use sendfile and even KTLS sendfile on Linux and FreeBSD.

So, this gave me an interesting thought earlier this evening, as I
actually do use sendfile() a lot (it's not very portable, so I have
wrappers around sendfile, copy_file_range, splice, and other fun functions).

I was reading the requirement for :bytes earlier today and noted that it specifically does not include dot-stuffing characters, and I recalled
then that in SMTP and NNTP, I strip out any dot-stuffed lines as I
thought was convention. However, when *sending* the articles to a
client, I have just been using ~sendfile, for efficiency. I realize now
this is illegal, since I'm not adding the dot-stuffing back.

Taking a look at INN, I noticed it counts the number of dot-stuffed
lines and subtracts it from the value it provides for :bytes, which is
what the RFC says to do. But it looks like INN doesn't remove the
dot-stuffing when receiving an article. This actually seems like the
more elegant way to do things; if you never remove the dot-stuffing, you
don't need to add it back, and then you can do efficient things like use sendfile rather than going line by line. So in my case, I think I don't
need to care about receiving dot-stuffed lines since I can store them in
the spool and just have it be transparent.

On the other hand, this wouldn't work for SMTP, since with
BDAT/CHUNKING, you may not need to do dot-stuffing.

I guess what I'm trying to say is that for SMTP, because there are
multiple ways to send/receive messages, it makes sense to store the
messages in the "queue" normalized (without dot-stuffing), but since
there's only one way in NNTP, it makes more sense to just store the
stuffed lines in the spool directly. Is this about the gist of it?

(And implicitly, I'm assuming that in the future, it would be hard to
support other extensions like SMTP has since I would assume most news
server are storing the dots in the spool; of course, such extensions are probably of little value to NNTP considering most articles are small).
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/12/26 19:46, InterLinked wrote:

On 5/10/2026 8:59 PM, Kevin Bowling wrote:

Data movement could optionally use sendfile and even KTLS sendfile on
Linux and FreeBSD.

So, this gave me an interesting thought earlier this evening, as I
actually do use sendfile() a lot (it's not very portable, so I have
wrappers around sendfile, copy_file_range, splice, and other fun
functions).

I was reading the requirement for :bytes earlier today and noted that it specifically does not include dot-stuffing characters, and I recalled
then that in SMTP and NNTP, I strip out any dot-stuffed lines as I
thought was convention. However, when *sending* the articles to a
client, I have just been using ~sendfile, for efficiency. I realize now
this is illegal, since I'm not adding the dot-stuffing back.

Taking a look at INN, I noticed it counts the number of dot-stuffed
lines and subtracts it from the value it provides for :bytes, which is
what the RFC says to do. But it looks like INN doesn't remove the dot- stuffing when receiving an article. This actually seems like the more elegant way to do things; if you never remove the dot-stuffing, you
don't need to add it back, and then you can do efficient things like use sendfile rather than going line by line. So in my case, I think I don't
need to care about receiving dot-stuffed lines since I can store them in
the spool and just have it be transparent.

On the other hand, this wouldn't work for SMTP, since with BDAT/
CHUNKING, you may not need to do dot-stuffing.

I guess what I'm trying to say is that for SMTP, because there are
multiple ways to send/receive messages, it makes sense to store the
messages in the "queue" normalized (without dot-stuffing), but since
there's only one way in NNTP, it makes more sense to just store the
stuffed lines in the spool directly. Is this about the gist of it?

(And implicitly, I'm assuming that in the future, it would be hard to support other extensions like SMTP has since I would assume most news
server are storing the dots in the spool; of course, such extensions are probably of little value to NNTP considering most articles are small).

Can you post some example transactions showing your thoughts?

One thing that comes to mind is probably to synthesize the header(s) and sendfile at an offset of the body. The synthesized part could be stack allocated. Likewise any trailer if 'wireformat' is off.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/12/2026 11:30 PM, Kevin Bowling wrote:

On 5/12/26 19:46, InterLinked wrote:

I guess what I'm trying to say is that for SMTP, because there are
multiple ways to send/receive messages, it makes sense to store the
messages in the "queue" normalized (without dot-stuffing), but since
there's only one way in NNTP, it makes more sense to just store the
stuffed lines in the spool directly. Is this about the gist of it?

(And implicitly, I'm assuming that in the future, it would be hard to
support other extensions like SMTP has since I would assume most news
server are storing the dots in the spool; of course, such extensions
are probably of little value to NNTP considering most articles are
small).

Can you post some example transactions showing your thoughts?

One thing that comes to mind is probably to synthesize the header(s) and sendfile at an offset of the body.-a The synthesized part could be stack allocated.-a Likewise any trailer if 'wireformat' is off.

That is similar to what I'm talking about, though I think wireformat in
INN is just line endings (whether to store the CR or not). I was talking
about storing the leading dot in dot-stuffed lines in the spool (same
idea really). I believe that INN does this unconditionally without any
way to disable it... because why would you want to? Unlike SMTP, it does
not appear to me there are any use cases in NNTP (currently) where it
would be useful to normalize them out of the message.

In SMTP, you may or may not need to dot-stuff depending on how the
message gets sent, so there, it seems logical to store messages with the
dot removed.

But suppose NNTP ever had an extension like CHUNKING with a BDAT
command. (I don't really see such a thing being useful, outside of
binary groups maybe, but let's suppose.) Then there could be cases where
you wouldn't send articles dot-stuffed, but that would vary by
transmission and in that case, maybe it would make sense to normalize
the message, so you could frame it in different ways. And yeah, you
could use sendfile for parts of the message, but not all of it and there
would be more parsing involved.

There are a lot of potentially useful extensions in SMTP and IMAP that
could be useful in NNTP, but admittedly I don't see this being a likely scenario.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

InterLinked <nntp@phreaknet.org> writes:

That is similar to what I'm talking about, though I think wireformat in
INN is just line endings (whether to store the CR or not). I was talking about storing the leading dot in dot-stuffed lines in the spool (same idea really).

wireformat controls both in INN. If you set wireformat to false with
tradspool, INN will remove the dot stuffing when storing articles as well
as changing the line endings.

I believe that INN does this unconditionally without any way to disable
it... because why would you want to?

For compatibility with external programs that want to look at the articles
in a tradspool spool directly without using INN's storage library. Back in
the day, there used to be some of those. Less common now!
--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/9/26 16:26, Kevin Bowling wrote:

Hi,

I've been looking at inn as I run it:-a tradspool/ovsqlite on FreeBSD+ZFS with unlimited retention and trying to fix some scaling issues.

ovsqlite: Add direct reader mode for nnrpd with WAL https://github.com/InterNetNews/inn/pull/338

expireover: Add bloom filter for fast history existence checks https://github.com/InterNetNews/inn/pull/339

expire: skip per-article SMretrieve via cancel tombstone log https://github.com/InterNetNews/inn/pull/340

The first two have been running on csiph.com for a few days without
issue.-a I haven't yet deployed the last one yet but will publish results
in the PR when I do.

I am running the tombstone patch as well now and pushed a couple minor cleanups to the PR.

One obvious in hindsight benefit to all this is you'll no longer blow
out your page cache/ARC for nightly spool maintenance. So aside from
the maintenance itself being faster, everything else the system is doing
is potentially more efficient.

The general theme of all three is to reduce random or sync I/O.-a In the case of expireover and expire, the general problem is somewhat similar
but the root cause and fix are fairly different.

expireover checks history for every article and this turns into a very
slow and random serial I/O for every article in the history DB.-a The new option is to use a bloom filter with a configurable false positive rate, chosen for some acceptable RAM vs accuracy tradeoff by the admin and
their spool size.-a The bloom filter is created with a much more
efficient streaming read of the entire history file.-a In the case of a false positive (bloom claims article in history, but expire removed it),
it gets cleaned up on the next expireover run (entropy introduced by
hash churn make this probabilistic reality).-a On my system this is a
1100x improvement, 2 weeks down to 18 minutes.

expire is a bit more involved.-a We don't want the same bloom tradeoff, exactness trades a bit of RAM (pending deletes) for totally eliminating unnecessary disk I/O for verification.-a If we keep a tombstone log of articles expireover deletes, and any cancels, and we trust the state of
the spool, we can eliminate all speculative work and only deal with the actual history cleanup needed.-a On my system this will net a 2800x improvement, 2 days down to sub minute.

With the expire tombstone, I can make 'nnrpdcheckart' an extremely
efficient hash lookup, so OVER and friends don't need to check every
single article's existence and only clean any pending tombstone entries.

Both of these suppose integrity of the spool; if you 'rm' articles
instead of using 'sm -r' you'll need to do a patrol expire to clean up
your spool; expireover builds from history so it would see the new
history state the next time it runs.-a The first time expire runs it wont have a tombstone so it will 'upgrade' to a known state cleaning any
issues.. a gap is only if you decide to manually botch the spool after
that.

You'd avoid the worst of some of this with CNFS, but the unlimited
retention case gets a little awkward, and tradspool should scale to a
text feed with over a billion articles with these changes on modern hardware.

The ovsqlite direct reader mode allows each nnrpd to open the sqlite DB directly, in read only mode.-a This eliminates costly IPC and exclusive locking to the ovsqlite-server which is now only concerned with innd.
The WAL file also helps with writes as they turn into streaming I/O and
the main DB does not create the same level of synchronous write load.
This is conceptually similar to how BDB used to work.

I have these gated on the FreeBSD news/inn-current port behind an EXPERIMENTAL flag (expire tombstones will come once I'm satisfied).

I'd be curious to hear any feedback or reviews from other inn admins
that can test the patch set.

Regards,
Kevin

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On May 20, 2026 at 1:15:51rC>AM CDT, "Kevin Bowling" <kevin.bowling@kev009.com> wrote:

I'd be curious to hear any feedback or reviews from other inn admins
that can test the patch set.

Regards,
Kevin

Hey Kevin,

My news.blueworldhosting.com spool is tradspool with ovsqlite... I currently run news.daily with noexpire and noexpireover, since I don't wish to expire anything. That said, I do have over 350 million articles on news.blueworldhosting.com:

# wc -l history
350579886 history

I'm willing to try out all the new features you've help implement to provide more real life metrics.

Thanks for all of the effort you've put into improving INN.

Cheers,

Jesse
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/21/26 17:58, Jesse Rehmer wrote:

On May 20, 2026 at 1:15:51rC>AM CDT, "Kevin Bowling" <kevin.bowling@kev009.com>
wrote:

I'd be curious to hear any feedback or reviews from other inn admins
that can test the patch set.

Regards,
Kevin

Hey Kevin,

My news.blueworldhosting.com spool is tradspool with ovsqlite... I currently run news.daily with noexpire and noexpireover, since I don't wish to expire anything. That said, I do have over 350 million articles on news.blueworldhosting.com:

# wc -l history
350579886 history

I'm willing to try out all the new features you've help implement to provide more real life metrics.

I think the ovsqlite changes Julien just landed would be most useful for
you, the WAL thing might really help with your writer benchmarks
especially on ZFS where the correctness guarantees add a fair bit of
latency.

I'm not sure how noexpire and noexpireover interact with the history dbz maintenance, presumably there is still some work in cancel handling? If
so the expire tombstones might be relevant, and it helps with readers if
you run with 'nnrpdcheckart'

Thanks for all of the effort you've put into improving INN.

Cheers,

Jesse

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On May 21, 2026 at 8:58:49rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com> wrote:

I think the ovsqlite changes Julien just landed would be most useful for
you, the WAL thing might really help with your writer benchmarks
especially on ZFS where the correctness guarantees add a fair bit of
latency.

I'm lazy, so I'll probably wait for a development snapshot to be posted.

I'm not sure how noexpire and noexpireover interact with the history dbz maintenance, presumably there is still some work in cancel handling? If
so the expire tombstones might be relevant, and it helps with readers if
you run with 'nnrpdcheckart'

Sorry, I should have clarified more, what I meant was that I typically run it that way, but for experimentation, I'd be open to running it prior to updating and again after.

I'm also not sure how my current options interact with the history file. I thought cancels were handled when the cancel is processed, and the overview removal happens immediately with ovsqlite? I could be completely wrong, this
is an area I've not done much more than cursing makehistory requiring a gigantic amount of memory for it to generate an empty and appropriately sized dbz, otherwise it always pukes. For my spool I had to increase RAM to 64GB to generate a 400,000,000 dbz. Under normal operation, my server doesn't need
more than 16GB of RAM.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/21/26 19:15, Jesse Rehmer wrote:

On May 21, 2026 at 8:58:49rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com>
wrote:

I think the ovsqlite changes Julien just landed would be most useful for
you, the WAL thing might really help with your writer benchmarks
especially on ZFS where the correctness guarantees add a fair bit of
latency.

I'm lazy, so I'll probably wait for a development snapshot to be posted.

I'm not sure how noexpire and noexpireover interact with the history dbz
maintenance, presumably there is still some work in cancel handling? If
so the expire tombstones might be relevant, and it helps with readers if
you run with 'nnrpdcheckart'

Sorry, I should have clarified more, what I meant was that I typically run it that way, but for experimentation, I'd be open to running it prior to updating
and again after.

ovsqlite should be in the next snapshop whenever that fires today/tomorrow.

I'm also not sure how my current options interact with the history file. I thought cancels were handled when the cancel is processed, and the overview removal happens immediately with ovsqlite? I could be completely wrong, this is an area I've not done much more than cursing makehistory requiring a gigantic amount of memory for it to generate an empty and appropriately sized dbz, otherwise it always pukes. For my spool I had to increase RAM to 64GB to generate a 400,000,000 dbz. Under normal operation, my server doesn't need more than 16GB of RAM.

How big is your history dbz? I suspect the answer to all this is what
Russ was hinting at, and maybe use sqlite for history so it can do in
place maintenance.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On May 21, 2026 at 9:29:57rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com> wrote:

ovsqlite should be in the next snapshop whenever that fires today/tomorrow.

Excellent!

I'm also not sure how my current options interact with the history file. I >> thought cancels were handled when the cancel is processed, and the overview >> removal happens immediately with ovsqlite? I could be completely wrong, this >> is an area I've not done much more than cursing makehistory requiring a
gigantic amount of memory for it to generate an empty and appropriately sized
dbz, otherwise it always pukes. For my spool I had to increase RAM to 64GB to
generate a 400,000,000 dbz. Under normal operation, my server doesn't need >> more than 16GB of RAM.

How big is your history dbz? I suspect the answer to all this is what
Russ was hinting at, and maybe use sqlite for history so it can do in
place maintenance.

When you say how big is my history dbz, do you mean in file size?

$ ls -l history*
-rw-r--r-- 1 news news 34350282002 May 21 22:17 history
-rw-r--r-- 1 news news 52 May 21 22:16 history.dir
-rw-r--r-- 1 news news 4500000000 May 21 22:16 history.hash
-rw-r--r-- 1 news news 6000000000 May 21 22:16 history.index
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On May 21, 2026 at 9:29:57rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com> wrote:

How big is your history dbz? I suspect the answer to all this is what
Russ was hinting at, and maybe use sqlite for history so it can do in
place maintenance.

I must have missed Russ's comment about using sqlite for the history file, but my personal opinion is the history operations are the largest bottleneck when running a larger scale INN instance. With CNFS and ovsqlite expiration operations are not so terrible. Room for improvement sure, but the main thing I've noticed when feeding hundreds of millions of articles between servers is the history writes are the biggest overall bottleneck I encounter on a regular basis. I have messed with the various knobs related to this and never got the throughput I wanted, or what Diablo is capable of.

Of course, I'm not doing regular expire operations, but if I recall correctly, when I was doing them it could take days to complete. That did create issues for users.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/21/26 20:20, Jesse Rehmer wrote:

On May 21, 2026 at 9:29:57rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com>
wrote:

ovsqlite should be in the next snapshop whenever that fires today/tomorrow.

Excellent!

I'm also not sure how my current options interact with the history file. I >>> thought cancels were handled when the cancel is processed, and the overview >>> removal happens immediately with ovsqlite? I could be completely wrong, this
is an area I've not done much more than cursing makehistory requiring a
gigantic amount of memory for it to generate an empty and appropriately sized
dbz, otherwise it always pukes. For my spool I had to increase RAM to 64GB to
generate a 400,000,000 dbz. Under normal operation, my server doesn't need >>> more than 16GB of RAM.

How big is your history dbz? I suspect the answer to all this is what
Russ was hinting at, and maybe use sqlite for history so it can do in
place maintenance.

When you say how big is my history dbz, do you mean in file size?

$ ls -l history*
-rw-r--r-- 1 news news 34350282002 May 21 22:17 history
-rw-r--r-- 1 news news 52 May 21 22:16 history.dir
-rw-r--r-- 1 news news 4500000000 May 21 22:16 history.hash

Effectively this hash amount (plus dynamic sizing) has to be in RAM with
the current algorithm and data structure. Does that sound about right,
or do you know what the process was peaking at in terms of memory (RSS)?

-rw-r--r-- 1 news news 6000000000 May 21 22:16 history.index

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Kevin Bowling <kevin.bowling@kev009.com> writes:

How big is your history dbz? I suspect the answer to all this is what
Russ was hinting at, and maybe use sqlite for history so it can do in
place maintenance.

I would be very curious to see if that would work. The dbz code is fairly optimized, but it's also very old. I'm sure there are better algorithms,
but I don't know if a general-purpose database can go as fast. If it can,
that would offload a lot of complexity.

History is one of those cases where there's probably some better algorithm
that would be optimized for that specific purpose, but figuring out those algorithms is only fun for a particular set of people and then it's hard
to find people to maintain them. :)
--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/21/26 20:24, Jesse Rehmer wrote:

On May 21, 2026 at 9:29:57rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com>
wrote:

How big is your history dbz? I suspect the answer to all this is what
Russ was hinting at, and maybe use sqlite for history so it can do in
place maintenance.

I must have missed Russ's comment about using sqlite for the history file, but
my personal opinion is the history operations are the largest bottleneck when running a larger scale INN instance. With CNFS and ovsqlite expiration operations are not so terrible. Room for improvement sure, but the main thing I've noticed when feeding hundreds of millions of articles between servers is the history writes are the biggest overall bottleneck I encounter on a regular
basis. I have messed with the various knobs related to this and never got the throughput I wanted, or what Diablo is capable of.

It was in an unrelated thread with InterLinked about maybe using
databases instead of home grown formats.

Of course, I'm not doing regular expire operations, but if I recall correctly,
when I was doing them it could take days to complete. That did create issues for users.

It should be manageable with my current patch stack to run expire with
your current count but writing out a new history at this scale every
night is a bit wasteful.

I will take a look at history/dbz and see if there are any easy wins and
do some napkin math to see if sqlite would be viable for the use case.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/21/26 22:17, Russ Allbery wrote:

Kevin Bowling <kevin.bowling@kev009.com> writes:

How big is your history dbz? I suspect the answer to all this is what
Russ was hinting at, and maybe use sqlite for history so it can do in
place maintenance.

I would be very curious to see if that would work. The dbz code is fairly optimized, but it's also very old. I'm sure there are better algorithms,
but I don't know if a general-purpose database can go as fast. If it can, that would offload a lot of complexity.

They occupy two extremes in a sense; DBZ is eventually a perfect hashing
CDB. SQLite is a B-tree storage engine with an expressive (and
relatively expensive) query language on top.

DBZ:
Fast lookup
Optimal disk footprint (modulo compression and encoding schemes)
Good write performance (fwrite, fflush then batched dbz maintenance)
Most expensive maintenance (entire file rewrite)
Not crash safe, fragile

SQLite:
Slower lookup (probably about a magnitude, microsecond to 10s of
microseconds)
Less dense disk footprint (probably 3-5x, still negligible compared to
spool)
Good write performance (WAL may greatly outperform DBZ on chaotic
working sets even though DBZ is in theory more efficient)
Extremely fast maintenance (N expires turn into N row deletes, VACUUM eventually or scheduled for compaction)
One of the most robust code bases ever against known failures

The biggest justification is probably operational, all the quirks and
oddities of history basically turn into simple SQL statements (remember, expire, etc).

I'm not certain how a hissqlite would interact with my recent patch: in particular being able to efficiently read the whole history enchilada
into a bloom filter with custom hash function is perfection for
expireover. I think sqlite has a bloom filter function internally so
there may be a way to hook in and do something similar but it'll
probably need to progress to a prototype to figure out all the details.

History is one of those cases where there's probably some better algorithm that would be optimized for that specific purpose, but figuring out those algorithms is only fun for a particular set of people and then it's hard
to find people to maintain them. :)

It strikes me that LMDB would probably be a more direct fit for this,
and if you squint OpenLDAP and inn aren't that different (although inn eventually leafs down to the file server load that sendfile would
naturally fit where LDAP is more of a key/value store). But sqlite is
already well integrated, basically maintenance free, and there is a cost
to bringing in more dependencies.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Il 22/05/2026 02:58, Jesse Rehmer ha scritto:

anything. That said, I do have over 350 million articles on news.blueworldhosting.com:

# wc -l history
350579886 history

Hi Jesse,

My backend server have more a lot, with CNFS and tradindexed without any trouble (text and binary), and when CNFS-spool running out-of-space, a
new CNFS via cronjob was created before that:

news@fontana ~ $ wc -l /var/lib/news/history
314671106276 /var/lib/news/history
(2,8TB)

My free server with tradspool and tradindexed (but have only 8-9 year's
of text history)

news@paganini:~$ wc -l /var/lib/news/history
98363376 /var/lib/news/history
(7GB)

But this configuration cause me a lot of trouble due to the old setup
AND old hardware... especially for old hardware...

Why I'm using tradindexed and not other method? tradindexed it's better
on restore time in case of catastrophyc crash, and better tested than
other's.

I'm currently working on writing an NNTP proxy that can support multiple backends and other cool features. As soon as I have a stable build, I'll publish something, especially since I'm thinking of shutting down
paganini and retiring him due to his age limit.

To get back to the topic, I'd also be interested in "plumping up" my
history with articles, so Jesse if you'd care to share with me what you
find, it would be greatly appreciated. (Or I'll write to you privately)



Sincerely
--
Ivo Gandolfo


--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/22/26 10:43 AM, Ivo Gandolfo wrote:

To get back to the topic, I'd also be interested in "plumping up" my
history with articles, so Jesse if you'd care to share with me what you find, it would be greatly appreciated. (Or I'll write to you privately)

On that subject, I've noticed that the Google groups .mbox files loaded
into Blueworld hosting are incomplete. It's not obvious unless you
actually load all headers, make a list of threads, and *look at them*,
but when you examine the threadlist you'll see there's a lot of
discussions missing.

And it's not just XNA articles or something similar. It looks like
whoever loaded these mbox files made a mistake that resulted in
thousands of messages being left out per newsgroup.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Kevin Bowling <kevin.bowling@kev009.com> writes:

They occupy two extremes in a sense; DBZ is eventually a perfect hashing
CDB.

Yeah, I have in the past considered actually using CDB for the older
entries, but the complexity didn't seem worth it.

The biggest justification is probably operational, all the quirks and oddities of history basically turn into simple SQL statements (remember, expire, etc).

Yes.

It strikes me that LMDB would probably be a more direct fit for this,
and if you squint OpenLDAP and inn aren't that different (although inn eventually leafs down to the file server load that sendfile would
naturally fit where LDAP is more of a key/value store). But sqlite is
already well integrated, basically maintenance free, and there is a cost
to bringing in more dependencies.

Oh, that's an interesting idea. But yes, adding yet another database isn't
all that appealing.
--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On May 22, 2026 at 11:44:23rC>AM CDT, "D Finnigan" <dog_cow@macgui.com> wrote:

On 5/22/26 10:43 AM, Ivo Gandolfo wrote:

To get back to the topic, I'd also be interested in "plumping up" my
history with articles, so Jesse if you'd care to share with me what you
find, it would be greatly appreciated. (Or I'll write to you privately)

On that subject, I've noticed that the Google groups .mbox files loaded
into Blueworld hosting are incomplete. It's not obvious unless you
actually load all headers, make a list of threads, and *look at them*,
but when you examine the threadlist you'll see there's a lot of
discussions missing.

And it's not just XNA articles or something similar. It looks like
whoever loaded these mbox files made a mistake that resulted in
thousands of messages being left out per newsgroup.

What's on the publicly available on news.blueworldhosting.com is minimal compared to what I have in totality. I worked with Billy G while he was still around to pump ~7TB of articles into another server. I have yet to combine everything together and re-order article numbering chronologically.

I cannot claim to have 'everything', but there are several terabytes of articles that aren't on news.blueworldhosting.com, yet.

Once I get the entire archive in a sane manner I will swing news.blueworldhosting.com to that server.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Hi Jesse,

ovsqlite should be in the next snapshop whenever that fires today/tomorrow.

Looks like the official site

https://downloads.isc.org/isc/inn/snapshots/

is 1 day behind. It is a mirror of

https://archives.eyrie.org/software/inn/snapshots/

where you can already download inn-2.8-20260521.tar.gz or inn-CURRENT-20260522.tar.gz which are both snapshots containing WAL and
direct reader mode contributed by Kevin.
--
Julien |eLIE

-2-aMettez-vous de profil avec les |-paules de face et ne bougeons plus je
vous prie.-a-+ (Ast|-rix)

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/22/26 1:06 PM, Jesse Rehmer wrote:

On May 22, 2026 at 11:44:23rC>AM CDT, "D Finnigan" <dog_cow@macgui.com> wrote:

On 5/22/26 10:43 AM, Ivo Gandolfo wrote:

To get back to the topic, I'd also be interested in "plumping up" my
history with articles, so Jesse if you'd care to share with me what you
find, it would be greatly appreciated. (Or I'll write to you privately)

On that subject, I've noticed that the Google groups .mbox files loaded
into Blueworld hosting are incomplete. It's not obvious unless you
actually load all headers, make a list of threads, and *look at them*,
but when you examine the threadlist you'll see there's a lot of
discussions missing.

And it's not just XNA articles or something similar. It looks like
whoever loaded these mbox files made a mistake that resulted in
thousands of messages being left out per newsgroup.

What's on the publicly available on news.blueworldhosting.com is minimal compared to what I have in totality. I worked with Billy G while he was still around to pump ~7TB of articles into another server. I have yet to combine everything together and re-order article numbering chronologically.

I cannot claim to have 'everything', but there are several terabytes of articles that aren't on news.blueworldhosting.com, yet.

Once I get the entire archive in a sane manner I will swing news.blueworldhosting.com to that server.

That's all great.

But what I'm talking about is that for the older articles, only the
articles that were cross-posted are showing up in the newsgroups. If an article was only posted to that single group, it doesn't appear.

Check it out. If I pick a few Big-8 groups at random and look through
the threads from the mid- to late-90s, I find there are *only*
cross-posted articles. But maybe that's just an artifact of not having
loaded everything you've got.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On May 22, 2026 at 1:55:28rC>PM CDT, "D Finnigan" <dog_cow@macgui.com> wrote:

Check it out. If I pick a few Big-8 groups at random and look through
the threads from the mid- to late-90s, I find there are *only*
cross-posted articles. But maybe that's just an artifact of not having
loaded everything you've got.

Yes, the 90s is completely full of holes and was from an attempt that was stopped and moved to another server with an array of SSDs, CNFS buffers, and a shitload of RAM so we could move faster. I've said it many times, the current spool available to the public is full of holes.

I will begin to make progress consolidating everything I have and getting it fed to a new server over the weekend. That said, I'll never claim to have everything.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/21/26 22:19, Kevin Bowling wrote:

On 5/21/26 20:24, Jesse Rehmer wrote:

On May 21, 2026 at 9:29:57rC>PM CDT, "Kevin Bowling"
<kevin.bowling@kev009.com>
wrote:

How big is your history dbz?-a I suspect the answer to all this is what
Russ was hinting at, and maybe use sqlite for history so it can do in
place maintenance.

I must have missed Russ's comment about using sqlite for the history
file, but
my personal opinion is the history operations are the largest
bottleneck when

After looking around in the code, you are very correct.

dbz's putcore() is doing msync(MS_ASYNC) over the entire .index/.hash
mmap on every sync, regardless of how many entries actually changed
(dbz.c L1338-1342). So cost scales with table size, not with write count.

That would be completely eliminated with atomic commit of a WAL or CoW
DB. So there's a good chance SQLite would scale better than hisv6,
enough that I think it's worth a prototype.

It will be interesting to see how a B-tree works for the hisgrep lookup
path. In principle it should be fine, especially if the history size
spills past the working set onto disk.

running a larger scale INN instance. With CNFS and ovsqlite expiration
operations are not so terrible. Room for improvement sure, but the
main thing
I've noticed when feeding hundreds of millions of articles between
servers is
the history writes are the biggest overall bottleneck I encounter on a
regular
basis. I have messed with the various knobs related to this and never
got the
throughput I wanted, or what Diablo is capable of.

It was in an unrelated thread with InterLinked about maybe using
databases instead of home grown formats.

Of course, I'm not doing regular expire operations, but if I recall
correctly,
when I was doing them it could take days to complete. That did create
issues
for users.

It should be manageable with my current patch stack to run expire with
your current count but writing out a new history at this scale every
night is a bit wasteful.

I will take a look at history/dbz and see if there are any easy wins and
do some napkin math to see if sqlite would be viable for the use case.

--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 5/9/2026 7:26 PM, Kevin Bowling wrote:

Hi,

I've been looking at inn as I run it:-a tradspool/ovsqlite on FreeBSD+ZFS with unlimited retention and trying to fix some scaling issues.

ovsqlite: Add direct reader mode for nnrpd with WAL https://github.com/InterNetNews/inn/pull/338

expireover: Add bloom filter for fast history existence checks https://github.com/InterNetNews/inn/pull/339

expire: skip per-article SMretrieve via cancel tombstone log https://github.com/InterNetNews/inn/pull/340

The first two have been running on csiph.com for a few days without
issue.-a I haven't yet deployed the last one yet but will publish results
in the PR when I do.

The general theme of all three is to reduce random or sync I/O.-a In the case of expireover and expire, the general problem is somewhat similar
but the root cause and fix are fairly different.

expireover checks history for every article and this turns into a very
slow and random serial I/O for every article in the history DB.-a The new option is to use a bloom filter with a configurable false positive rate, chosen for some acceptable RAM vs accuracy tradeoff by the admin and
their spool size.-a The bloom filter is created with a much more
efficient streaming read of the entire history file.-a In the case of a false positive (bloom claims article in history, but expire removed it),
it gets cleaned up on the next expireover run (entropy introduced by
hash churn make this probabilistic reality).-a On my system this is a
1100x improvement, 2 weeks down to 18 minutes.

expire is a bit more involved.-a We don't want the same bloom tradeoff, exactness trades a bit of RAM (pending deletes) for totally eliminating unnecessary disk I/O for verification.-a If we keep a tombstone log of articles expireover deletes, and any cancels, and we trust the state of
the spool, we can eliminate all speculative work and only deal with the actual history cleanup needed.-a On my system this will net a 2800x improvement, 2 days down to sub minute.

I've been thinking about this recently as part of some of other work I
have been doing and so in the past few days, have had a renewed interest
in your optimizations.

A week or two ago, I was trying to figure out how to suck articles in a
manner that preserves their ordering across all groups in which an
article appears. In the end, I think there is really only one solution.
Save all the message IDs + date header + Xref header for the articles
you want to suck, sort them by Xref header with a fallback to Date, and
then suck the articles in that order. (This is quite a bit slower than
just going group by group, since you need to change the GROUP quite
often, though I suspect faster (and nicer) to the server being sucked
from than requesting by Message ID.) This won't work if there are a lot
of articles being sucked by a wimpy machine, but fortunately I'm not
operating at the scale.

I've been doing this in two passes, first saving the Message-IDs to a
file and then sucking those, so that operations can be resumed and are idempotent. I check if an article was already in history to avoid asking
for it again. I was surprised that even with a few hundred thousand
articles, when resuming a partial suck, just weeding out all the
duplicates would take many minutes, due to all the history traversals. Naturally, this made me think of using a Bloom filter (and I was
reminded of the optimization you did here and its encouraging results)
and indeed, that did speed that up quite a bit. I chose to retain
accuracy in that case, so if the bloom filter answers yes, I check
history anyways to confirm, but if there are a lot of negative matches
then it goes much faster.

Although I added a Bloom filter for just this one operation, I'm now
toying with the concept of having one in the background to use for all operations. For example, for IHAVE/CHECK/TAKETHIS, at the moment I check history to see if the Message-ID is present. I don't see why I can't
just create a Bloom filter at startup, adding new articles to it as
received, and recreating the filter on expire operations, and in the
case of accepting an article we don't have, then a history scan is no
longer necessary, even with a small Bloom filter (somewhat mediocre
false positive rate).

As I understand it, only recently with your change does INN use Bloom
filters at all, but I'm curious if you have any thoughts about adopting
them for other purposes like this (either in INN, or news software in general). I'm thinking a lot of history related operations could benefit
from them, but maybe there is something too good to be true here...
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 6/23/26 15:14, InterLinked wrote:

On 5/9/2026 7:26 PM, Kevin Bowling wrote:

Hi,

I've been looking at inn as I run it:-a tradspool/ovsqlite on
FreeBSD+ZFS with unlimited retention and trying to fix some scaling
issues.

ovsqlite: Add direct reader mode for nnrpd with WAL
https://github.com/InterNetNews/inn/pull/338

expireover: Add bloom filter for fast history existence checks
https://github.com/InterNetNews/inn/pull/339

expire: skip per-article SMretrieve via cancel tombstone log
https://github.com/InterNetNews/inn/pull/340

The first two have been running on csiph.com for a few days without
issue.-a I haven't yet deployed the last one yet but will publish
results in the PR when I do.

The general theme of all three is to reduce random or sync I/O.-a In
the case of expireover and expire, the general problem is somewhat
similar but the root cause and fix are fairly different.

expireover checks history for every article and this turns into a very
slow and random serial I/O for every article in the history DB.-a The
new option is to use a bloom filter with a configurable false positive
rate, chosen for some acceptable RAM vs accuracy tradeoff by the admin
and their spool size.-a The bloom filter is created with a much more
efficient streaming read of the entire history file.-a In the case of a
false positive (bloom claims article in history, but expire removed
it), it gets cleaned up on the next expireover run (entropy introduced
by hash churn make this probabilistic reality).-a On my system this is
a 1100x improvement, 2 weeks down to 18 minutes.

expire is a bit more involved.-a We don't want the same bloom tradeoff,
exactness trades a bit of RAM (pending deletes) for totally
eliminating unnecessary disk I/O for verification.-a If we keep a
tombstone log of articles expireover deletes, and any cancels, and we
trust the state of the spool, we can eliminate all speculative work
and only deal with the actual history cleanup needed.-a On my system
this will net a 2800x improvement, 2 days down to sub minute.

I've been thinking about this recently as part of some of other work I
have been doing and so in the past few days, have had a renewed interest
in your optimizations.

A week or two ago, I was trying to figure out how to suck articles in a manner that preserves their ordering across all groups in which an
article appears. In the end, I think there is really only one solution.
Save all the message IDs + date header + Xref header for the articles
you want to suck, sort them by Xref header with a fallback to Date, and
then suck the articles in that order. (This is quite a bit slower than
just going group by group, since you need to change the GROUP quite
often, though I suspect faster (and nicer) to the server being sucked
from than requesting by Message ID.) This won't work if there are a lot
of articles being sucked by a wimpy machine, but fortunately I'm not operating at the scale.

I've been doing this in two passes, first saving the Message-IDs to a
file and then sucking those, so that operations can be resumed and are idempotent. I check if an article was already in history to avoid asking
for it again. I was surprised that even with a few hundred thousand articles, when resuming a partial suck, just weeding out all the
duplicates would take many minutes, due to all the history traversals. Naturally, this made me think of using a Bloom filter (and I was
reminded of the optimization you did here and its encouraging results)
and indeed, that did speed that up quite a bit. I chose to retain
accuracy in that case, so if the bloom filter answers yes, I check
history anyways to confirm, but if there are a lot of negative matches
then it goes much faster.

Although I added a Bloom filter for just this one operation, I'm now
toying with the concept of having one in the background to use for all operations. For example, for IHAVE/CHECK/TAKETHIS, at the moment I check history to see if the Message-ID is present. I don't see why I can't
just create a Bloom filter at startup, adding new articles to it as received, and recreating the filter on expire operations, and in the
case of accepting an article we don't have, then a history scan is no
longer necessary, even with a small Bloom filter (somewhat mediocre
false positive rate).

As I understand it, only recently with your change does INN use Bloom filters at all, but I'm curious if you have any thoughts about adopting
them for other purposes like this (either in INN, or news software in general). I'm thinking a lot of history related operations could benefit from them, but maybe there is something too good to be true here...

It is in lib/, depending on the usage it may need to be tweaked slightly
to accept a comparator (different than the MD5-optimization) or offer different function signatures for different uses.

You may find the benchmarks in https://github.com/InterNetNews/inn/pull/349#issuecomment-4760898620 interesting. With that I wouldn't assume history read path is a big bottleneck in most cases without seeing some evidence.

Keep in mind hiscache sits in front of CHECK, which is a HASH variant of
what you might be suggesting.

Lookup might benefit from a cache, but remember it is done by nnrpd so
it scales by processes (and inversely makes cache sharing harder).

I would default to HASH style of implementation (and windowing, like
LRU, to implement small bounded caches) without a good overriding
reason; the Bloom filter is a specific optimization where space
efficiency and totality (in one direction, as you noticed) are both
required.
--- Synchronet 3.22a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On 6/25/2026 12:15 AM, Kevin Bowling wrote:

On 6/23/26 15:14, InterLinked wrote:

On 5/9/2026 7:26 PM, Kevin Bowling wrote:

Hi,

I've been looking at inn as I run it:-a tradspool/ovsqlite on
FreeBSD+ZFS with unlimited retention and trying to fix some scaling
issues.

ovsqlite: Add direct reader mode for nnrpd with WAL
https://github.com/InterNetNews/inn/pull/338

expireover: Add bloom filter for fast history existence checks
https://github.com/InterNetNews/inn/pull/339

expire: skip per-article SMretrieve via cancel tombstone log
https://github.com/InterNetNews/inn/pull/340

The first two have been running on csiph.com for a few days without
issue.-a I haven't yet deployed the last one yet but will publish
results in the PR when I do.

The general theme of all three is to reduce random or sync I/O.-a In
the case of expireover and expire, the general problem is somewhat
similar but the root cause and fix are fairly different.

expireover checks history for every article and this turns into a
very slow and random serial I/O for every article in the history DB.
The new option is to use a bloom filter with a configurable false
positive rate, chosen for some acceptable RAM vs accuracy tradeoff by
the admin and their spool size.-a The bloom filter is created with a
much more efficient streaming read of the entire history file.-a In
the case of a false positive (bloom claims article in history, but
expire removed it), it gets cleaned up on the next expireover run
(entropy introduced by hash churn make this probabilistic reality).
On my system this is a 1100x improvement, 2 weeks down to 18 minutes.

expire is a bit more involved.-a We don't want the same bloom
tradeoff, exactness trades a bit of RAM (pending deletes) for totally
eliminating unnecessary disk I/O for verification.-a If we keep a
tombstone log of articles expireover deletes, and any cancels, and we
trust the state of the spool, we can eliminate all speculative work
and only deal with the actual history cleanup needed.-a On my system
this will net a 2800x improvement, 2 days down to sub minute.

I've been thinking about this recently as part of some of other work I
have been doing and so in the past few days, have had a renewed
interest in your optimizations.

A week or two ago, I was trying to figure out how to suck articles in
a manner that preserves their ordering across all groups in which an
article appears. In the end, I think there is really only one
solution. Save all the message IDs + date header + Xref header for the
articles you want to suck, sort them by Xref header with a fallback to
Date, and then suck the articles in that order. (This is quite a bit
slower than just going group by group, since you need to change the
GROUP quite often, though I suspect faster (and nicer) to the server
being sucked from than requesting by Message ID.) This won't work if
there are a lot of articles being sucked by a wimpy machine, but
fortunately I'm not operating at the scale.

I've been doing this in two passes, first saving the Message-IDs to a
file and then sucking those, so that operations can be resumed and are
idempotent. I check if an article was already in history to avoid
asking for it again. I was surprised that even with a few hundred
thousand articles, when resuming a partial suck, just weeding out all
the duplicates would take many minutes, due to all the history
traversals. Naturally, this made me think of using a Bloom filter (and
I was reminded of the optimization you did here and its encouraging
results) and indeed, that did speed that up quite a bit. I chose to
retain accuracy in that case, so if the bloom filter answers yes, I
check history anyways to confirm, but if there are a lot of negative
matches then it goes much faster.

Although I added a Bloom filter for just this one operation, I'm now
toying with the concept of having one in the background to use for all
operations. For example, for IHAVE/CHECK/TAKETHIS, at the moment I
check history to see if the Message-ID is present. I don't see why I
can't just create a Bloom filter at startup, adding new articles to it
as received, and recreating the filter on expire operations, and in
the case of accepting an article we don't have, then a history scan is
no longer necessary, even with a small Bloom filter (somewhat mediocre
false positive rate).

As I understand it, only recently with your change does INN use Bloom
filters at all, but I'm curious if you have any thoughts about
adopting them for other purposes like this (either in INN, or news
software in general). I'm thinking a lot of history related operations
could benefit from them, but maybe there is something too good to be
true here...

It is in lib/, depending on the usage it may need to be tweaked slightly
to accept a comparator (different than the MD5-optimization) or offer different function signatures for different uses.

At the moment, I'm only thinking about them for Message-IDs / history,
since that's probably the biggest bottleneck in regular usage I can
think of.

You may find the benchmarks in https://github.com/InterNetNews/inn/pull/349#issuecomment-4760898620 interesting.-a With that I wouldn't assume history read path is a big bottleneck in most cases without seeing some evidence.

Yeah, that is interesting. It looks like hisv6 is sort of a flat file
from what I can see although it hashes the Message-IDs, which I do not.
IIRC, an earlier history implementation in INN didn't hash Message-IDs.
I would guess without hashing, the performance speedup for random
lookups would be even greater.

Keep in mind hiscache sits in front of CHECK, which is a HASH variant of what you might be suggesting.

Hmm, yeah, that probably would be even better for that particular case,
given you probably only care about the most recent articles.

At some point though, you will need to check if the Message ID exists in history at all, and a cache can't do that, but a Bloom filter can, at
least if the answer is "no" there you don't need to scan history at all.
So having it as a stopgap method still seems helpful - check the cache,
then the Bloom filter, then scan history.

I will certainly add a small cache for "recent" articles as well, that
would help even more.

The use case I mentioned before here was one where not all the Message
IDs were recent and so a cache would not have been sufficient. It was basically checking an arbitrarily long list of Message IDs to see if
they were in history or not. A Bloom filter sped this up by about 6x in
a test I did, even when I still checked history anyways when the answer
was 'yes', which was good enough for me. (For the edge case if all the articles are in history, there is no speedup, but that was somewhat
unlikely here.)

Lookup might benefit from a cache, but remember it is done by nnrpd so
it scales by processes (and inversely makes cache sharing harder).

Ah, yeah that makes sense then. My news software is a single
multi-threaded process (actually just one part of one), so I think
in-memory structures like Bloom filters or hash tables would both work a
lot better for me since I can keep the filter up to date easily.

I would default to HASH style of implementation (and windowing, like
LRU, to implement small bounded caches) without a good overriding
reason; the Bloom filter is a specific optimization where space
efficiency and totality (in one direction, as you noticed) are both required.

I like Bloom filters for "does this article exist in the entirety of
history at all"? I don't think there is any other way to do that.
Certainly there are other cases where that may not be the most
appropriate question to ask upfront.

Similar to your hash suggestion, the next bottleneck I've stumbled upon
is when doing NEWNEWS, a client gets a bunch of Message-IDs, and then
probably will use OVER or the like to get more details on those
articles. I found on some systems, that was quite slow, even for a small number of articles, since, again, it ended up scanning history for every article to find it from the Message-ID.

Obviously I should already be caching the most recent articles in some
way so I don't need to scan history for them. This is similar enough to
the CHECK cache use case so probably the same thing could be used for
both of them. I'm thinking maybe a fixed size ring buffer with hashes of
the most recent articles, with offsets into the history file.

It would seem to me both for CHECK and NEWNEWS + OVER patterns, a FIFO
style ring buffer makes sense - are you suggesting an LRU policy from
traffic analysis of article requests, in addition to, or instead of,
just caching the N most recent articles?
--- Synchronet 3.22a-Linux NewsLink 1.2