I made a web thing: http://csiph.com/groups
It's like google groups but it sucks less.
I made a web thing: http://csiph.com/groups
It's like google groups but it sucks less.
On 5/1/2026 12:31 PM, Kevin Bowling wrote:
I made a web thing: http://csiph.com/groups
It's like google groups but it sucks less.
Very nice! I was going to do something like this at some point, but this
is basically what I intended to do (and with posting, on top of that).
Is this directly integrated specifically with the CSIPH news spool, or
using NNTP to talk to the news server? I was thinking recently it would
be helpful to have something like this for private hierarchies or just testing in general.
On 5/1/26 5:40 PM, InterLinked wrote:
Is this directly integrated specifically with the CSIPH news spool, or
using NNTP to talk to the news server? I was thinking recently it
would be helpful to have something like this for private hierarchies
or just testing in general.
I wrote a from scratch NNTP library in async Rust (Tokio), so the webui
uses that with a connection pool to talk to csiph's inn nnrp.-a The webui manages some memory caches of the newsgroup list, per group overviews,
jwz article threading.-a The main bottleneck right now is inn OVER performance, I need to look into that a bit more but sent a couple
patches for ovsqlite that might help.-a Getting threading to be useful requires a big OVER, around 10k articles, which I called eras.-a When the user pages near the border of an era, I create a temporary union I
called a tandem with the next era, so you can generally have threading
work 10k in either direction.
The webui also manages search indexing (Tantivy) over NNTP and there is
a small user binary from the same codebase that is used for inn's nnrpd
so one account gives access to web and nnrp.
On 5/1/2026 9:22 PM, Kevin Bowling wrote:
On 5/1/26 5:40 PM, InterLinked wrote:
Is this directly integrated specifically with the CSIPH news spool,
or using NNTP to talk to the news server? I was thinking recently it
would be helpful to have something like this for private hierarchies
or just testing in general.
I wrote a from scratch NNTP library in async Rust (Tokio), so the
webui uses that with a connection pool to talk to csiph's inn nnrp.
The webui manages some memory caches of the newsgroup list, per group
overviews, jwz article threading.-a The main bottleneck right now is
inn OVER performance, I need to look into that a bit more but sent a
couple patches for ovsqlite that might help.-a Getting threading to be
useful requires a big OVER, around 10k articles, which I called eras.
When the user pages near the border of an era, I create a temporary
union I called a tandem with the next era, so you can generally have
threading work 10k in either direction.
The webui also manages search indexing (Tantivy) over NNTP and there
is a small user binary from the same codebase that is used for inn's
nnrpd so one account gives access to web and nnrp.
Very impressive. Not sure how feasible this would be, but is this
something that you would consider open sourcing that others can run with other news servers?
On 5/1/26 7:00 PM, InterLinked wrote:
Very impressive. Not sure how feasible this would be, but is this something that you would consider open sourcing that others can run with other news servers?
Eventually, the NNTP lib first. It all will need some cleanup, somewhat specific to my setup and FreeBSD right now.
Kevin Bowling <kevin.bowling@kev009.com> posted:
On 5/1/26 7:00 PM, InterLinked wrote:
Very impressive. Not sure how feasible this would be, but is this
something that you would consider open sourcing that others can run with >>> other news servers?
Eventually, the NNTP lib first. It all will need some cleanup, somewhat
specific to my setup and FreeBSD right now.
InterLinked, you might be interested in the code for my usenet/web interface, which is open-sourced at https://chiselapp.com/user/cmacleod/repository/newsgrouper/home
Like Kevin, I had to do some work to clean it up for publication, however there are still a few things hard-coded which really should be configurable.
Some differences from Kevin's code:
- My Newsgrouper code is in Tcl - some may consider this a bug, to me
it's a feature. :-)
- Newsgrouper can pull overview info and posts from multiple nntp servers,
caching them locally using a mixture of Sqlite and Redis and presenting
them as a unified whole.
- Newsgrouper cannot support full-text search of posts, which CSIPH does.
- The user interface is of course significantly different. Newsgrouper
uses a little Javascript to make it possible to drive most functionality
from the keyboard (but is still usable without Javascript). It has
various refinements like support for Face and X-Face images, which I
imagine CSIPH may add in time.
- CSIPH appears not to handle base64 encoding at present - see
http://csiph.com/group/cn.fan/22724 - Newsgrouper does handle this,
the same post can be seen at:
https://newsgrouper.org/%3C867brmgy5g.fsf@gmail.com%3E .
- Kevin says "The main bottleneck right now is inn OVER performance".
I avoid that by keeping overview data in a Sqlite table, pulling only
new OVER data from the nntp server(s) as required, then I can do fast
queries against that db table to display thread and post lists.
- Newsgrouper can pull overview info and posts from multiple nntp servers,
caching them locally using a mixture of Sqlite and Redis and presenting
them as a unified whole.
Since it is all NNTP based I could point it to other servers, but it
would create a fairly unusual usage pattern if not done with permission
and forethought.
- Newsgrouper cannot support full-text search of posts, which CSIPH does.
This turned out to be a fair amount of work, I had to do custom
tokenizer, clustering, some hacky normalization before BM25 so the same thread with a bunch of quoting doesn't overscore, signatures don't blow
out keywords, etc. My goal is to replace/displace Google Groups so it
is a key feature. I probably have to reindex at least once more as I
fixed some date parsing stuff but the current index is pretty good and
got past alt.* and into comp.* now, a couple more days and it'll be
fully populated.
- The user interface is of course significantly different. Newsgrouper
uses a little Javascript to make it possible to drive most functionality
from the keyboard (but is still usable without Javascript). It has
various refinements like support for Face and X-Face images, which I
imagine CSIPH may add in time.
There will never be any JS on CSIPH because it is meant to be used from retro systems. I've tested back to Mosaic. You'll need cookies for
auth but that is still Netscape 1/IE2 era. It does come with tradeoffs
but I like the aesthetic.
- CSIPH appears not to handle base64 encoding at present - see
http://csiph.com/group/cn.fan/22724 - Newsgrouper does handle this,
the same post can be seen at:
https://newsgrouper.org/%3C867brmgy5g.fsf@gmail.com%3E .
Thanks, I'll fix this. Text encodings and time formats turned out to be quite a tarpit.
- Kevin says "The main bottleneck right now is inn OVER performance".
I avoid that by keeping overview data in a Sqlite table, pulling only
new OVER data from the nntp server(s) as required, then I can do fast
queries against that db table to display thread and post lists.
I think I've got this nailed down https://github.com/InterNetNews/inn/pull/338
I do aggressively cache things, but the particular use of overview for threading is extremely heavy. My overview DB is 26GB highly compressed
and I don't want to over-optimize the webui over NNTP users.
Do you have any protection against bots?
Kevin Bowling <kevin.bowling@kev009.com> posted:
On 5/1/26 7:00 PM, InterLinked wrote:
Very impressive. Not sure how feasible this would be, but is this
something that you would consider open sourcing that others can run with >>> other news servers?
Eventually, the NNTP lib first. It all will need some cleanup, somewhat
specific to my setup and FreeBSD right now.
InterLinked, you might be interested in the code for my usenet/web interface, which is open-sourced at https://chiselapp.com/user/cmacleod/repository/newsgrouper/home
Like Kevin, I had to do some work to clean it up for publication, however there are still a few things hard-coded which really should be configurable.
Some differences from Kevin's code:
- My Newsgrouper code is in Tcl - some may consider this a bug, to me
it's a feature. :-)
Thanks Colin! This is a nice resource as well.
I'll admit, I'm more partial to the way Kevin's site works. What I was intending to put together myself was basically a stateless* NNTP client
that showed all the groups and group info and then showed articles when
you clicked on them (which is what CSIPH does, and more, and more efficiently at that). And FWIW, I like the aesthetic too.
My goal is to replace/displace Google Groups
I probably have to reindex at least once more as I
fixed some date parsing stuff
I see that the URLs contain article numbers, which may change in case of
a renumbering (it may happen one day, who knows). It could be
interesting to have a permanent link feature with a URL containing the Message-ID.
Hi Kevin,
My goal is to replace/displace Google Groups
I really hope you'll manage to provide such a useful alternative!
I see that the URLs contain article numbers, which may change in case of
a renumbering (it may happen one day, who knows).-a It could be
interesting to have a permanent link feature with a URL containing the Message-ID.
I probably have to reindex at least once more as I fixed some date
parsing stuff
Did you try to use a library like https://docs.rs/dateparser/latest/ dateparser/ which seems to handle lots of date formats?
Regarding searches, how is the language taken into account?
Searching "Usenet" with French languages give me articles written in English.
-a http://csiph.com/search?q=usenet&lang=fr
How often do you run innreport?-a It seems to be started automatically at 04:15 but not every day?
-a http://csiph.com/innreport/
Hi Kevin,
There's massive contention in nnrp right now, it will let up a bit
once search indexing is complete and https://github.com/InterNetNews/
inn/ pull/338 is the ultimate root cause fix I will patch it in when I
have a moment.
Thanks a lot for the improvements you propose to ovsqlite; I'll have a
look at it soon.
A great webnews interface, thanks for sharing it!
If I may suggest: it would look nicer if the Date colum was somehow normalized instead of taking the raw contents of the header field.
The newsgroups descriptions are not up-to-date.-a For instance fr.comp.sys.raspberry-pi currently has no description, and some other descriptions are old ones.
One thing that would be interesting is adopting IMAP's THREAD and SORT
extensions for NNTP https://datatracker.ietf.org/doc/html/rfc5256
That would be interesting indeed, though it adds a bit of load at server side.-a It is also quite a work to add support for that in all overview methods of INN (unless it would be ovsqlite-specific for instance), as
well as in other still used news servers.
On 5/2/26 2:15 AM, Colin Macleod wrote:
- CSIPH appears not to handle base64 encoding at present - see
-a-a http://csiph.com/group/cn.fan/22724 - Newsgrouper does handle this,
-a-a the same post can be seen at:
-a-a https://newsgrouper.org/%3C867brmgy5g.fsf@gmail.com%3E .
Thanks, I'll fix this.-a Text encodings and time formats turned out to be quite a tarpit.
On 5/2/26 5:28 AM, Kevin Bowling wrote:
On 5/2/26 2:15 AM, Colin Macleod wrote:
- CSIPH appears not to handle base64 encoding at present - see
-a-a http://csiph.com/group/cn.fan/22724 - Newsgrouper does handle this, >> -a-a the same post can be seen at:
-a-a https://newsgrouper.org/%3C867brmgy5g.fsf@gmail.com%3E .
Thanks, I'll fix this.-a Text encodings and time formats turned out to be quite a tarpit.
I think this is fixed, there is multipart and base64 handling in place.
I made a test post just now, which worked ok.
I would suggest having a warning that the email you set in your profile
when creating an account *will be exposed* in the From line when you post. When I created my account I was not sure if this required a real email address (for newsgrouper I require that a user registers with a real email which I do not expose publicly).
On 5/5/26 06:19, Colin Macleod wrote:
I made a test post just now, which worked ok.
I would suggest having a warning that the email you set in your profile when creating an account *will be exposed* in the From line when you post. When I created my account I was not sure if this required a real email address (for newsgrouper I require that a user registers with a real email which I do not expose publicly).
I intend to do the same thing, create a second field called the display email. The system will require (and send a validation link) to the registration email to cut down on false signups. Then posts will only
show the display email which can be invalid.
I see that the URLs contain article numbers, which may change in case
of a renumbering (it may happen one day, who knows).-a It could be
interesting to have a permanent link feature with a URL containing the
Message-ID.
Added
I checked a few but none of them handle the real world corpus.-a In my
NNTP library I currently have the following, I'm sure it is missing a
lot but it seems easy to extend/maintain
How often do you run innreport?-a It seems to be started automatically
at 04:15 but not every day?
-a-a http://csiph.com/innreport/
The current problem is expireover takes around the gap you see to
complete.
If I may suggest: it would look nicer if the Date colum was somehow
normalized instead of taking the raw contents of the header field.
Done.-a Raw date format is still available from Show all headers in an article.
Looks like I need to update newsgroups file.-a I saw some references that actsync can do this, but no examples, is it possible to have one
actsync.cfg cover this?-a And I guess it needs to be more careful than active, should merge not remove entries?
There's some background here https://www.jwz.org/doc/threading.html
I made a web thing: http://csiph.com/groups
It's like google groups but it sucks less.
The internals would need some rework to actually handle multi-server,
article numbers are still too convenient for i.e. efficient cache
structures and efficient read tracking, but that could still work with a tuple if needed.
different groups. For a cross-posted article I provide links to see
it in the context of each other group where it exists, but I can't use >timestamp-based links for this because they might vary, so here I use
Colin Macleod <user7@newsgrouper.org.invalid> wrote or quoted:
different groups. For a cross-posted article I provide links to see
it in the context of each other group where it exists, but I can't use >timestamp-based links for this because they might vary, so here I use
The "normalized" approach in a relational database would be to have
one group for posts,
But since you mention "links", and relational database terminology
does not have "links", it's possible you are using some other type
of database - then the above approach might not be appropriate.
Now, we just need to know whether a post that just came
from a server already is in the database. One could use
the message ID for that, but message IDs do not have to be
unique over a long timespan of decades I think, so one might
combine the message ID with the date or something.
I made a web thing: http://csiph.com/groups
It's like google groups but it sucks less.
| Sysop: | Amessyroom |
|---|---|
| Location: | Fayetteville, NC |
| Users: | 65 |
| Nodes: | 6 (0 / 6) |
| Uptime: | 07:37:19 |
| Calls: | 862 |
| Files: | 1,311 |
| Messages: | 264,829 |