I'm pleased to announce that the archive I've been working on for several years is finally as complete as it can be, and is ready for public consumption.
The archive contains articles beginning in early 1981 and span to current day.
It is far from complete, as I'm sure the source material was imperfect.
There are over 1.1 billion articles consuming over 5 terabytes.
Article numbering in this archive is in chronological order. Posting to the archive is not permitted.
There is a period of time, roughly 1995-2000, where valid timestamps were not present. The dates are correct, but all timestamps are midnight UTC:
Date: Thu, 30 Mar 1995 00:00:00 +0000
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
The news.blueworldhosting.com server that has been in existence will be retired in favor of the filtered archive, once it is available. A separate announcement for this will be made with a planned cutover and retirement date.
The archive is available at archive.usenet.blueworldhosting.com via ports 119 (clear text) and 563 (TLS).
I will not make a 'web' version available. There are others who have taken on that endeavor, and they're free to use this archive.
I encourage others to replicate the archive, but do not recommend attempting to pull/suck articles. This is a fruitless endeavor and will likely take a better part of a year to complete.
If you have the storage space and knowledge of INN/Diablo I will gladly push the archive to you, or make the CNFS buffers, history, and overview databases available for download.
It took over two weeks to feed the articles to this server from the 'repository' server I used to collect them using innxmit with near 0ms latency
and a large SSD storage array.
Cheers,
Jesse Rehmer
You are a legend, thanks for doing this. I need to order some DRAM
(ouch) so I can support this and will try to make arrangements to grab a
feed from you so I can see if my FTS can scale that large.
Regards,
Kevin
I will not make a 'web' version available. There are others who have taken on that endeavor, and they're free to use this archive.
I'm pleased to announce that the archive I've been working on for several >years is finally as complete as it can be, and is ready for public >consumption.
The archive contains articles beginning in early 1981 and span to current day. >It is far from complete, as I'm sure the source material was imperfect.
There are over 1.1 billion articles consuming over 5 terabytes.
Article numbering in this archive is in chronological order. Posting to the >archive is not permitted.
There is a period of time, roughly 1995-2000, where valid timestamps were not >present. The dates are correct, but all timestamps are midnight UTC:
Date: Thu, 30 Mar 1995 00:00:00 +0000
The archive contains a lot of spam, but we're working on a filtered version of >the archive.
The news.blueworldhosting.com server that has been in existence will be >retired in favor of the filtered archive, once it is available. A separate >announcement for this will be made with a planned cutover and retirement date.
The archive is available at archive.usenet.blueworldhosting.com via ports 119 >(clear text) and 563 (TLS).
I will not make a 'web' version available. There are others who have taken on >that endeavor, and they're free to use this archive.
I encourage others to replicate the archive, but do not recommend attempting >to pull/suck articles. This is a fruitless endeavor and will likely take a >better part of a year to complete.
If you have the storage space and knowledge of INN/Diablo I will gladly push >the archive to you, or make the CNFS buffers, history, and overview databases >available for download.
It took over two weeks to feed the articles to this server from the >'repository' server I used to collect them using innxmit with near 0ms latency >and a large SSD storage array.
Cheers,
Jesse Rehmer
Article numbering in this archive is in chronological order.
I'm pleased to announce that the archive I've been working on for several years is finally as complete as it can be, and is ready for public consumption.
I'm pleased to announce that the archive I've been working on for several years is finally as complete as it can be, and is ready for public consumption.
The archive contains articles beginning in early 1981 and span to current day.
It is far from complete, as I'm sure the source material was imperfect.
There are over 1.1 billion articles consuming over 5 terabytes.
Article numbering in this archive is in chronological order.
I'm pleased to announce that the archive I've been working on for several years is finally as complete as it can be, and is ready for public consumption.
The archive contains articles beginning in early 1981 and span to current day.
It is far from complete, as I'm sure the source material was imperfect.
There are over 1.1 billion articles consuming over 5 terabytes.
Article numbering in this archive is in chronological order. Posting to the archive is not permitted.
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
I'm pleased to announce that the archive I've been working on for several years is finally as complete as it can be, and is ready for public consumption.
Jesse Rehmer wrote:
I'm pleased to announce that the archive I've been working on for several
years is finally as complete as it can be, and is ready for public
consumption.
What's the process for requesting erasure of personal data from the
archive, e.g. removal or anonymization of names and addresses?
Individuals might not have access to old e-mail accounts going back that
far anymore.
https://gdpr.eu/right-to-be-forgotten/
Sure, but Jesse doesn't live or reside in Europe. So he's not required
to comply with the GDPR at all.
The fun part about the GDPR is that it doesn't matter where the operator lives. The service is accessible in the EU.
Il 18/08/2026 20:44, Nomen Nescio ha scritto:
The fun part about the GDPR is that it doesn't matter where the operator
lives. The service is accessible in the EU.
This is also true, but I'd like you to tell me who will enforce this >extrajudicial rule. As you well know, legislators only act for the big >players (who at least have local offices they can rely on to enforce
their decisions, see BigG, Microsoft, etc.).
In the various legal cases I've witnessed over the years here in the EU, >involving non-EU issues, everyone has achieved the classic Pyrrhic
victory: they won the case in court (in 99% of cases, the defendant
didn't even have a lawyer present in the courtroom, so they won at the >preliminary stage), but the judge is careful not to force the case, much >less issue international letters rogatory, so there's no way to enforce
the ruling.
This is especially true if the case is civil and not criminal (and the
GDPR is entirely civil, 0% criminal). I've only seen anything moving in >criminal cases.https://duckduckgo.com/?q=pyrrhic+victory&ia=web
I myself was sued some time ago (a user of my server wasn't exactly so >upright). The plaintiff won the case, I was acquitted because I was >considered an ISP and therefore protected, and the user was found
guilty. Too bad he was a TOR user, they never managed to track him down,
let alone make him serve his sentence.
As I said, a Pyrrhic victory.
A Pyrrhic victory is a win that costs the victor so much that it's essentially >a defeat. The term comes from King Pyrrhus of Epirus, after costly Roman >battles in 280 and 279 BCE.[end quoted "search assist"]
Encyclopedia Britannica Wikipedia
Definition of Pyrrhic Victory
A Pyrrhic victory is a win that comes at such a high cost to the victor that >it is essentially equivalent to defeat. The term originates from King Pyrrhus >of Epirus, who fought against the Romans in the Battles of Heraclea and Asculum
in 280 and 279 BCE. Despite winning these battles, Pyrrhus suffered devastating
losses that weakened his forces significantly.
Historical Context
Key Battles
Battle Year Victor Cost to Victor Outcome for Victor Battle of 280 Pyrrhus Heavy casualties, Strategic advantage
Heraclea BCE of Epiruse including elite fighters but weakened army
Battle of 279 Pyrrhus Loss of many commanders Ultimately led to Asculum BCE of Epirus and troops his defeat in 275 BCE
After the Battle of Asculum, Pyrrhus reportedly stated, "One more such victory >and we are lost," highlighting the unsustainable nature of his triumphs.
Broader Implications
The concept of a Pyrrhic victory extends beyond military contexts. It can >apply to various fields, including:
Sports: A team may win a game but lose a key player to injury.
Politics: A political party may win an election but face backlash that
undermines its future stability.
Business: A company might achieve a market win but incur losses that
jeopardize its financial health.
In all these cases, the costs associated with the victory overshadow the >benefits, leading to a situation where the victor is left in a precarious >position.
I'm pleased to announce that the archive I've been working on for several years is finally as complete as it can be, and is ready for public consumption.
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
The news.blueworldhosting.com server that has been in existence will be retired in favor of the filtered archive, once it is available. A separate announcement for this will be made with a planned cutover and retirement date.
Jesse Rehmer <jesse.rehmer@blueworldhosting.com> posted:
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
The news.blueworldhosting.com server that has been in existence will be retired in favor of the filtered archive, once it is available. A separate announcement for this will be made with a planned cutover and retirement date.
First, congratulations on a great achievement!
Do you have any timescale for when the filtered archive will be available? The index db I'm using for Newsgrouper has quite a bit of rubbish in it.
I would like to rebuild this based on your archive, but it probably makes sense to wait for the filtered version?
I created my index db by using XOVER on each group, doing some processing
on that data (e.g. I only record the last msgid in References) and loading that into the database. I no longer depend on nntp server numbering, I
index posts by group and timestamp. Is it reasonable to run that import process again over nntp, or would you prefer I get the data by some other means?
Colin Macleod <user7@newsgrouper.org.invalid> posted:
Jesse Rehmer <jesse.rehmer@blueworldhosting.com> posted:
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
The news.blueworldhosting.com server that has been in existence will be
retired in favor of the filtered archive, once it is available. A separate >>> announcement for this will be made with a planned cutover and retirement date.
First, congratulations on a great achievement!
Do you have any timescale for when the filtered archive will be available? >> The index db I'm using for Newsgrouper has quite a bit of rubbish in it.
I would like to rebuild this based on your archive, but it probably makes
sense to wait for the filtered version?
I created my index db by using XOVER on each group, doing some processing
on that data (e.g. I only record the last msgid in References) and loading >> that into the database. I no longer depend on nntp server numbering, I
index posts by group and timestamp. Is it reasonable to run that import
process again over nntp, or would you prefer I get the data by some other
means?
Now I see that you offered to make the INN data available for download.
I'm not familiar with INN's internal data structures, but I would imagine that the overview database would have the info I need. If it's in sqlite form that would also be easy to work with.
Hi Jesse,
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
Incidentally, if you intend to run a new instance of INN for the
filtered version of the archive, you could go with the new hissqlite
backend (history in a SQLite database).
There's also a hissqlite-convert tool to convert an existing hisv6
database to hissqlite but I don't know how long the downtime will be for
1.1 billion articles.
On Aug 18, 2026 at 10:13:42rC>AM CDT, "Julien |eLIE" <iulius@nom-de-mon-site.com.invalid> wrote:
Hi Jesse,
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
Incidentally, if you intend to run a new instance of INN for the
filtered version of the archive, you could go with the new hissqlite
backend (history in a SQLite database).
There's also a hissqlite-convert tool to convert an existing hisv6
database to hissqlite but I don't know how long the downtime will be for
1.1 billion articles.
I do plan to give it a try. If performance is poor I'll start with hisv6 and convert to hissqlite after ingestion.
It's fine to use XOVER and reprocess the new spool and I'm also more than happy to provide a copy of the SQLite DB if that would make it better for you.
I don't have a solid timeframe for the filtered archive, probably another month or two, unless I get in gear.
I'm pleased to announce that the archive I've been working on for several years is finally as complete as it can be, and is ready for public consumption.
Ivo Gandolfo wrote:
Sure, but Jesse doesn't live or reside in Europe. So he's not required
to comply with the GDPR at all.
The fun part about the GDPR is that it doesn't matter where the operator lives. The service is accessible in the EU.
On 8/19/26 3:36 PM, Jesse Rehmer wrote:
On Aug 18, 2026 at 10:13:42rC>AM CDT, "Julien |eLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:
Hi Jesse,
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
Incidentally, if you intend to run a new instance of INN for the
filtered version of the archive, you could go with the new hissqlite
backend (history in a SQLite database).
There's also a hissqlite-convert tool to convert an existing hisv6
database to hissqlite but I don't know how long the downtime will be for >>> 1.1 billion articles.
I do plan to give it a try. If performance is poor I'll start with hisv6 and >> convert to hissqlite after ingestion.
The main benefit of hissqlite is msgid lookup performance and lack of periodic maintenance (hisv6 has the expensive rebuild the entire DB
nightly). The hisbench util would be informative before making a
decision one way or the other to see if it scales to the necessary level
in all operations.
On fast storage you might expect 100k writes/s with a lot of write amplification. The converter might be a little faster due to aggressive batching that lowers write amplification. At steady text feed rates the write amplification is trivial.
I'm pleased to announce that the archive I've been working on for several years is finally as complete as it can be, and is ready for public consumption.
The archive contains articles beginning in early 1981 and span to current day.
It is far from complete, as I'm sure the source material was imperfect.
There are over 1.1 billion articles consuming over 5 terabytes.
Article numbering in this archive is in chronological order. Posting to the archive is not permitted.
There is a period of time, roughly 1995-2000, where valid timestamps were not present. The dates are correct, but all timestamps are midnight UTC:
Date: Thu, 30 Mar 1995 00:00:00 +0000
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
The news.blueworldhosting.com server that has been in existence will be retired in favor of the filtered archive, once it is available. A separate announcement for this will be made with a planned cutover and retirement date.
The archive is available at archive.usenet.blueworldhosting.com via ports 119 (clear text) and 563 (TLS).
I will not make a 'web' version available. There are others who have taken on that endeavor, and they're free to use this archive.
I encourage others to replicate the archive, but do not recommend attempting to pull/suck articles. This is a fruitless endeavor and will likely take a better part of a year to complete.
If you have the storage space and knowledge of INN/Diablo I will gladly push the archive to you, or make the CNFS buffers, history, and overview databases available for download.
It took over two weeks to feed the articles to this server from the 'repository' server I used to collect them using innxmit with near 0ms latency
and a large SSD storage array.
Cheers,
Jesse Rehmer
On Aug 19, 2026 at 6:38:49rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com>
wrote:
On 8/19/26 3:36 PM, Jesse Rehmer wrote:
On Aug 18, 2026 at 10:13:42rC>AM CDT, "Julien |eLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:
Hi Jesse,
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
Incidentally, if you intend to run a new instance of INN for the
filtered version of the archive, you could go with the new hissqlite
backend (history in a SQLite database).
There's also a hissqlite-convert tool to convert an existing hisv6
database to hissqlite but I don't know how long the downtime will be for >>>> 1.1 billion articles.
I do plan to give it a try. If performance is poor I'll start with hisv6 and
convert to hissqlite after ingestion.
The main benefit of hissqlite is msgid lookup performance and lack of
periodic maintenance (hisv6 has the expensive rebuild the entire DB
nightly). The hisbench util would be informative before making a
decision one way or the other to see if it scales to the necessary level
in all operations.
On fast storage you might expect 100k writes/s with a lot of write
amplification. The converter might be a little faster due to aggressive
batching that lowers write amplification. At steady text feed rates the
write amplification is trivial.
On my system hisv6 is the winner:
hisv6 entries=100000000 bytes=13833579210 path=history-bench-IoaaYO/hisv6
write: 995.137s 100489 entries/s
sync: 0.000s
close: 1.274s
open-ro: 0.000s
check: 82.695s 1209256 entries/s
walk: 97.806s 1022434 entries/s
lookup: 74.182s 13480 entries/s
missing: 0.889s 1124256 entries/s
hissqlite entries=100000000 bytes=5709291520 path=history-bench-IoaaYO/hissqlite
write: 16452.204s 6078 entries/s
sync: 0.000s
close: 0.012s
open-ro: 0.001s
check: 12333.236s 8108 entries/s
walk: 118.499s 843888 entries/s
lookup: 125.102s 7993 entries/s
missing: 122.916s 8136 entries/s
Nomen Nescio <nobody@dizum.com> writes:
Ivo Gandolfo wrote:
Sure, but Jesse doesn't live or reside in Europe. So he's not required
to comply with the GDPR at all.
The fun part about the GDPR is that it doesn't matter where the operator
lives. The service is accessible in the EU.
The only realistic way for EU bureaucrats to do anything about it is if service in question is provided by commercial organization that is
making money off of EU customers. And even then it can be endless legal battle in which someone will have to prove that yes, it is in fact their posts that they want to be removed.
Also, can you please let me know which posts exactly should be removed,
so I can reply to them quoting their whole content?
On 8/20/26 4:47 AM, Jesse Rehmer wrote:
On Aug 19, 2026 at 6:38:49rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com>
wrote:
On 8/19/26 3:36 PM, Jesse Rehmer wrote:
On Aug 18, 2026 at 10:13:42rC>AM CDT, "Julien |eLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:
Hi Jesse,
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
Incidentally, if you intend to run a new instance of INN for the
filtered version of the archive, you could go with the new hissqlite >>>>> backend (history in a SQLite database).
There's also a hissqlite-convert tool to convert an existing hisv6
database to hissqlite but I don't know how long the downtime will be for >>>>> 1.1 billion articles.
I do plan to give it a try. If performance is poor I'll start with hisv6 and
convert to hissqlite after ingestion.
The main benefit of hissqlite is msgid lookup performance and lack of
periodic maintenance (hisv6 has the expensive rebuild the entire DB
nightly). The hisbench util would be informative before making a
decision one way or the other to see if it scales to the necessary level >>> in all operations.
On fast storage you might expect 100k writes/s with a lot of write
amplification. The converter might be a little faster due to aggressive >>> batching that lowers write amplification. At steady text feed rates the >>> write amplification is trivial.
On my system hisv6 is the winner:
Agreed.
hisv6 entries=100000000 bytes=13833579210 path=history-bench-IoaaYO/hisv6
write: 995.137s 100489 entries/s
sync: 0.000s
close: 1.274s
open-ro: 0.000s
check: 82.695s 1209256 entries/s
walk: 97.806s 1022434 entries/s
lookup: 74.182s 13480 entries/s
missing: 0.889s 1124256 entries/s
hissqlite entries=100000000 bytes=5709291520
path=history-bench-IoaaYO/hissqlite
write: 16452.204s 6078 entries/s
sync: 0.000s
close: 0.012s
open-ro: 0.001s
check: 12333.236s 8108 entries/s
walk: 118.499s 843888 entries/s
lookup: 125.102s 7993 entries/s
missing: 122.916s 8136 entries/s
One thing to check, if it is on ZFS you want zfs "recordsize=4K" to
match the SQLite pagesize, otherwise you are getting a massive ZFS write amplification on top of SQLite page rebalancing. I use the new
historypath inn.conf config to put it on its own dataset.
On Aug 20, 2026 at 6:12:35rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com>
wrote:
On 8/20/26 4:47 AM, Jesse Rehmer wrote:
On Aug 19, 2026 at 6:38:49rC>PM CDT, "Kevin Bowling" <kevin.bowling@kev009.com>
wrote:
On 8/19/26 3:36 PM, Jesse Rehmer wrote:
On Aug 18, 2026 at 10:13:42rC>AM CDT, "Julien |eLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:
Hi Jesse,
The archive contains a lot of spam, but we're working on a filtered version of
the archive.
Incidentally, if you intend to run a new instance of INN for the
filtered version of the archive, you could go with the new hissqlite >>>>>> backend (history in a SQLite database).
There's also a hissqlite-convert tool to convert an existing hisv6 >>>>>> database to hissqlite but I don't know how long the downtime will be for >>>>>> 1.1 billion articles.
I do plan to give it a try. If performance is poor I'll start with hisv6 and
convert to hissqlite after ingestion.
The main benefit of hissqlite is msgid lookup performance and lack of
periodic maintenance (hisv6 has the expensive rebuild the entire DB
nightly). The hisbench util would be informative before making a
decision one way or the other to see if it scales to the necessary level >>>> in all operations.
On fast storage you might expect 100k writes/s with a lot of write
amplification. The converter might be a little faster due to aggressive >>>> batching that lowers write amplification. At steady text feed rates the >>>> write amplification is trivial.
On my system hisv6 is the winner:
Agreed.
hisv6 entries=100000000 bytes=13833579210 path=history-bench-IoaaYO/hisv6
write: 995.137s 100489 entries/s
sync: 0.000s
close: 1.274s
open-ro: 0.000s
check: 82.695s 1209256 entries/s
walk: 97.806s 1022434 entries/s
lookup: 74.182s 13480 entries/s
missing: 0.889s 1124256 entries/s
hissqlite entries=100000000 bytes=5709291520
path=history-bench-IoaaYO/hissqlite
write: 16452.204s 6078 entries/s
sync: 0.000s
close: 0.012s
open-ro: 0.001s
check: 12333.236s 8108 entries/s
walk: 118.499s 843888 entries/s
lookup: 125.102s 7993 entries/s
missing: 122.916s 8136 entries/s
One thing to check, if it is on ZFS you want zfs "recordsize=4K" to
match the SQLite pagesize, otherwise you are getting a massive ZFS write
amplification on top of SQLite page rebalancing. I use the new
historypath inn.conf config to put it on its own dataset.
Thanks for reminding me about that, the performance is improved with a 4k record size:
hissqlite entries=100000000 bytes=5709291520 path=/rpool/history/test/hissqlite
write: 8754.066s 11423 entries/s
sync: 0.000s
close: 0.032s
open-ro: 0.000s
check: 1875.500s 53319 entries/s
walk: 42.869s 2332697 entries/s
lookup: 17.998s 55560 entries/s
missing: 18.025s 55479 entries/s
In news.software.nntp Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
I'm pleased to announce that the archive I've been working on for several
years is finally as complete as it can be, and is ready for public
consumption.
That's great :) Thanks for your work!
If you want, you can check my archive at news-archive.chmurka.net (port
121 -- that's important as the IP is shared with news.chmurka.net that
uses port 119). If there are any posts you don't have, but want, then feel free to download (suck, or let me know and we'll transfer them more efficiently).
TLS is on 565.
I'll check later if you have posts I don't (I carry only pl.* and alt.pl.* plus a few individual Polish groups) and if yes, download them.
This irks me as it is important (to me) to keep article numbers >chronologically, and there is no good way to merge.
Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote or quoted:
This irks me as it is important (to me) to keep article numbers
chronologically, and there is no good way to merge.
It is possible that - in the future - there still will be many
offers of historic articles, and one just cannot always renumber
everything or reject adding the offer, so the best solution is
to abandon the idea of keeping the article numbers chronological.
This irks me as it is important (to me) to keep article numbers chronologically, and there is no good way to merge.
| Sysop: | Amessyroom |
|---|---|
| Location: | Fayetteville, NC |
| Users: | 76 |
| Nodes: | 6 (0 / 6) |
| Uptime: | 60:36:32 |
| Calls: | 1,265 |
| Calls today: | 1 |
| Files: | 1,362 |
| D/L today: |
4 files (9,632K bytes) |
| Messages: | 286,219 |