• Request for spam filtering assistance

    From Jesse Rehmer@jesse.rehmer@blueworldhosting.com to news.software.nntp on Tue Oct 6 01:06:47 2026
    From Newsgroup: news.software.nntp

    Fellow News Admins,

    As you may know, I put together a large archive, and want feed it to another server and filter with Cleanfeed.

    I'm seeking cleanfeed.local entries, and bad_* files from news admins to
    filter as much as possible. I have 45 years of articles to filter, so nothing is irrelevant.

    You can reach me via e-mail, no need to publicly share.

    Cheers,

    Jesse
    --- Synchronet 3.22a-Linux NewsLink 1.2
  • From Jason Evans@news@theuse.net to news.software.nntp on Tue Oct 6 16:47:38 2026
    From Newsgroup: news.software.nntp

    Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:

    I'm seeking cleanfeed.local entries, and bad_* files from news admins to filter as much as possible. I have 45 years of articles to filter, so nothing is irrelevant.

    Why not keep the archives whole, "warts and all"? Yes, cutting out the spam will
    greatly reduce the amount of data that you need to store but an unedited archive
    would be valuable also.
    --- Synchronet 3.22a-Linux NewsLink 1.2
  • From Adam H. Kerman@ahk@chinet.com to news.software.nntp on Tue Oct 6 17:26:58 2026
    From Newsgroup: news.software.nntp

    Jason Evans <news@theuse.net> wrote:
    Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:

    I'm seeking cleanfeed.local entries, and bad_* files from news admins to >>filter as much as possible. I have 45 years of articles to filter, so nothing >>is irrelevant.

    Why not keep the archives whole, "warts and all"? Yes, cutting out the
    spam will greatly reduce the amount of data that you need to store but
    an unedited archive would be valuable also.

    You don't explain what the value is. A Hipclone DOS attack would have
    bee in the millions of articles or contol messages in the hundreds of thousands. The Chinese spam attack via Google Groups was hundreds of
    articles a day per group.

    If I'm looking for discussion, that's one hell of a lot of bad hits. I
    don't want to find spam hits in the index, making the archive of less
    value to me.

    No one would have seen raw spam in its most massive quantity anythere.
    There's always some countermeasure in place else the system gets
    overwhelmed to the oiint of shutdown.
    --- Synchronet 3.22a-Linux NewsLink 1.2
  • From Jesse Rehmer@jesse.rehmer@blueworldhosting.com to news.software.nntp on Tue Oct 6 23:01:49 2026
    From Newsgroup: news.software.nntp

    On Oct 6, 2026 at 11:47:38rC>AM CDT, "Jason Evans" <news@theuse.net> wrote:

    Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:

    I'm seeking cleanfeed.local entries, and bad_* files from news admins to
    filter as much as possible. I have 45 years of articles to filter, so nothing
    is irrelevant.

    Why not keep the archives whole, "warts and all"? Yes, cutting out the spam will
    greatly reduce the amount of data that you need to store but an unedited archive
    would be valuable also.

    The full archive will remain at archive.usenet.blueworldhosting.com. The filtered archive will replace the news.blueworldhosting.com server.
    --- Synchronet 3.22a-Linux NewsLink 1.2
  • From aw@aw@somewhere.invalid (Adam W.) to news.software.nntp on Wed Oct 7 11:34:23 2026
    From Newsgroup: news.software.nntp

    Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:

    You can reach me via e-mail, no need to publicly share.

    I can share publicly, as this is mostly historic now.

    Here's the filtering code I used when preparing articles for news-archive.chmurka.net. It's based on a Python filter on
    news.chmurka.net. Spam filtering is in checkArticle(). In short:
    a blacklist of Froms, all Thai subjects and froms, some keywords
    in Subject related to Kindle, and one blacklisted URL in body.

    http://www.chmurka.net/r/usenet/classify.cpp
    --- Synchronet 3.22a-Linux NewsLink 1.2
  • From Billy G.@42-contact-42@pugleaf.net to news.software.nntp on Mon Nov 9 19:35:27 2026
    From Newsgroup: news.software.nntp

    On 10/6/26 01:06, Jesse Rehmer wrote:
    I'm seeking cleanfeed.local entries, and bad_* files from news admins to filter as much as possible. I have 45 years of articles to filter, so
    nothing
    is irrelevant.

    We took a different route.
    Instead of collecting filter lists, we labelled the archive itself.
    All numbers are from archive.usenet.blueworldhosting.com, read over
    plain NNTP.


    RESULT

    A list of spam Message-IDs as INN history lines.
    Load them into a fresh INN and it refuses those articles as duplicates. Cleanfeed never sees them.

    There are two lists:

    current 35.8M Message-IDs
    core 27.5M, flagged by each of the last three passes

    Each entry says what flagged it: label, NoCeM, rule or local model.
    About 3% of the archive is flagged.
    We estimate 1 in 22 flags is a real article.
    The core list makes fewer mistakes.

    This is a backlog cleanup.
    99.6% of the flagged articles are older than 2025.
    Most are from 1996-2010, about 25M.
    The Google Groups wave adds 2.0M in 2023 and 0.6M in 2024.
    2025 and 2026 together have about 140,000.
    New spam is cleanfeed-ng's job.


    SIZE

    54,580 groups, 1.5 billion overview lines.
    Crossposts count once, so about 1.15 billion articles.
    A full pass reads all of OVER on 4 connections with COMPRESS DEFLATE,
    capped at 500 Mbit/s.
    It takes 2.5 to 3.5 hours.
    606 GB of overview text came over the wire as 102 GB.


    HOW

    Labelling every article with a language model would cost about $20,000.
    So we label a sample and train a small local model on it.

    1. Sample.
    One random window of articles per group, about 400,000 in all.
    Claude Haiku labelled them from the headers: spam, ham or unsure.
    The prompt says losing a real article is worse than letting spam
    through.

    2. Local model.
    Logistic regression on the headers: Subject, From, groups,
    crossposts, line count, reply or not.
    It does 23,000 overview lines per second per core.
    It calls spam only at 97% precision, ham at 98%.
    Everything in between is let through.

    3. The undecided.
    Haiku labels them again, now with the first 600 bytes of the body.
    One label per From+Subject+line count covers all copies.
    This runs through the batch API at half price.
    Batches of 100,000 came back in 10 to 17 minutes.

    4. Other sources.
    - NoCeM notices: 60.8M Message-IDs.
    We skip four issuers.
    Too much of what they listed was ordinary posts:
    UltraHippo 1,077 of 1,523 checked (71%)
    robot@pasdenom.info 27 of 28
    nocem-fr@alphanet.ch 49 of 83
    i2pn2-nocem@i2pn2.org 83 of 305 (27%, mostly de.*)
    UltraHippo alone listed 44.8M of the 60.8M.
    - Subject rules, not applied to replies or abuse groups.
    - A From blacklist (Adam's, see below).
    - Copy count: 20 or more copies of the same From+Subject+lines,
    unless labelled ham.

    14.8M keys are labelled: 11.3M ham, 2.5M spam, 1.0M unsure.


    CHECKS

    After each pass we pull a few thousand random flagged articles per
    source.
    Haiku labels them again with the body.
    Where it disagrees, Claude Sonnet decides.
    These articles are never used for training.

    Share confirmed as spam:

    flagged by first third now flags now
    language-model labels 95.8% 78.2% 92.9% 7.3M
    NoCeM 64.8% 94.2% 93.8% 4.5M
    copy count 54.5% 85.4% 95.8% 2.5M
    subject rules, From 90.3% 98.7% 98.7% 2.0M
    local model 71.8% 87.8% ~93%* 19.5M

    flagged in all 52.5M 48.1M 35.8M
    estimated spam 37.0M 41.5M ~33.7M
    estimated not spam 15.5M 6.0M ~1.6M
    undecided, let through 164.0M 69.8M 17.8M

    * not spot-checked yet, estimated

    The third pass caught more spam, but 1 in 8 flags was a real article.
    Most of those were job ads and classifieds in groups made for them.
    Spam reports in news.admin.net-abuse.* got hit too.
    Offers now count as spam only when posted where they don't belong.


    COST

    $378 in API fees, at list price, batch at half.
    4.6 billion input tokens, 0.48 billion output, almost all Haiku.
    The Sonnet checks are $9.94 of it.
    Plus one machine and about 100 GB of NNTP per pass.


    DISK

    SQLite database 24 GB (NoCeM 9.3, flags 7.5, labels 5.2)
    history, current list 35.8M lines, about 2 GB


    LOOK AT IT

    https://mbox-mo1.pugleaf.net:8182/

    Read-only, no login.

    - Groups: type a prefix, see the flagged count and share per group.
    - Browse: enter a group name.
    It shows the newest articles, marked "in filter", "unsure" or
    "clean".
    Click the article number to read one.
    - Look up: paste a Message-ID to see if and why it is flagged.
    - Labels and NoCeM: the raw labels and every NoCeM issuer.

    Group and article pages are limited to 20 a minute.


    Adam W. <aw@somewhere.invalid> wrote:
    a blacklist of Froms, all Thai subjects and froms, some keywords
    in Subject related to Kindle, and one blacklisted URL in body.

    We tested your rules against our labels.

    - From list: 266 of your 624 addresses show up, on 1,847 articles.
    96.6% of those were spam.
    We added the list.
    In the last pass it flagged 74,000 articles.
    67 of 70 checked were spam.
    - Kindle rule: 145 labelled, all spam.
    Added too: 6,900 flagged, 6 of 6 checked were spam.
    - 0xE0 can't be used archive-wide.
    It is "a grave" in Latin-1 and starts the Indic scripts in UTF-8.


    CODE AND DATA

    code (MIT)
    https://github.com/go-while/usenet-spamclassifier

    data (CC0)
    https://archive.org/details/usenet-spamclassifier-bwh-2026-11-09

    The data is 7z-compressed TSV.
    It has both lists, all six passes, the labels, the spot checks, NoCeM,
    the history file, the models and the full database.
    No article bodies.
    archive.org also offers it as a torrent.

    Work in progress: data upload to archive.
    --
    .......
    Billy G. (go-while)
    https://pugleaf.net
    @Newsgroup: rocksolid.nodes.help
    irc.pugleaf.net:6697 (SSL) #lounge
    TOR-IRC: ij7fmstcudrwyty5p6iloodeenur4wksovlhbyrfubat7eixplmrxvqd.onion:6667 --- Synchronet 3.22a-Linux NewsLink 1.2
  • From Colin Macleod@user7@newsgrouper.org.invalid to news.software.nntp on Mon Nov 16 17:42:06 2026
    From Newsgroup: news.software.nntp

    "Billy G." <42-contact-42@pugleaf.net> posted:

    CODE AND DATA

    code (MIT)
    https://github.com/go-while/usenet-spamclassifier

    data (CC0)
    https://archive.org/details/usenet-spamclassifier-bwh-2026-11-09


    Thanks very much for this. I've just downloaded groups.tsv and flagged-core-in-last-three-passes.tsv . My plan is to:

    - Load the message-ids from the flagged file into a sqlite spam db.
    - Use a modified version of my db_load_over script to pull XOVER
    data from archive.usenet.blueworldhosting.com, filtering out those
    articles with message-ids in the spam db.
    - From this populate a new cleaner sqlite overview db for Newsgrouper.
    --
    Colin Macleod ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ https://cmacleod.me.uk
    AI Bros: All your bits are belong to us!

    FEED GAZA NOW! SAVE GAZA NOW! HOUSE GAZA NOW! FEED GAZA NOW!
    --- Synchronet 3.22a-Linux NewsLink 1.2