From Newsgroup: news.software.nntp

Hi,

I need to shrink one of my cycbuffs, but keeping articles (including their numbers) intact. Creating a new cycbuff, moving articles to it, and then adding it to inn instead of the old cycbuff, will also do the trick. Is it possible? If yes, then how to best approach it? It would be best if I
didn't have to rebuild overview after that...

I'm using a word "cycbuff", not "metacycbuff", because it's a single file
(so a metacycbuff I want to shrink consists of only one cycbuff).

Thanks.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Hi Adam,

I need to shrink one of my cycbuffs, but keeping articles (including their numbers) intact. Creating a new cycbuff, moving articles to it, and then adding it to inn instead of the old cycbuff, will also do the trick. Is it possible? If yes, then how to best approach it? It would be best if I
didn't have to rebuild overview after that...

I do not unfortunately know an easy way to achieve that.
I am aware of a program named "respool" in the contrib directory of INN
but I have never tested it... It would do what you want according to
its comment "Refile articles into the storage manager under the current storage.conf rules, deleting articles from their old place in the spool." However, rebuilding overview would still be needed.

Usually, if I want to change where or how I store articles, I set up
another instance of INN and refeed all my articles to this new instance
(in slave mode so as to keep the original article numbers).
--
Julien |eLIE

-2-aCorruptissima republica plurimae leges.-a-+

--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Julien +LIE <iulius@nom-de-mon-site.com.invalid> wrote:

Usually, if I want to change where or how I store articles, I set up
another instance of INN and refeed all my articles to this new instance
(in slave mode so as to keep the original article numbers).

I have to move my server to another machine and want to reshape the
spooling system, so I thought about using the slave mode, but I'm not convinced that articles on the slave server will have exactly duplicated numbering. I'll feel safer just moving files. Or are my fears
unsubstantiated?

I'm worried about the word "attempts" in the inn.conf doc:

https://linux.die.net/man/5/inn.conf

"If set, INN attempts to duplicate exactly the article numbering"

Plus this:

"The upstream should be careful to always feed articles in order
(innfeed(8) can have problems with this in the event of a backlog)."

How to do it?

In general, slave mode doesn't seem to be convincing, as if anything goes wrong with numbering, results might be unpleasant for users, but hard to
spot for me...

But maybe I'm overparanoid? I never used this mode before.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Hi Adam,

I have to move my server to another machine and want to reshape the
spooling system

That's the opportunity to improve the design of your storage method(s).

so I thought about using the slave mode, but I'm not
convinced that articles on the slave server will have exactly duplicated numbering. I'll feel safer just moving files. Or are my fears unsubstantiated?

The slave mode (xrefslave set to true in inn.conf) will keep the same
article numbers; you can rely on it.
Please have a look at "6.4 Feed all articles on a server to another server":
https://www.eyrie.org/~eagle/faqs/inn.html#S6.4

I'm worried about the word "attempts" in the inn.conf doc:

https://linux.die.net/man/5/inn.conf

"If set, INN attempts to duplicate exactly the article numbering"

I have just had a look at how innd behaves if it does not manage to
parse the Xref header field containing the original article number. It
just rejects the article and does not store it.

The wording in the documentation is unfortunate. I suggest changing it
to "If set, INN duplicates exactly the article numbering [...]" and
adding "If there is no Xref header field or it is unparseable, the
article is rejected."

Plus this:

"The upstream should be careful to always feed articles in order
(innfeed(8) can have problems with this in the event of a backlog)."

How to do it?

If you follow the procedure in the above FAQ, the articles are sent by
arrival time on the old news server, so the article numbers are in order (assuming it was not running itself as a slave).

In the wording in the documentation, I doubt innfeed will actually have
a problem. I would suggest to reword it this way:

The upstream server should be careful to feed articles in order so as
to keep the history file of the slave server chronologically ordered
(news readers fetching articles with NEWNEWS, that is to say by
arrival time, may otherwise retrieve articles in disorder).

See "doc/FAQ" (subject 6.4) for an example of use of this parameter
when feeding all articles on a server to another server, keeping their
chronological ordering.

In general, slave mode doesn't seem to be convincing, as if anything goes wrong with numbering, results might be unpleasant for users, but hard to
spot for me...

But maybe I'm overparanoid? I never used this mode before.

I hope the new wordings are now better. Thanks for pointing that out!

I have used several times that parameter when installing a new news
server, and never run into any issue.
--
Julien |eLIE

-2-aUn classique est quelque chose que tout le monde voudrait avoir lu et
que personne ne veut lire.-a-+ (Mark Twain)

--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Julien +LIE <iulius@nom-de-mon-site.com.invalid> wrote:

I have to move my server to another machine and want to reshape the
spooling system

That's the opportunity to improve the design of your storage method(s).

Yup... But it's too late now, I moved the server by copying files. Which didn't solve the problem, obviously, but I wanted to be 1000% sure users
won't run into any numbering problems.

The slave mode (xrefslave set to true in inn.conf) will keep the same article numbers; you can rely on it.
Please have a look at "6.4 Feed all articles on a server to another server":
https://www.eyrie.org/~eagle/faqs/inn.html#S6.4

" and qx/sm -q -H "$_" | grep Xref/ =~ / fr\.\S+:\d+/"

perl always sounded like magic spells to me...

The wording in the documentation is unfortunate. I suggest changing it
to "If set, INN duplicates exactly the article numbering [...]" and
adding "If there is no Xref header field or it is unparseable, the
article is rejected."

This sounds more convincing :)

I'm wondering if I should set up a new server (on the same machine) with
my desired spooling layout, feed articles to it using this method, and
then just swap the databases (active, overview, spool + spool config).
Would it work (and keep the article numbers exactly as they are now)?

If you follow the procedure in the above FAQ, the articles are sent by arrival time on the old news server, so the article numbers are in order (assuming it was not running itself as a slave).

This is a little confusing to me.

"The result file contains tokens ordered by arrival time on the old server (which is usually roughly the same as the posting time). In case the
history file was not populated chronologically, it is better to sort it by posting time so that articles are fed in the right order."

If the history file on the old server isn't in order (so it's messed up,
some older articles have newer numbers) and I sort it this way, I'll end
up having gaps in the numbering, right?

So, let's say there's an empty group on a new server, and the first
article sent has Xref 3. There's no 1 or 2 yet. What would happen then?
Would the article be accepted?

If yes, then why is it important to send articles in date order, if
they'll end up with correct numbers anyway?

If not, then why is it important to send articles in date order, if they
can be out of order on the source server? Shouldn't we send them based on
Xref (article number) order? Numbering will be messed up on the new server
in the same way it's messed up on the old server then, but with xref slave it's desirable, isn't it?

I hope the new wordings are now better. Thanks for pointing that out!

Yes, it's more convincing now :)
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On Sun, 21 Dec 2025 11:05:40 +0100
Julien |eLIE <iulius@nom-de-mon-site.com.invalid> wrote:

Hi Adam,

I need to shrink one of my cycbuffs, but keeping articles
(including their numbers) intact. Creating a new cycbuff, moving
articles to it, and then adding it to inn instead of the old
cycbuff, will also do the trick. Is it possible? If yes, then how
to best approach it? It would be best if I didn't have to rebuild
overview after that...

I do not unfortunately know an easy way to achieve that.
I am aware of a program named "respool" in the contrib directory of
INN but I have never tested it... It would do what you want
according to its comment "Refile articles into the storage manager
under the current storage.conf rules, deleting articles from their
old place in the spool." However, rebuilding overview would still be
needed.

Usually, if I want to change where or how I store articles, I set up
another instance of INN and refeed all my articles to this new
instance (in slave mode so as to keep the original article numbers).

I have mentioned before that there really needs to be a way to compress articles within a cycbuff file. It's probably not as simple as:
1. Find the first deleted article and calculate the number of bytes
used.
2. take that number of bytes from the end of the first article and
rewrite them noting the end point. Re-write some pointer to where the
new article is.
3. rinse and repeat sort of stuff until you've gone through the file
and shuffled up all the bytes over the deleted articles.
Like compressing your message file on your Apple II BBS but on a bigger
scale :)
Obviously it would take someone who knows the format of cycbuffs (not
me) knowledge how the pointers and stuff work with inn (also, not me)
and can do it in C so it's fast (definitely not me) :)
--
End Of The Line BBS - Plano, TX
telnet endofthelinebbs.com 23
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Hi Adam,

" and qx/sm -q -H "$_" | grep Xref/ =~ / fr\.\S+:\d+/"

perl always sounded like magic spells to me...

sm -q -H "$_" | grep Xref

extracts the Xref header field from the article, using sm:
https://www.eyrie.org/~eagle/software/inn/docs/sm.html

this header =~ / fr\.\S+:\d+/

then checks whether the article was stored in a newsgroup belonging to
the fr.* hierarchy. This way, only articles for fr.* match the condition.

I'm wondering if I should set up a new server (on the same machine) with
my desired spooling layout, feed articles to it using this method, and
then just swap the databases (active, overview, spool + spool config).
Would it work (and keep the article numbers exactly as they are now)?

Yes, you can do that if you want. It would retain the article numbers.

"The result file contains tokens ordered by arrival time on the old server (which is usually roughly the same as the posting time). In case the
history file was not populated chronologically, it is better to sort it by posting time so that articles are fed in the right order."

If the history file on the old server isn't in order (so it's messed up,
some older articles have newer numbers) and I sort it this way, I'll end
up having gaps in the numbering, right?

Gaps are allowed in the numbering anyway (cancelled articles generate
gaps as their article number is not reused).

So, let's say there's an empty group on a new server, and the first
article sent has Xref 3. There's no 1 or 2 yet. What would happen then?
Would the article be accepted?

Yes, the article will be accepted, and still be article number 3 in the
new news server.

If yes, then why is it important to send articles in date order, if
they'll end up with correct numbers anyway?

Some people prefer that because of the way news clients work.
If you have only article number 3 and a client reads the newsgroup, it
will fetch this article. Afterwards, it will only try to download
articles whose number is higher than 3. So if you receive articles 1
and 2 later, the client won't know they exist.

Note that when the server has article 3, and a bit later 1 and 2, and a
client reads the newsgroup at that time, it will naturally fetch all
these 3 articles.

After all, it only matters for clients if they access the news server
during the time of the migration. Maybe this precision should be added
to the documentation to make it clearer?

If not, then why is it important to send articles in date order, if they
can be out of order on the source server? Shouldn't we send them based on Xref (article number) order? Numbering will be messed up on the new server> in the same way it's messed up on the old server then, but with xref

slave

it's desirable, isn't it?

Yes, the slave will have the same messed up numbers, if any, as the old server. Migrating is just the opportunity to do a bit of reordering if desired :)
Xref order corresponds to arrival time.
--
Julien |eLIE

-2-aJe suis capable du meilleur comme du pire, mais pour le pire, c'est
moi le meilleur.-a-+ (Coluche)

--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

In addition to my previous message:

If yes, then why is it important to send articles in date order, if
they'll end up with correct numbers anyway?

Some people prefer that because of the way news clients work.
If you have only article number 3 and a client reads the newsgroup, it
will fetch this article.-a Afterwards, it will only try to download
articles whose number is higher than 3.-a So if you receive articles 1
and 2 later, the client won't know they exist.

Note that when the server has article 3, and a bit later 1 and 2, and a client reads the newsgroup at that time, it will naturally fetch all
these 3 articles.

After all, it only matters for clients if they access the news server
during the time of the migration.-a Maybe this precision should be added
to the documentation to make it clearer?

It will also affect the expiration process. In self-expiring storage
methods like CNFS, articles expire in the same order they have arrived
when your cyclic buffer wraps. Older articles received out of order
will then expire after more recent ones.
With other storage methods, you may want to use the -p flag with expire
and expireover to make expiration decisions relative to posting dates
instead of arrival dates.
--
Julien |eLIE

-2-aLes femmes pardonnent parfois |a celui qui brusque l'occasion, mais
jamais |a celui qui la manque.-a-+ (Talleyrand)

--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Hi Nigel,

I have mentioned before that there really needs to be a way to compress articles within a cycbuff file.

Yes, your feature request is not forgotten
https://github.com/InterNetNews/inn/issues/303

Since then, Jesse also asked for the possibility to compress (zlib)
articles stored in CNFS buffers.

Obviously it would take someone who knows the format of cycbuffs (not
me) knowledge how the pointers and stuff work with inn (also, not me)
and can do it in C so it's fast (definitely not me) :)

Yes, it needs knowledge and also the time to implement that.
--
Julien |eLIE

-2-aSum imus.-a-+

--- Synchronet 3.21a-Linux NewsLink 1.2