From Newsgroup: news.software.nntp

On Nov 18, 2025 at 8:44:13rC>AM CST, "Julien |eLIE" <iulius@nom-de-mon-site.com.invalid> wrote:

Hi Jesse,

I have been working with Billy G., having him push articles from his pugleaf >> instance to an INN instance running on an OmniOS VM. Tonight innd crashed with
the following:

Nov 16 22:33:33 omnios-inn innd: [ID 608925 news.crit] SERVER cant malloc
18446744073699012160 bytes at interface.c line 622: Not enough space

Is the INN instance running in slave mode?
Do you happen to have the culprit article which made INN crash?

If the INN instance is not running in slave mode, well, there's another
issue to find...

It is not running in slave mode. Not sure if this would point to anything meaningful, but these are two parameters that I've modified in inn.conf:

datamovethreshold: 1048576
icdsynccount: 300000

For reasons I haven't dug deeper on, INN is *painfully* slow to write to the history file on ZFS filesystems (FreeBSD and illumos distros tested, have briefly tested on Linux with XFS and it was much better).

Leaving icdsynccount at the default of 10 results in horrid throughput. No matter the number of connections throughput doesn't go much above 1-3Mbps. I played around *a lot* with various values and found the best throughput above 100000. At the current value, we get well over 100Mbps throughput. Obviously this is 'risky' but in this use-case we're just playing around.

We've ran through a few cycles of Billy feeding over 1 billion articles to
this instance, and this was the first time it crashed. That said, this round I had him add some additional newsgroups, so its possible there could be a very strangely formatted article.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

In article <87tsyrf9ag.fsf@hope.eyrie.org>,
Russ Allbery <eagle@eyrie.org> wrote:

I am still a bit confused, though, because the value of groupslen doesn't >seem to be something sensible like -2. If I'm doing the math correctly,
it's something more like -10539456, which is a hard value to justify if
the problem is the above bug.

The number in hex is 0xffffffffff5f2e40, and the non-f part
corresponds to the printable characters _.@ which is hardly conclusive
but might be due to being overwritten by a malformed mail address.

Are there any other examples?

-- Richard
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On Nov 18, 2025 at 2:04:27rC>PM CST, "Richard Tobin" <Richard Tobin> wrote:

In article <87tsyrf9ag.fsf@hope.eyrie.org>,
Russ Allbery <eagle@eyrie.org> wrote:

I am still a bit confused, though, because the value of groupslen doesn't
seem to be something sensible like -2. If I'm doing the math correctly,
it's something more like -10539456, which is a hard value to justify if
the problem is the above bug.

The number in hex is 0xffffffffff5f2e40, and the non-f part
corresponds to the printable characters _.@ which is hardly conclusive
but might be due to being overwritten by a malformed mail address.

Are there any other examples?

-- Richard

If it could be helpful to anyone who wants to take a look, I've placed the
core dump that was generated at https://usenet.blueworldhosting.com/core

# du -A core
119533 core

# sha256sum core 04fd0e53ca39252d456d81fd128609fbb8b667987e2c6b358c3fa0b17fb7ee31 core

Unfortunately with the buffered 'news' logfile, I don't think I can determine exactly which article may have caused the crash.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

In article <10fijun$ac8$1@nnrp.usenet.blueworldhosting.com>,
Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:

If it could be helpful to anyone who wants to take a look, I've placed the >core dump that was generated at https://usenet.blueworldhosting.com/core

I started to download it, but the connection is very slow and it would
take over an hour! Can you gzip it? (Core files often compress very
well.)

-- Richard
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On Nov 18, 2025 at 3:07:25rC>PM CST, "Richard Tobin" <Richard Tobin> wrote:

In article <10fijun$ac8$1@nnrp.usenet.blueworldhosting.com>,
Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:

If it could be helpful to anyone who wants to take a look, I've placed the >> core dump that was generated at https://usenet.blueworldhosting.com/core

I started to download it, but the connection is very slow and it would
take over an hour! Can you gzip it? (Core files often compress very
well.)

-- Richard

I've been having provider issues and awaiting an appointment to swap equipment so the connection has been unreliable lately.

Try: https://usenet.blueworldhosting.com/core.gz
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

richard@cogsci.ed.ac.uk (Richard Tobin) writes:

Russ Allbery <eagle@eyrie.org> wrote:

I am still a bit confused, though, because the value of groupslen doesn't >>seem to be something sensible like -2. If I'm doing the math correctly, >>it's something more like -10539456, which is a hard value to justify if
the problem is the above bug.

The number in hex is 0xffffffffff5f2e40, and the non-f part
corresponds to the printable characters _.@ which is hardly conclusive
but might be due to being overwritten by a malformed mail address.

The proximate code is:

static bool
MatchGroups(const char *g, int len, const char *pattern, bool exactmatch)
{
char *group, *groups, *q;
int i, lastwhite;
enum uwildmat matched;
bool wanted = false;

q = groups = xmalloc(len + 1);

This is consistent with len = -10539457.

The length comes from ARTstore:

if (innconf->storeonxref) {
arth.groups = data->Replic;
arth.groupslen = data->ReplicLength;
} else {
arth.groups = HDR(HDR__NEWSGROUPS);
arth.groupslen = HDR_LEN(HDR__NEWSGROUPS);
}

I donrCOt think JesserCOs said what storeonxref is set to but the default is true, so IrCOm going to assume thatrCOs what it is until I hear otherwise.

ReplicLength is set in ARTassignnumbers, which is the kind of ad-hoc
string builder that tends to be full of bugs. It has a number of issues
but the most relevant is:

* If the collection of group names (and article numbers etc) adds up to
more than 2GB then len will overflow to negative. This could explain
the outcome seen, although it would need a pathologically large input
article.

Other observations about ARTassignnumbers, which I donrCOt think can
explain the behavior seen:

* A pathhost bigger than the default xref buffer size (2049 bytes) would
generate a buffer overflow.

* An article number longer than ARTNUMPRINTSIZE could fail to expand the
buffer when needed. Currently you get ten digits, and the starting
buffer size is 2049 bytes, so this is probably not an issue in
real life.

* If there are no groups at all then the buffer expansion never happens
at all, so nothing explicitly accounts for the CRLF at the end -
i.e. the commented assumption rCychecked during the reallocation aboverCO
is violated. But the default buffer size is more than enough in that
case.

* p and len are tracked separately, which is pointless and invites
error, but I donrCOt see any way to get them out of sync apart from the
overflow issue described above.

* ReplicLength is set in an unnecessarily complicated way, because
len - (q + 1 - data->Xref) == Path.used - 1
but I donrCOt see any realistic way for that to end up negative or for
q+1 to run past the end of the buffer.
--
https://www.greenend.org.uk/rjk/
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On Nov 18, 2025 at 5:19:01rC>PM CST, "Richard Kettlewell" <invalid@invalid.invalid> wrote:

richard@cogsci.ed.ac.uk (Richard Tobin) writes:

Russ Allbery <eagle@eyrie.org> wrote:

I am still a bit confused, though, because the value of groupslen doesn't >>> seem to be something sensible like -2. If I'm doing the math correctly,
it's something more like -10539456, which is a hard value to justify if
the problem is the above bug.

The number in hex is 0xffffffffff5f2e40, and the non-f part
corresponds to the printable characters _.@ which is hardly conclusive
but might be due to being overwritten by a malformed mail address.

The proximate code is:

static bool
MatchGroups(const char *g, int len, const char *pattern, bool exactmatch)
{
char *group, *groups, *q;
int i, lastwhite;
enum uwildmat matched;
bool wanted = false;

q = groups = xmalloc(len + 1);

This is consistent with len = -10539457.

The length comes from ARTstore:

if (innconf->storeonxref) {
arth.groups = data->Replic;
arth.groupslen = data->ReplicLength;
} else {
arth.groups = HDR(HDR__NEWSGROUPS);
arth.groupslen = HDR_LEN(HDR__NEWSGROUPS);
}

I donrCOt think JesserCOs said what storeonxref is set to but the default is true, so IrCOm going to assume thatrCOs what it is until I hear otherwise.

You are correct, storeonxref is true and xrefslave is false.

ReplicLength is set in ARTassignnumbers, which is the kind of ad-hoc
string builder that tends to be full of bugs. It has a number of issues
but the most relevant is:

* If the collection of group names (and article numbers etc) adds up to
more than 2GB then len will overflow to negative. This could explain
the outcome seen, although it would need a pathologically large input
article.

Well, I do have icdsynccount set to 300000, and we are feeding this machine over a billion articles from another machine, so the rate/throughput of incoming articles is very high.
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:

"Richard Kettlewell" <invalid@invalid.invalid> wrote:

ReplicLength is set in ARTassignnumbers, which is the kind of ad-hoc
string builder that tends to be full of bugs. It has a number of issues
but the most relevant is:

* If the collection of group names (and article numbers etc) adds up to
more than 2GB then len will overflow to negative. This could explain
the outcome seen, although it would need a pathologically large input
article.

Well, I do have icdsynccount set to 300000, and we are feeding this
machine over a billion articles from another machine, so the
rate/throughput of incoming articles is very high.

To trigger the issue in ARTassignnumbers yourCOd need a single article
where the collection of group names plus article numbers added up to
more than 2GB (in fact nearly 4GB for the specific value seen here).
ItrCOd take a crosspost to millions of groups and the article header alone would be multiple gigabytes.

If your peer does have such a large article then thatrCOs probably the
cause. It doensrCOt seem likely but you never know l-)

If not then further analysis is needed. The coredump posted might shed
light on it but I donrCOt have a debugger that understands Solaris
coredumps (and if finding one is harder than apt-get install something,
itrCOs probably not happening).
--
https://www.greenend.org.uk/rjk/
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On Nov 19, 2025 at 2:22:11rC>AM CST, "Richard Kettlewell" <invalid@invalid.invalid> wrote:

To trigger the issue in ARTassignnumbers yourCOd need a single article
where the collection of group names plus article numbers added up to
more than 2GB (in fact nearly 4GB for the specific value seen here).
ItrCOd take a crosspost to millions of groups and the article header alone would be multiple gigabytes.

If your peer does have such a large article then thatrCOs probably the
cause. It doensrCOt seem likely but you never know l-)

The articles are coming from a newly written NNTP stack and the transfer tool has had some bugs we've worked out together. It's possible we hit a bug on the sending side and received some garbage.

If not then further analysis is needed. The coredump posted might shed
light on it but I donrCOt have a debugger that understands Solaris
coredumps (and if finding one is harder than apt-get install something, itrCOs probably not happening).

Argh, I didn't think about that. I doubt mdb or other compatible debuggers are available on Linux. If anyone can direct me what to do with the core file
using mdb/gdb I'm happy to do so.

That said, OmniOS is easy to install and OpenIndiana has a Live-CD. (I don't expect anyone to go that far to help, but I like pushing people away from Linux.)
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

Hi Jesse,

If it could be helpful to anyone who wants to take a look, I've placed the core dump that was generated at https://usenet.blueworldhosting.com/core

Thanks for the core file. Unfortunately, I do not manage to get
anything from it. I have successfully opened it with mdb on Solaris
11.4 but do not actually know how mdb works.
I did not manage to find helpful examples in online documentation too.

$c

libinn.so.9.0.2`x_malloc+0x4f()
libinnstorage.so.3.1.3`SMgetsub+0x1b8()
libinnstorage.so.3.1.3`SMstore+0x87()

Does someone know whether it is possible to have some insights about the variables and their contents in the SMgetsub and x_malloc calls?
--
Julien |eLIE

-2-aTuto, cito, iucunde.-a-+ (Esculape)

--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On Nov 25, 2025 at 12:22:09rC>AM CST, "Julien |eLIE" <iulius@nom-de-mon-site.com.invalid> wrote:

Hi Jesse,

If it could be helpful to anyone who wants to take a look, I've placed the >> core dump that was generated at https://usenet.blueworldhosting.com/core

Thanks for the core file. Unfortunately, I do not manage to get
anything from it. I have successfully opened it with mdb on Solaris
11.4 but do not actually know how mdb works.
I did not manage to find helpful examples in online documentation too.

$c

libinn.so.9.0.2`x_malloc+0x4f()
libinnstorage.so.3.1.3`SMgetsub+0x1b8()
libinnstorage.so.3.1.3`SMstore+0x87()

Does someone know whether it is possible to have some insights about the variables and their contents in the SMgetsub and x_malloc calls?

gdb should be available on Solaris 11.4, if that's more comfortable. I don't currently have Solaris 11.4 installed, but see gdb in Oracle's online man pages.

I'm swimming in the middle of the ocean without a raft when it comes to this stuff. :-)
--- Synchronet 3.21a-Linux NewsLink 1.2

From Newsgroup: news.software.nntp

On Nov 25, 2025 at 6:55:13rC>AM CST, "Jesse Rehmer" <jesse.rehmer@blueworldhosting.com> wrote:

On Nov 25, 2025 at 12:22:09rC>AM CST, "Julien |eLIE" <iulius@nom-de-mon-site.com.invalid> wrote:

Hi Jesse,

If it could be helpful to anyone who wants to take a look, I've placed the >>> core dump that was generated at https://usenet.blueworldhosting.com/core

Thanks for the core file. Unfortunately, I do not manage to get
anything from it. I have successfully opened it with mdb on Solaris
11.4 but do not actually know how mdb works.
I did not manage to find helpful examples in online documentation too.

$c

libinn.so.9.0.2`x_malloc+0x4f()
libinnstorage.so.3.1.3`SMgetsub+0x1b8()
libinnstorage.so.3.1.3`SMstore+0x87()

Does someone know whether it is possible to have some insights about the
variables and their contents in the SMgetsub and x_malloc calls?

gdb should be available on Solaris 11.4, if that's more comfortable. I don't currently have Solaris 11.4 installed, but see gdb in Oracle's online man pages.

I'm swimming in the middle of the ocean without a raft when it comes to this stuff. :-)

Cross-posting to comp.unix.solaris (probably should have done that from the start).

If anyone checking comp.unix.solaris is interested in helping us debug an
issue with INN on OmniOS, we could use some pointers.

I have a core file from a crash that we'd like to get more details on, but
none of us are very familiar with debugging on illumos or Solaris.
--- Synchronet 3.21a-Linux NewsLink 1.2