| Sysop: | Amessyroom |
|---|---|
| Location: | Fayetteville, NC |
| Users: | 23 |
| Nodes: | 6 (0 / 6) |
| Uptime: | 46:48:29 |
| Calls: | 583 |
| Files: | 1,138 |
| Messages: | 111,069 |
Hi Jesse,
Over these past years, I have often seen questions about syntax checks.
Maybe laxmid should allow more Message-IDs than only the ones with ".."
and two "@"?
Strictly speaking, a dot (".") must be followed by another non-special chars, so <a.@b> and <a@b.> are invalid per RFC.
I suggest to change the behaviour of laxmid so that innd accepts even
more Message-IDs. For instance in the common dot-atom-text syntax, just checking we have "<", at least one non-special char, "@", at least onenon-special char, and ">".
no-fold-literal is kept untouched but dot-atom-text is changed.
The syntax per RFC is:
msg-id = "<" msg-id-core ">"
msg-id-core = id-left "@" id-right
id-left = dot-atom-text
id-right = dot-atom-text / no-fold-literal
dot-atom-text = 1*atext *("." 1*atext)
no-fold-literal = "[" *mdtext "]"
mdtext = %d33-61 / ; The rest of the US-ASCII
%d63-90 / ; characters not including
%d94-126 ; ">", "[", "]", or "\"
atext = ALPHA / DIGIT / ; Printable US-ASCII
"!" / "#" / ; characters not including
"$" / "%" / ; specials. Used for atoms.
"&" / "'" /
"*" / "+" /
"-" / "/" /
"=" / "?" /
"^" / "_" /
"`" / "{" /
"|" / "}" /
"~"
laxmid would accept for innd:
dot-atom-text = 1*(atext / "." / "@")
At least, I think it would cope with all Message-IDs in the wild. (Are
there ones without any "@" at all?)
As for nnrpd, laxmid would go on having the current behaviour of
allowing ".." and two "@" as this was a request in 2017 from a news
admin with users having broken posting agents sending such Message-IDs.
No need for now to allow the injection of even more broken Message-IDs.
Any thoughts about that change?
laxmid would accept for innd:
dot-atom-text = 1*(atext / "." / "@")
At least, I think it would cope with all Message-IDs in the wild.
As for nnrpd, laxmid would go on having the current behaviour of
allowing ".." and two "@" as this was a request in 2017 from a news
admin with users having broken posting agents sending such Message-IDs.
No need for now to allow the injection of even more broken Message-IDs.
Any thoughts about that change?
Bringing this up again because I have found BNews Message-IDs cannot be injected without modification, and there are a ton of them from various sources I'd rather not attempt to modify. Once source is online via NNTP, so easy to use pullnews or suck, which would be the path of least resistance.
<bnews.sri-unix.2509> - 435 Syntax error in message-ID
Hi Jesse,
laxmid would accept for innd:
dot-atom-text = 1*(atext / "." / "@")
At least, I think it would cope with all Message-IDs in the wild.
As for nnrpd, laxmid would go on having the current behaviour of
allowing ".." and two "@" as this was a request in 2017 from a news
admin with users having broken posting agents sending such Message-IDs.
No need for now to allow the injection of even more broken Message-IDs.
Any thoughts about that change?
Bringing this up again because I have found BNews Message-IDs cannot be
injected without modification, and there are a ton of them from various
sources I'd rather not attempt to modify. Once source is online via NNTP, so >> easy to use pullnews or suck, which would be the path of least resistance. >>
<bnews.sri-unix.2509> - 435 Syntax error in message-ID
Sorry for having forgotten your request. I bet I was waiting for your approval
of my suggestion of change (innd would accept 0 to 2 '@', but not nnrpd whose behaviour would remain unchanged) before starting to work on it.
I think the following patch will work:
--- a/lib/messageid.c
+++ b/lib/messageid.c
@@ -127,8 +127,8 @@ InitializeMessageIDcclass(void)
** When stripspaces is true, whitespace at the beginning and at the end
** of MessageID are discarded.
**
-** When laxsyntax is true, '@' can occur twice in MessageID, and '..' is -** also accepted in the left part of MessageID.
+** When laxsyntax is true, '@' can occur twice in MessageID, or never occur,
+** and '..' is also accepted in the left part of MessageID.
*/
bool
IsValidMessageID(const char *MessageID, bool stripspaces, bool laxsyntax) @@ -155,6 +155,12 @@ IsValidMessageID(const char *MessageID, bool stripspaces,
bool laxsyntax)
/* Scan local-part: "<dot-atom-text". */> if (*p++ != '<')>
return false;
+
+ /* In case there's no '@' in the Message-ID and laxsyntax is set, just
+ * check the syntax of the Message-ID as though it had no left part. */ + if (laxsyntax && strchr((const char *) p, '@') == NULL)
+ return IsValidRightPartMessageID((const char *) p, stripspaces, true);
+
for (;; p++) {
if (midatomchar(*p)) {
while (midatomchar(*++p))
--- a/nnrpd/post.c
+++ b/nnrpd/post.c
@@ -471,6 +471,10 @@ ProcessHeaders(char *idbuff, bool needmoderation)
if (!IsValidMessageID(HDR(HDR__MESSAGEID), true, laxmid)) {
return "Can't parse Message-ID header field body";
}
+ /* Do not accept a Message-ID without an '@', even if laxmid is set. */ + if (laxmid && strchr(HDR(HDR__MESSAGEID), '@') == NULL) {
+ return "Missing @ in Message-ID header field body";
+ }
/* Set the Path header field. */
if (HDR(HDR__PATH) == NULL || PERMaccessconf->strippath) {
If you can confirm it suits your need, and you are now able to inject BNews articles downloaded by pullnews, it would be great.
I'll also add a note in the documentation to warn that when laxmid is set, remote peers may reject articles with a syntactically invalid Message-ID.
This does get past the Message-ID header issue, but presents a new one with the Date header.
437 Bad "Date" header field -- "Fri Jul 9 03:46:46 1982"
I was looking at lib/date.c but it's a bit complex for me to digest. I
see references in comments to "lax mode" and not sure if this is an undocumented option or maybe removed in the past?
innd always uses lax mode for date parsing.
That date is in ctime(3) format, which isn't supported by INN even in lax mode. That format was already forbidden in the first article format
standard (RFC 850) from June 1983, and a lot of articles before that are
in the completely incompatible A News format that INN has never attempted
to parse. It looks like you have a transitional article that is in B News format but is still using the ctime(3) format for Date.
RFC 850 says:
Note in particular that ctime format:
Wdy Mon DD HH:MM:SS YYYY
is not acceptable because it is not a valid ARPANET date.
However, since older software still generates this format,
news implementations are encouraged to accept this format
and translate it into an acceptable format.
I wouldn't object to supporting this in INN in lax mode, but it's not entirely trivial to add without accidentally breaking something else
because the order of the date elements is significantly different than a standardized date. It would take someone a bit of time to figure out how
to safely incorporate it into parsedate_rfc5322_lax. (For example, the
code that skips over a leading day of the week would also currently skip
over the month.) It might be easiest to add a separate ctime parser and to just attempt a ctime parse whenever the date is otherwise invalid. I'm not sure how wide of a range of formats the old ctime dates came in.
Mostly they've all been in the same format, except a few outliers that
have invalid Date and Posted headers, but do have a Date-Received header that's more appropriate.
The outliers seem to follow this pattern:
Date: Wed, 31-Dec-69 18:59:59 EDT
Posted: Wed Dec 31 18:59:59 1969
Date-Received: Sun, 28-Jul-85 00:57:37 EDT
Date: Wed, 31-Dec-69 18:59:59 EDT
Posted: Wed Dec 31 18:59:59 1969
Date-Received: Sun, 28-Jul-85 00:57:37 EDT
Those I assume will require changing the Date header, at minimum before they'd
be accepted.
from email.utils import format_datetime,parsedate_to_datetime
format_datetime(parsedate_to_datetime("Wed, 31-Dec-69 18:59:59 EDT")) 'Wed, 31 Dec 1969 18:59:59 -0400'
I did find a few more Message-ID variations that are rejected:
<[OFFICE-3]GVT-RICH-490UQ> - 435 Syntax error in message-ID
<366@mimir..dmt.oz> - 435 Syntax error in message-ID
<[MC.LCS.MIT.EDU].851959.860315.KFL> - 435 Syntax error in message-ID
Hi Jesse,
Date: Wed, 31-Dec-69 18:59:59 EDT
Posted: Wed Dec 31 18:59:59 1969
Date-Received: Sun, 28-Jul-85 00:57:37 EDT
Those I assume will require changing the Date header, at minimum before they'd
be accepted.
Wouldn't it be possible to somehow rewrite the Date header field before injecting the article? (The old one could be kept in an X-Date header field.) There may be news readers that are unable to parse them too.
In Python:
'Wed, 31 Dec 1969 18:59:59 -0400'from email.utils import format_datetime,parsedate_to_datetime
format_datetime(parsedate_to_datetime("Wed, 31-Dec-69 18:59:59 EDT"))
I did find a few more Message-ID variations that are rejected:
<[OFFICE-3]GVT-RICH-490UQ> - 435 Syntax error in message-ID
<366@mimir..dmt.oz> - 435 Syntax error in message-ID
<[MC.LCS.MIT.EDU].851959.860315.KFL> - 435 Syntax error in message-ID
These are indeed invalid domain names. I am under the impression that
the laxsyntax check should just ensure there are 1 to 248 (authorized)
chars surrounded by brackets, without verifying the number, order and
place of '.', '[', ']', etc.
Date: Wed, 31-Dec-69 18:59:59 EDT
Wouldn't it be possible to somehow rewrite the Date header field before
injecting the article? (The old one could be kept in an X-Date header
field.) There may be news readers that are unable to parse them too.
Will have to do something to bring these articles in. Since these exist on a server that speaks (limited) NNTP, I am trying to bring in as many as possible
without having to download, modify, and inject.
I did find a few more Message-ID variations that are rejected:These are indeed invalid domain names. I am under the impression that
<[OFFICE-3]GVT-RICH-490UQ> - 435 Syntax error in message-ID
<366@mimir..dmt.oz> - 435 Syntax error in message-ID
<[MC.LCS.MIT.EDU].851959.860315.KFL> - 435 Syntax error in message-ID >>
the laxsyntax check should just ensure there are 1 to 248 (authorized)
chars surrounded by brackets, without verifying the number, order and
place of '.', '[', ']', etc.
In my humble opinion, that's all it should need to do. This seems inline with how Cyclone and Diablo seem to behave by default (at least how I see commercial operators have them configured).
Wouldn't it be possible to somehow rewrite the Date header field before injecting the article? (The old one could be kept in an X-Date header field.) There may be news readers that are unable to parse them too.
In Python:
from email.utils import format_datetime,parsedate_to_datetime
format_datetime(parsedate_to_datetime("Wed, 31-Dec-69 18:59:59 EDT")) 'Wed, 31 Dec 1969 18:59:59 -0400'
As your use is very specific, and writing a proper parsing of that kind of dates is time-consuming and complicated (as Russ told us), I wonder
whether the faster approach wouldn't be to add a new level of control in syntaxchecks:
syntaxchecks: [ laxmid laxdate ]
which would just take the current time (= arrival time) as the posting
date when the Date header field exists and is not parseable. The
information is recorded in the history file and overview for expiry
purpose, so it shouldn't break anything as far as I see.
Julien <iulius@nom-de-mon-site.com.invalid> writes:
As your use is very specific, and writing a proper parsing of that kind of >>dates is time-consuming and complicated (as Russ told us), I wonder
whether the faster approach wouldn't be to add a new level of control in >>syntaxchecks:
syntaxchecks: [ laxmid laxdate ]
which would just take the current time (= arrival time) as the posting
date when the Date header field exists and is not parseable. The >>information is recorded in the history file and overview for expiry >>purpose, so it shouldn't break anything as far as I see.
Maybe "ignoredate" in this case instead of "laxdate"? This is a bit
different than what "laxmid" means (ignore invalid message IDs and use the >message ID anyway), since it's ignoring the date entirely for the purposes
of all the other things INN does with dates. That way we would keep
laxdate in case we ever want to enable strict standards-enforcing date >parsing or provide a different laxer date parser that understands ctime(3) >and dashes, etc.
The actual field stored in overview for clients will still be the value of >the Date header so far as I can see, so that will be invalid (not parsable
by clients) unless you put something different into the overview. That's >already the case with the existing lax date parsing, so might not matter
and will be true for any of the proposals for handling old dates. I'm not >sure how many clients try to parse the date information in overview and do >something with it.
Russ Allbery <eagle@eyrie.org> wrote:
Julien <iulius@nom-de-mon-site.com.invalid> writes:
As your use is very specific, and writing a proper parsing of that kind of >>> dates is time-consuming and complicated (as Russ told us), I wonder
whether the faster approach wouldn't be to add a new level of control in >>> syntaxchecks:
syntaxchecks: [ laxmid laxdate ]
which would just take the current time (= arrival time) as the posting
date when the Date header field exists and is not parseable. The
information is recorded in the history file and overview for expiry
purpose, so it shouldn't break anything as far as I see.
Maybe "ignoredate" in this case instead of "laxdate"? This is a bit
different than what "laxmid" means (ignore invalid message IDs and use the >> message ID anyway), since it's ignoring the date entirely for the purposes >> of all the other things INN does with dates. That way we would keep
laxdate in case we ever want to enable strict standards-enforcing date
parsing or provide a different laxer date parser that understands ctime(3) >> and dashes, etc.
The actual field stored in overview for clients will still be the value of >> the Date header so far as I can see, so that will be invalid (not parsable >> by clients) unless you put something different into the overview. That's
already the case with the existing lax date parsing, so might not matter
and will be true for any of the proposals for handling old dates. I'm not
sure how many clients try to parse the date information in overview and do >> something with it.
Could I make a suggestion as a nonprogrammer but someone who has spent countless hours putting data into a consistent pattern or good syntax?
Think of the Date header as temporary and that at some point, it might
be nice if it reflected the original date but now in modern syntax. I'm suggesting this as a Date header that reflects when it was appended to
an archive is not going to be helpful to a newsreader when it comes to sorting. And a whole lot of articles are going to share an identical
Date header as articles are going to be appended in huge batches.
Add a special X- header with an explicit header reflecting where the
article came from in a specific pattern, so it can be readily found.
Then another X- header with a preliminary analysis of the Date string.
In the example
Date: Wed, 31-Dec-69 18:59:59 EDT
Leave the punctuation as is. Look for the alphanumeric pattern. X
capital letter, x lower case letter, N numeral, _ space
X-Date-Pattern: Xxx,_NN-Xxx-NN_NN:NN:NN_XXX
Some time later, perhaps a year later, someone might analyze this. If a three-alpha recognizeable as a day is in the first Xxx group (which
might be an XXX), then it's a day. It might have an optional "." and "," isn't always going to be present as a separator. Similarly, the second three-alpha might be a month.
The two-digit year could be confused with a time element but there
probably won't be dates prior to the Unix epoch.
The three-alpha time zone isn't necessarily unique worldwide, but prior
to a certain date we know that the articles were from the United States
only.
If the transformation into modern syntax results in a logical day-date combination, then that's passed one test that the transformation was
valid.
There's going to be a lot of eyeballing necessary but this could
possibly be a way to choose which articles have dates requiring further analysis.