open https://gitlab.synchro.net/main/sbbs/-/issues/1225

## Symptom

A terminal-server node session over SSH was torn down mid-door with no user action, no idle timeout, and no crash of the external program. The client reconnected 11 seconds later on a fresh source port.

## Log

Node 11, SyncTERM client (CTerm 1.332) over SSH, running a graphics-heavy
door that streams sixel frames. Door name redacted as `<door>`.

```
22:33:43.29 <Digital Man> Executing external: <door> /sbbs/node11/DOOR32.SYS 22:33:45.52 <Digital Man> <door>: frames as sixel
22:33:46.18 SSH WARNING 'Invalid packet <Unknown type> (236), expected
SSH_MSG_CHANNEL_DATA (94)' (-32) popping data from input_thread 22:33:46.18 input thread terminated (received 123278 bytes in 13151 blocks) 22:33:46.18 output thread terminated (sent 179433865 bytes in 125287 blocks,
1432 average, 0 short)
22:33:46.18 passthru thread terminated
22:33:52.80 <Digital Man> external Timeout waiting for output buffer to empty 22:33:52.94 <Digital Man> logout initiated
```

Everything from 22:33:46.18 onward is fallout: the door kept rendering into a dead socket for 6.6 seconds, hit the output-buffer drain timeout, and then exited through postxtrn normally. No core file was written for it.

## What happens

cryptlib decrypted an inbound SSH packet header carrying message type 236
where channel data was expected, and returned CRYPT_ERROR_BADDATA (-32).

In `src/sbbs3/main.cpp`, `input_thread()` treats every `crypt_pop_channel_data()`
error other than CRYPT_ERROR_COMPLETE and CRYPT_ERROR_TIMEOUT as fatal: it
logs via GCES and breaks out of the loop, which ends the input thread and
takes the node down with it. So a single bad packet is unconditionally fatal.

That is defensible for SSH -- once the transport is out of frame there is nothing to resync to -- but it is worth confirming that is the intent rather than an accident of the error-handling structure, because the same path also catches errors that are not framing failures.

## Why the packet type is suspicious

In `3rdp/src/cl/session/ssh2_rd.c`, `readPacketSSH2()` peeks the pad length
and packet type out of the decrypted header and calls `checkPacketValid()` *before* the packet MAC is verified -- the "we've passed the crypto stage" comment sits after that check. So a bogus type here does not mean the peer authentically sent message 236; it is what a desynced or corrupted stream
looks like at the first decrypted block.

236 is in SSH's private-use message range (192-255). Nothing in the protocol sends it, and cryptlib does not implement SSH rekeying, so a mishandled
key re-exchange is not an explanation either.

The preceding padding-length and packet-length checks both passed, which constrains the corruption only weakly -- most random bytes pass those.

## Frequency

Exactly one occurrence of this message across three rotated log files
(roughly 15 GB, Aug 2 through Aug 10). The same client had pushed 517 MB through the same code path in the preceding session without incident. This
is not a recurring failure, and no reproduction is known.

## Notes on what it is not

- Not a crash of the external program: no core, normal startup logging, clean
exit through postxtrn afterwards.
- Not an inactivity timeout.
- Not a door writing to or reading from an inherited descriptor it should not
touch: the door in question obtains its socket through the same shared
DOOR32.SYS helper every other door here uses, and no other door has ever
produced this.
- TCP would have delivered the bytes in order, so the corruption is inside one
of the two cryptlib endpoints rather than on the wire.

## Suggested next steps

Filed to have a record rather than because a fix is obvious. To make progress if it recurs:

- raise SSHErrorLevel on the terminal server for more cryptlib detail
- capture the TCP stream for the affected client, which at least settles
whether the byte stream itself was intact
- consider logging the surrounding decrypted header bytes on BADDATA, so a
recurrence shows whether the stream is offset by a fixed amount (a framing
bug) or is garbage (a keystream or memory problem)

Possibly related, both closed: #302 (cryptlib internal consistency check failures affecting SSH/TLS).

-- *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1225#note_10180

## Six more occurrences, a second error code, and a correction

Two users (Tracker1 and DM) reported SSH sessions dropping over the past
week, which sent me back through the logs. This is not a one-off, and the
`-32` packet-type case filed above is the rarer of two shapes the same
defect takes.

### The other error code

The common variant returns **`CRYPT_ERROR_FAILED` (-16), "Internal
consistency check failed"** out of the same `crypt_pop_channel_data()`
call, and kills the node the same way:

```
17:51:09 Node 1 SSH note 'Internal consistency check failed' (-16) popping data from input_thread
17:51:12 Node 1 <Tracker1> logout initiated
17:51:13 Node 1 thread terminated (0 node threads remain, 6547 clients served) ```

Note the severity: cryptlib rates this **`note`**, not `ERROR`. The session dies regardless — see the mechanism section below.

### Occurrences

| When | Host | User | Error | Time into session | Activity at drop | |---|---|---|---|---|---|
| Jun 3 07:41:21 | vert | Arqueiro | -16 | ≈45 s | after logon event |
| Jun 3 07:43:37 | vert | Arqueiro | -16 | ≈90 s | after logon event |
| Aug 11 17:51:09 | vert | Tracker1 | -16 | 70 s | idle at menu |
| Aug 11 17:53:59 | vert | Tracker1 | -16 | 2 m 28 s | idle, after a node message |
| Aug 11 22:33:46 | git | Digital Man | -32 | mid-door | sixel door (this issue) |
| Aug 14 02:57:55 | vert | R3vlibre | -16 | 4 m 52 s | idle |
| Aug 14 04:32:57 | vert | R3vlibre | -16 | 1 h 33 m | idle, after exiting Utopia |

Arqueiro's June pair means this predates the past week; it is not a new regression, just newly noticed.

### Tracker1's Aug 11 sequence

The most informative run, because he kept reconnecting:

```
17:49:52 SSH connect, TERM xterm-256color, UTF-8 -> logon 17:49:59 -> DROPPED 17:51:09 (-16)
17:51:24 SSH connect, TERM xterm-256color, UTF-8 -> logon 17:51:31 -> DROPPED 17:53:59 (-16)
17:54:12 SSH connect -> session establishment failed 17:54:20
17:54:25 Telnet -> logon 17:54:33 -> clean logout 18:05:26
18:08:01 SSH connect, TERM ansi, CP437 -> logon 18:08:03 -> fine through 18:46+
```

So SSH is not simply broken for him: his fourth attempt, over SSH, ran
nearly 40 minutes without trouble. What changed between the sessions that
died and the one that lived was the terminal mode.

### An observation, offered with its sample size

Every one of the seven drops above was a **UTF-8** session. The two SSH sessions in the sample that were **CP437** (Tracker1's 18:08 `ansi`
session, and a `syncterm` session on Aug 13) both survived. Counting all
SSH sessions on vert that reached terminal-type detection over Aug 11/13/14:
6 UTF-8, of which 4 dropped; 2 CP437, of which 0 dropped.

Two of two is not evidence. It is, however, cheap to act on: an attempt to reproduce should use **UTF-8 with an xterm-family TERM over SSH**, since
that is where every observed failure lives. The failures are on the input
path, so if this holds it is presumably about what the client *sends* -- a UTF-8 terminal emits different and more input (cursor-position and device attribute reports, mouse encoding) than a CP437 one.

### This is not SyncTERM-specific

That was the initial suspicion and it does not survive the logs. Terminal
types by protocol on vert across Aug 11/13/14:

```
464 Telnet DUMB 5 SSH xterm-256color
35 Telnet syncterm 1 SSH xterm
20 Telnet ANSI 1 SSH syncterm
16 Telnet PETSCII 1 SSH ansi
8 Telnet RIP (+ smaller Telnet types)
```

Only **8 SSH sessions reached logon on vert in three days**, and 4 of them
were killed. The 35 `syncterm` sessions were all Telnet, so they were never exposed to this path and say nothing either way. Meanwhile the `-32` drop
in the description above *was* a SyncTERM client. Both sides of the client question are represented among the failures; the constant is SSH.

### Correction to the description above

The description reasons that "the corruption is inside one of the two
cryptlib endpoints." That premise needs revisiting: **SyncTERM's SSH client
is DeuceSSH** (`src/ssh/`, see the header comment in
`src/syncterm/ssh.c`), not cryptlib. So for the SyncTERM case the two
endpoints are DeuceSSH (client) and cryptlib (server), and "both endpoints
are cryptlib" no longer bounds where to look.

For what it is worth, DeuceSSH's auto-rekey is **not** an explanation: `rekey_seconds` defaults to 0 (disabled) in `dssh_session_init`, and the
byte threshold is the per-cipher `bytes_per_key`, which is 64 GiB for every
AES module. Neither could have fired on any session here. The description's
"no rekeying" reasoning holds, for a different reason than stated.

### A correlation I tested and rejected

Each drop happens while SSH scanners are failing session establishment on
other nodes, which looks suggestive. It is not: session-establishment
failures occur in **26-55% of all logged minutes** on vert (measured across
the three days above). With seven events, co-occurrence is what chance predicts. Not evidence, and worth recording so it is not re-derived as a
lead.

### Mechanism

`src/sbbs3/main.cpp:2319-2333`. Everything except `CRYPT_ERROR_COMPLETE`
and `CRYPT_ERROR_TIMEOUT` is fatal:

```c
/* Handle the SSH error here... */
GCES(err, sbbs->cfg.node_num, sbbs->ssh_session, "popping data");
break; /* input thread exits -> node torn down */
```

`GCES` calls `get_crypt_error_string()`, which returns a **severity** along with the message -- that is why these log as `note` and `dbg` rather than `ERROR`. The `break` then ignores that severity entirely, so a condition cryptlib itself classifies as informational takes a logged-on user's
session down. The Telnet path immediately below tolerates transient
conditions (`if (rd == 0 && !socket_recvdone(sock, 0)) continue;`) in a way
the SSH path does not.

This is the amplifier rather than the cause, but it is the part that turns
a single cryptlib hiccup into a lost session, and it is separable from
whatever makes cryptlib return -16 in the first place.

Also landing here, and near-certainly benign: `CRYPT_ERROR_PARAM1` (-1) at teardown, 354 times in the Aug 10-14 log on git and 31 in a week on vert, always
immediately after `disconnecting client` -- the session handle is already
gone. Those cost nothing today only because the session was ending anyway.

### Relationship to #302

\#302 ("Cryptlib 'internal consistency check' failures cause SSH and TLS connection issues") is the same -16 condition. It was closed on the
assessment that it "doesn't appear to be fatal." On this path it is fatal. Worth reopening or explicitly folding into this issue.

### Diagnostic gap

We do not log the peer's SSH version string anywhere, so for the vert occurrences I can identify the TERM each client sent but not which client
sent it. Logging the peer ident on the terminal server would settle the
client question the next time this happens, and raising `SSHErrorLevel` on
vert would be cheap -- the hit rate there is high enough to catch a
recurrence quickly.

-- *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)

https://gitlab.synchro.net/main/sbbs/-/issues/1225#note_10180

## Six more occurrences, a second error code, and a correction

Two users (Tracker1 and DM) reported SSH sessions dropping over the past
week, which sent me back through the logs. This is not a one-off, and the
`-32` packet-type case filed above is the rarer of two shapes the same
defect takes.

### The other error code

The common variant returns **`CRYPT_ERROR_INTERNAL` (-16), "Internal consistency check failed"** out of the same `crypt_pop_channel_data()`
call, and kills the node the same way:

```
17:51:09 Node 1 SSH note 'Internal consistency check failed' (-16) popping data from input_thread
17:51:12 Node 1 <Tracker1> logout initiated
17:51:13 Node 1 thread terminated (0 node threads remain, 6547 clients served) ```

Note the severity: cryptlib rates this **`note`**, not `ERROR`. The session dies regardless — see the mechanism section below.

### Occurrences

| When | Host | User | Error | Time into session | Activity at drop | |---|---|---|---|---|---|
| Jun 3 07:41:21 | vert | Arqueiro | -16 | ≈45 s | after logon event |
| Jun 3 07:43:37 | vert | Arqueiro | -16 | ≈90 s | after logon event |
| Aug 11 17:51:09 | vert | Tracker1 | -16 | 70 s | idle at menu |
| Aug 11 17:53:59 | vert | Tracker1 | -16 | 2 m 28 s | idle, after a node message |
| Aug 11 22:33:46 | git | Digital Man | -32 | mid-door | sixel door (this issue) |
| Aug 14 02:57:55 | vert | R3vlibre | -16 | 4 m 52 s | idle |
| Aug 14 04:32:57 | vert | R3vlibre | -16 | 1 h 33 m | idle, after exiting Utopia |

Arqueiro's June pair means this predates the past week; it is not a new regression, just newly noticed.

### Tracker1's Aug 11 sequence

The most informative run, because he kept reconnecting:

```
17:49:52 SSH connect, TERM xterm-256color, UTF-8 -> logon 17:49:59 -> DROPPED 17:51:09 (-16)
17:51:24 SSH connect, TERM xterm-256color, UTF-8 -> logon 17:51:31 -> DROPPED 17:53:59 (-16)
17:54:12 SSH connect -> session establishment failed 17:54:20
17:54:25 Telnet -> logon 17:54:33 -> clean logout 18:05:26
18:08:01 SSH connect, TERM ansi, CP437 -> logon 18:08:03 -> fine through 18:46+
```

So SSH is not simply broken for him: his fourth attempt, over SSH, ran
nearly 40 minutes without trouble. What changed between the sessions that
died and the one that lived was the terminal mode.

### An observation, offered with its sample size

Every one of the seven drops above was a **UTF-8** session. The two SSH sessions in the sample that were **CP437** (Tracker1's 18:08 `ansi`
session, and a `syncterm` session on Aug 13) both survived. Counting all
SSH sessions on vert that reached terminal-type detection over Aug 11/13/14:
6 UTF-8, of which 4 dropped; 2 CP437, of which 0 dropped.

Two of two is not evidence. It is, however, cheap to act on: an attempt to reproduce should use **UTF-8 with an xterm-family TERM over SSH**, since
that is where every observed failure lives. The failures are on the input
path, so if this holds it is presumably about what the client *sends* -- a UTF-8 terminal emits different and more input (cursor-position and device attribute reports, mouse encoding) than a CP437 one.

### This is not SyncTERM-specific

That was the initial suspicion and it does not survive the logs. Terminal
types by protocol on vert across Aug 11/13/14:

```
464 Telnet DUMB 5 SSH xterm-256color
35 Telnet syncterm 1 SSH xterm
20 Telnet ANSI 1 SSH syncterm
16 Telnet PETSCII 1 SSH ansi
8 Telnet RIP (+ smaller Telnet types)
```

Only **8 SSH sessions reached logon on vert in three days**, and 4 of them
were killed. The 35 `syncterm` sessions were all Telnet, so they were never exposed to this path and say nothing either way. Meanwhile the `-32` drop
in the description above *was* a SyncTERM client. Both sides of the client question are represented among the failures; the constant is SSH.

### Correction to the description above

The description reasons that "the corruption is inside one of the two
cryptlib endpoints." That premise needs revisiting: **SyncTERM's SSH client
is DeuceSSH** (`src/ssh/`, see the header comment in
`src/syncterm/ssh.c`), not cryptlib. So for the SyncTERM case the two
endpoints are DeuceSSH (client) and cryptlib (server), and "both endpoints
are cryptlib" no longer bounds where to look.

For what it is worth, DeuceSSH's auto-rekey is **not** an explanation: `rekey_seconds` defaults to 0 (disabled) in `dssh_session_init`, and the
byte threshold is the per-cipher `bytes_per_key`, which is 64 GiB for every
AES module. Neither could have fired on any session here. The description's
"no rekeying" reasoning holds, for a different reason than stated.

### A correlation I tested and rejected

Each drop happens while SSH scanners are failing session establishment on
other nodes, which looks suggestive. It is not: session-establishment
failures occur in **26-55% of all logged minutes** on vert (measured across
the three days above). With seven events, co-occurrence is what chance predicts. Not evidence, and worth recording so it is not re-derived as a
lead.

### Mechanism

`src/sbbs3/main.cpp:2319-2333`. Everything except `CRYPT_ERROR_COMPLETE`
and `CRYPT_ERROR_TIMEOUT` is fatal:

```c
/* Handle the SSH error here... */
GCES(err, sbbs->cfg.node_num, sbbs->ssh_session, "popping data");
break; /* input thread exits -> node torn down */
```

`GCES` calls `get_crypt_error_string()`, which returns a **severity** along with the message -- that is why these log as `note` and `dbg` rather than `ERROR`. The `break` then ignores that severity entirely, so a condition cryptlib itself classifies as informational takes a logged-on user's
session down. The Telnet path immediately below tolerates transient
conditions (`if (rd == 0 && !socket_recvdone(sock, 0)) continue;`) in a way
the SSH path does not.

This is the amplifier rather than the cause, but it is the part that turns
a single cryptlib hiccup into a lost session, and it is separable from
whatever makes cryptlib return -16 in the first place.

Also landing here, and near-certainly benign: `CRYPT_ERROR_PARAM1` (-1) at teardown, 354 times in the Aug 10-14 log on git and 31 in a week on vert, always
immediately after `disconnecting client` -- the session handle is already
gone. Those cost nothing today only because the session was ending anyway.

### Relationship to #302

\#302 ("Cryptlib 'internal consistency check' failures cause SSH and TLS connection issues") is the same -16 condition. It was closed on the
assessment that it "doesn't appear to be fatal." On this path it is fatal. Worth reopening or explicitly folding into this issue.

### Diagnostic gap

We do not log the peer's SSH version string anywhere, so for the vert occurrences I can identify the TERM each client sent but not which client
sent it. Logging the peer ident on the terminal server would settle the
client question the next time this happens, and raising `SSHErrorLevel` on
vert would be cheap -- the hit rate there is high enough to catch a
recurrence quickly.

-- *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)