open https://gitlab.synchro.net/main/sbbs/-/work_items/1319

## Problem and motivation

Synchronet scripts need enough native Unicode support to implement editors and other terminal interfaces without maintaining Unicode algorithms and property tables in JavaScript. Distinguish language/runtime facilities from Synchronet's responsibilities so this work can proceed on the existing SpiderMonkey 1.8.5 embedding and remain useful if the engine is upgraded or replaced.

FSEditor provides a concrete example: its UTF-8 input decoder assembles a full code point, then calls `String.fromCharCode(val)`, truncating supplementary characters (e.g. U+1F600 becomes U+F600). Its output/save helpers encode individual `charCodeAt(0)` values, and editing assumes one UTF-16 code unit equals one character and one terminal column. Merely fixing the decoder or changing the engine does not address the remaining assumptions.

Extended grapheme clusters add another level: `👩🏽‍💻` is seven UTF-16 code units, four code points, and one extended grapheme cluster. Its terminal advance depends on the client's rendering and the selected terminal policy. Storage offsets, editing boundaries, and terminal columns must be explicit and separate.

Related reports: #1317 (internal editor Unicode input) and #719 (FSEditor UTF-8 paste). These are related context, not a claim that this work alone resolves either report.

## What an engine upgrade supplies, and what remains Synchronet's responsibility

| Capability | Modern SpiderMonkey / standard JavaScript | Synchronet work still required |
| --- | --- | --- |
| Supplementary code points | `String.fromCodePoint()`, `String.prototype.codePointAt()`, code-point string iteration | Update scripts and native string conversions to use/preserve complete characters; equivalent helpers if retaining 1.8.5 |
| Extended grapheme boundaries | `Intl.Segmenter` with `granularity: "grapheme"`, on a sufficiently recent build with Intl support | A stable host-facing capability/fallback for older engines and native consumers; editor integration and Unicode-version reporting |
| Word boundaries | `Intl.Segmenter` with `granularity: "word"` | Script integration and fallback if required on the current engine |
| Canonical normalization and character properties | `String.prototype.normalize()` and supported Unicode property escapes in regular expressions | Optional compatibility bindings where needed; avoid duplicating available engine functionality or silently normalizing stored user text |
| UTF-8 byte encoding/decoding | `TextEncoder` / `TextDecoder` are host APIs, not guaranteed by embedding a newer ECMAScript engine | Explicit, lossless byte ↔ Unicode conversion contracts and integration with files, sockets, message bodies, and console input/output |
| Terminal width, column clipping, and cursor advance | No standard ECMAScript terminal API | Native Synchronet APIs using the same policy as terminal output/cursor accounting, including cluster and presentation handling |
| Unicode console input and terminal/legacy charset translation | No engine knowledge of Synchronet console protocols or charsets | Native decoded input with explicit key/control/timeout/error behavior; retain legacy byte and CP437 interfaces |
| Unicode line-break opportunities | `Intl.Segmenter` provides grapheme/word/sentence segmentation, not a standard line-break mode | If general Unicode wrapping is included, expose native UAX #14 line-break opportunities and combine them with column measurement |

JavaScript strings retain UTF-16 code-unit indexing even after an engine upgrade. Existing `.length`, indexing, `charCodeAt()`, and slicing behavior will not change. Feature-detect the actual embedded runtime; availability in a browser does not establish availability in Synchronet's embedding.

## Proposed native capabilities

Names below describe the proposed contract, not a finalized public interface. Consider a dedicated Unicode namespace/object rather than extending the overloaded meanings of existing helpers.

### 1. Lossless conversion and code-point access

- Encode actual JS Unicode strings to UTF-8 bytes and decode UTF-8 bytes to actual JS Unicode strings, combining/splitting surrogate pairs correctly through U+10FFFF.
- Make byte representations explicit (including the legacy JS byte-string representation where needed). Specify byte lengths separately from UTF-16 offsets; define embedded-NUL handling.
- Define strict versus replacement behavior for malformed UTF-8 and unpaired UTF-16 surrogates. Reject invalid scalar values and overlong UTF-8 in strict mode. Do not silently truncate or convert through CP437.
- For the current engine, expose code-point construction/access/iteration helpers or thin compatibility wrappers. With a modern engine, prefer the standard methods instead of maintaining parallel algorithms.
- Preserve the existing meaning of `utf8_decode()`, which currently translates UTF-8 to CP437; introduce clearly distinct Unicode conversion entry points. Audit the existing UTF-16 path in `utf8_encode()` and native JS string marshalling for surrogate-pair correctness.

### 2. Grapheme segmentation

- Return extended grapheme clusters and their start/end offsets, or provide forward/backward boundary navigation. Define offsets as UTF-16 code-unit offsets for JS Unicode-string APIs; explicitly distinguish any UTF-8 byte-oriented native API.
- Support contextual resegmentation after insertion/deletion. A newly received combining mark, variation selector, modifier, or ZWJ sequence can change an existing cluster; one read operation or timeout does not establish a complete grapheme.
- Use a maintained native Unicode implementation, such as ICU's character break iterator, with UAX #29 conformance tests. Do not require scripts to ship their own Unicode tables or implement segmentation.
- Where `Intl.Segmenter` is available, scripts can use it through a small compatibility layer. Share the Unicode backend where practical and document any version differences between engine and Synchronet data.

### 3. Terminal measurement and boundary-safe clipping

- Measure strings/clusters in terminal columns using the active console's charset, zero-width/ambiguous-width settings, emoji presentation policy, and supported rendering behavior. Also make the core measurement usable without an active console through explicit options.
- Return enough information to map storage offsets to columns and fit/truncate a string to a column budget without splitting surrogate pairs or grapheme clusters. Define behavior when the first cluster cannot fit.
- Specify handling of tabs, control characters, Synchronet Ctrl-A attributes, and recognized terminal escapes. Keep formatting tokens out of text segmentation and never cut one in half.
- Account for variation selectors, modifiers, regional-indicator flag pairs, keycaps, and ZWJ sequences under the chosen client policy. Do not universally assume every cluster occupies one or two columns, or sum component widths as though each is rendered independently.
- Use the same implementation/policy in `console.strlen()`, clipping/wrapping, and native output cursor accounting. Document fallback behavior for unknown terminals; segmentation cannot guarantee that every client renders a sequence identically.

### 4. Unicode console input

- Expose a decoded Unicode text-input operation/event alongside existing key/byte APIs. Specify text versus special/control keys, timeout/disconnect results, invalid sequences, and handling of partial multibyte input across reads.
- Keep ANSI key parsing and existing control-key handling intact. Do not expose UTF-8 continuation bytes as separate editing commands.
- Input may return a code point or text chunk; scripts perform grapheme-aware editing on the surrounding buffer rather than waiting for a supposedly complete grapheme from the network.
- Keep CP437/other legacy-terminal translation explicit and compatible. `K_UTF8` currently suppresses CP437 translation; it does not itself decode a complete Unicode character into a JS string.

## Existing implementation gaps to address

Source baseline inspected: commit `087c17bd4aaae35575e518a07ae45684d3e6be43`.

- `exec/fseditor.js`: `shitty_inkey()`, `ch_to_string()`, `shitty_putc()`, and cursor/attribute indexing illustrate the script-level assumptions.
- `src/sbbs3/js_console.cpp`: `js_inkey()` returns a one-byte string; `js_strlen()` delegates to the terminal width calculation.
- `src/sbbs3/inkey.cpp`: `K_UTF8` bypasses UTF-8-to-CP437 conversion.
- `src/sbbs3/js_global.cpp`: UTF-8 helpers and the UTF-16 encoding path need clear contracts and supplementary-character coverage.
- `src/encode/utf8.c`: `utf8_str_total_width()` sums `unicode_width()` per decoded code point.
- `src/sbbs3/terminal.h`: `Terminal::bstrlen()` similarly sums code-point widths.
- `src/sbbs3/terminal.cpp`: native output cursor advance also uses individual code-point width.
- `src/xpdev/unicode.c`: existing width classifications are not an extended-grapheme segmentation implementation.

## Delivery and acceptance criteria

- [ ] Agree on the public API, representations, offset units, malformed-input behavior, terminal policy, and Unicode library/data/version strategy. Separate engine compatibility helpers from permanent Synchronet host functionality.
- [ ] Provide native conversion and segmentation on the current engine, or explicitly make the relevant part depend on an engine upgrade. An engine upgrade must not be a prerequisite for terminal measurement/input contracts.
- [ ] Verify standards-provided facilities on the selected modern SpiderMonkey build; avoid promising all of them from an unspecified upgrade.
- [ ] Round-trip ASCII, BMP characters, supplementary characters, embedded NUL where supported, and emoji sequences through the supported JS/native/file/console paths without truncation or accidental CP437 conversion. Test malformed input and surrogate errors under each documented policy.
- [ ] Test grapheme boundaries against the chosen Unicode version's GraphemeBreakTest data, including combining marks, CRLF, Hangul/Indic cases, flags, modifiers, keycaps, and ZWJ sequences.
- [ ] Verify measurement/clipping and native cursor accounting against the same explicit terminal policy. Cover zero-column clusters, presentation selectors, ambiguous widths, tabs/formatting, and insufficient column budgets.
- [ ] Exercise input split at every UTF-8 byte boundary, with timeout/disconnect/recovery behavior and special/control keys interleaved.
- [ ] Demonstrate FSEditor integration (in this work or a linked follow-up): lossless load/save; cursor movement, deletion, overwrite, and wrapping at grapheme boundaries; consistent attribute indexing and documented styling behavior when clusters merge.
- [ ] Preserve legacy scripts and byte/CP437 interfaces; document migration examples and feature/version detection.

This work item establishes reusable host capabilities. It does not require replacing SpiderMonkey, implementing Unicode shaping in the BBS server, or silently normalizing existing message content. Broader line-breaking, normalization/property compatibility, and migration of additional scripts can be separately scoped follow-ups.

## References

- [Intl.Segmenter](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/Segmenter) and [segment() output/offsets](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/Segmenter/segment)
- [JavaScript string encodings, code points, and grapheme clusters](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String#utf-16_characters_unicode_code_points_and_grapheme_clusters)
- [Unicode UAX #29: Text Segmentation](https://www.unicode.org/reports/tr29/)
- [Unicode UAX #14: Line Breaking](https://www.unicode.org/reports/tr14/)
- [ICU boundary analysis](https://unicode-org.github.io/icu/userguide/boundaryanalysis/)
- [Encoding API (host APIs)](https://encoding.spec.whatwg.org/)
--- SBBSecho 3.38-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)