Add the four double-byte CJK pages behind an on-by-default dbcs feature

Browse Source

Code pages 932 (Shift-JIS), 936 (GBK), 949 (Unified Hangul), and 950
(Big5), generated from the Unicode Consortium's Microsoft WINDOWS
tables. Each page carries a three-state single-byte table (map / lead /
undefined) plus sorted pair arrays binary-searched in both directions.

Where several codes decode to one scalar (932's NEC/IBM overlap, 950's
duplicated box-drawing rows), the encoder's winner is resolved from
Microsoft's own bestfit WCTABLE — used only for duplicate resolution,
never to import lossy best-fit mappings. The full decode direction was
cross-validated against Python's cp932/936/949/950 codecs: zero
mismatches over 60k+ pairs.

Decode gains TruncatedPair and UndefinedPair failure modes; decode_lossy
emits one U+FFFD per failed pair. Both feature configurations are
clippy-clean and tested.

...

This commit is contained in:

Claude Fable 5

2026-07-19 12:28:33 -05:00

parent 948135af72

commit 518250da2c

18 changed files with 327372 additions and 154 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

48948

data/bestfit936.txt Normal file

File diff suppressed because it is too large Load Diff