Claude Fable 5 518250da2c Add the four double-byte CJK pages behind an on-by-default dbcs feature ...

Code pages 932 (Shift-JIS), 936 (GBK), 949 (Unified Hangul), and 950
(Big5), generated from the Unicode Consortium's Microsoft WINDOWS
tables. Each page carries a three-state single-byte table (map / lead /
undefined) plus sorted pair arrays binary-searched in both directions.

Where several codes decode to one scalar (932's NEC/IBM overlap, 950's
duplicated box-drawing rows), the encoder's winner is resolved from
Microsoft's own bestfit WCTABLE — used only for duplicate resolution,
never to import lossy best-fit mappings. The full decode direction was
cross-validated against Python's cp932/936/949/950 codecs: zero
mismatches over 60k+ pairs.

Decode gains TruncatedPair and UndefinedPair failure modes; decode_lossy
emits one U+FFFD per failed pair. Both feature configurations are
clippy-clean and tested.

2026-07-19 12:28:33 -05:00

353 KiB

Raw Blame History

View Raw