Commit Graph

3 Commits

Author SHA1 Message Date
Claude Fable 5
2fe53fdee4 Add the Windows ANSI family: 874 Thai and 1250-1258 as WindowsEncoding
The ten single-byte ANSI pages join as a second enum alongside
DosEncoding, mirroring its API. Errors now carry a public Encoding
wrapper (Dos | Windows) so both families share the single-byte codec
functions; Encoding::from_code_page resolves a raw code page number
across families, DOS first.

data/ is reorganized into pc/, windows/, and bestfit/, and the
generated single-byte tables split into dos.rs and ansi.rs. Microsoft
publishes an identical Thai table for both families, so DosEncoding::
Cp874 and WindowsEncoding::Cp874 agree byte-for-byte (pinned by test).
v0.1.0
2026-07-19 12:40:36 -05:00
Claude Fable 5
518250da2c Add the four double-byte CJK pages behind an on-by-default dbcs feature
Code pages 932 (Shift-JIS), 936 (GBK), 949 (Unified Hangul), and 950
(Big5), generated from the Unicode Consortium's Microsoft WINDOWS
tables. Each page carries a three-state single-byte table (map / lead /
undefined) plus sorted pair arrays binary-searched in both directions.

Where several codes decode to one scalar (932's NEC/IBM overlap, 950's
duplicated box-drawing rows), the encoder's winner is resolved from
Microsoft's own bestfit WCTABLE — used only for duplicate resolution,
never to import lossy best-fit mappings. The full decode direction was
cross-validated against Python's cp932/936/949/950 codecs: zero
mismatches over 60k+ pairs.

Decode gains TruncatedPair and UndefinedPair failure modes; decode_lossy
emits one U+FFFD per failed pair. Both feature configurations are
clippy-clean and tested.
2026-07-19 12:28:33 -05:00
Claude Fable 5
948135af72 Initial crate: all 16 single-byte MS-DOS code pages, bidirectional
Tables are generated from the Unicode Consortium's Microsoft mappings
(VENDORS/MICSFT/PC), vendored in data/ and regenerated by
tools/gen_tables.py. Decode is fallible because 857/864/869/874 leave
byte values undefined; decode_lossy substitutes U+FFFD. Every table is
a verified bijection, so defined bytes round-trip exactly.
2026-07-19 12:09:54 -05:00