msdos-encodings

Bidirectional, emulator-grade conversion between Microsoft code pages and Unicode — without linking ICU. The crate owns the canonical Microsoft tables, so it stays small, self-contained, and cross-compiles cleanly.

These are strictly Microsoft's mappings as published by the Unicode Consortium (VENDORS/MICSFT/) — not the IBM or Oracle variants, which differ in a handful of slots.

use msdos_encodings::{DosEncoding, Encoding, WindowsEncoding};

let enc = DosEncoding::default(); // code page 437
assert_eq!(enc.decode(b"Caf\x82").unwrap(), "Café");
assert_eq!(enc.encode("Café").unwrap(), b"Caf\x82");

// Pick by numeric code page (e.g. from CHCP or locale data):
let enc = DosEncoding::from_code_page(866).unwrap();
assert_eq!(enc.decode(&[0x80]).unwrap(), "А"); // DOS Cyrillic Russian

// The Windows ANSI family is a separate enum:
let enc = WindowsEncoding::Cp1252;
assert_eq!(enc.decode(&[0x93, 0x80, 0x94]).unwrap(), "\u{201C}€\u{201D}");

// Or resolve a raw code page number across both families:
assert_eq!(Encoding::from_code_page(1251).unwrap().name(), "Windows Cyrillic");

// Some pages leave bytes undefined; decode is fallible, or go lossy:
assert!(DosEncoding::Cp874.decode(&[0xFF]).is_err());
assert_eq!(DosEncoding::Cp874.decode_lossy(&[0xFF]), "\u{FFFD}");

Status

• All sixteen single-byte DOS (OEM) pages — implemented, decode + encode: 437, 737, 775, 850, 852, 855, 857, 860, 861, 862, 863, 864, 865, 866, 869, 874.
• All four double-byte CJK pages — implemented, decode + encode: 932 (Shift-JIS), 936 (GBK), 949 (Unified Hangul), 950 (Big5). They sit behind the on-by-default dbcs feature; build with default-features = false for a lean single-byte-only crate.
• All ten single-byte Windows ANSI pages — implemented, decode + encode: 874 (Thai) and 1250–1258, as WindowsEncoding.

Fidelity notes

• Every single-byte table is a verified bijection: anything a page decodes re-encodes to the identical bytes.
• Many pages leave byte values undefined (DOS 857/864/869/874; every ANSI page except 1256 has holes in 0x80..=0x9F), so decode returns a Result; decode_lossy substitutes U+FFFD.
• Microsoft's CP864 (DOS Arabic) maps 0x25 to ARABIC PERCENT SIGN — the low half is not pure ASCII, and % itself is unmappable. This is faithful to the source table.
• CP932 and CP950 each have a handful of byte pairs that decode to the same character (932's NEC/IBM extension overlap, 950's duplicated box-drawing rows). All of them decode; re-encoding picks the code Windows picks, as resolved from Microsoft's own bestfit932/950.txt WCTABLE (vendored in data/bestfit/, used only for duplicate resolution — no lossy best-fit mappings are imported).
• Double-byte decode adds two failure modes: a lead byte at end of input (TruncatedPair) and an undefined lead/trail pair (UndefinedPair). decode_lossy emits one U+FFFD per failed pair.
• The full decode direction of all four CJK pages agrees exactly with Python's cp932/cp936/cp949/cp950 codecs (59k+ pairs, verified during development).
• Microsoft publishes the identical Thai table for both families, so DosEncoding::Cp874 and WindowsEncoding::Cp874 agree byte-for-byte (pinned by a test).

Regenerating tables

Everything in src/tables/ except mod.rs is generated from the vendored mapping files in data/ (pc/ and windows/ mapping tables, bestfit/ for duplicate resolution; needs Python ≥ 3.10 and rustfmt on PATH):

uv run tools/gen_tables.py

Scope

Text-encoding conversion only. Filename conventions (8.3 space padding, case folding, the 0x05/0xE5 deleted-entry dance) are deliberately out of scope and left to consumers, as is any Vietnamese (CP1258) combining-mark normalization.

Consumers

Designed to be shared by fsinspect and friends via path (and later git) dependency.

License

MIT