Add the Windows ANSI family: 874 Thai and 1250-1258 as WindowsEncoding
The ten single-byte ANSI pages join as a second enum alongside DosEncoding, mirroring its API. Errors now carry a public Encoding wrapper (Dos | Windows) so both families share the single-byte codec functions; Encoding::from_code_page resolves a raw code page number across families, DOS first. data/ is reorganized into pc/, windows/, and bestfit/, and the generated single-byte tables split into dos.rs and ansi.rs. Microsoft publishes an identical Thai table for both families, so DosEncoding:: Cp874 and WindowsEncoding::Cp874 agree byte-for-byte (pinned by test).
This commit is contained in:
42
README.md
42
README.md
@@ -1,15 +1,15 @@
|
||||
# msdos-encodings
|
||||
|
||||
Bidirectional, emulator-grade conversion between MS-DOS code pages and Unicode —
|
||||
without linking ICU. The crate owns the canonical Microsoft tables, so it stays
|
||||
small, self-contained, and cross-compiles cleanly.
|
||||
Bidirectional, emulator-grade conversion between Microsoft code pages and
|
||||
Unicode — without linking ICU. The crate owns the canonical Microsoft tables,
|
||||
so it stays small, self-contained, and cross-compiles cleanly.
|
||||
|
||||
These are strictly **Microsoft's** mappings as published by the Unicode
|
||||
Consortium (`VENDORS/MICSFT/PC/`) — not the IBM or Oracle variants, which
|
||||
Consortium (`VENDORS/MICSFT/`) — not the IBM or Oracle variants, which
|
||||
differ in a handful of slots.
|
||||
|
||||
```rust
|
||||
use msdos_encodings::DosEncoding;
|
||||
use msdos_encodings::{DosEncoding, Encoding, WindowsEncoding};
|
||||
|
||||
let enc = DosEncoding::default(); // code page 437
|
||||
assert_eq!(enc.decode(b"Caf\x82").unwrap(), "Café");
|
||||
@@ -19,26 +19,36 @@ assert_eq!(enc.encode("Café").unwrap(), b"Caf\x82");
|
||||
let enc = DosEncoding::from_code_page(866).unwrap();
|
||||
assert_eq!(enc.decode(&[0x80]).unwrap(), "А"); // DOS Cyrillic Russian
|
||||
|
||||
// Four pages leave bytes undefined; decode is fallible, or go lossy:
|
||||
// The Windows ANSI family is a separate enum:
|
||||
let enc = WindowsEncoding::Cp1252;
|
||||
assert_eq!(enc.decode(&[0x93, 0x80, 0x94]).unwrap(), "\u{201C}€\u{201D}");
|
||||
|
||||
// Or resolve a raw code page number across both families:
|
||||
assert_eq!(Encoding::from_code_page(1251).unwrap().name(), "Windows Cyrillic");
|
||||
|
||||
// Some pages leave bytes undefined; decode is fallible, or go lossy:
|
||||
assert!(DosEncoding::Cp874.decode(&[0xFF]).is_err());
|
||||
assert_eq!(DosEncoding::Cp874.decode_lossy(&[0xFF]), "\u{FFFD}");
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
- **All sixteen single-byte DOS pages** — implemented, decode + encode:
|
||||
- **All sixteen single-byte DOS (OEM) pages** — implemented, decode + encode:
|
||||
437, 737, 775, 850, 852, 855, 857, 860, 861, 862, 863, 864, 865, 866, 869, 874.
|
||||
- **All four double-byte CJK pages** — implemented, decode + encode:
|
||||
932 (Shift-JIS), 936 (GBK), 949 (Unified Hangul), 950 (Big5). They sit
|
||||
behind the on-by-default `dbcs` feature; build with
|
||||
`default-features = false` for a lean single-byte-only crate.
|
||||
- **All ten single-byte Windows ANSI pages** — implemented, decode + encode:
|
||||
874 (Thai) and 1250–1258, as `WindowsEncoding`.
|
||||
|
||||
## Fidelity notes
|
||||
|
||||
- Every single-byte table is a verified bijection: anything a page decodes
|
||||
re-encodes to the identical bytes.
|
||||
- Code pages 857, 864, 869, and 874 leave some byte values undefined, so
|
||||
`decode` returns a `Result`; `decode_lossy` substitutes U+FFFD.
|
||||
- Many pages leave byte values undefined (DOS 857/864/869/874; every ANSI
|
||||
page except 1256 has holes in `0x80..=0x9F`), so `decode` returns a
|
||||
`Result`; `decode_lossy` substitutes U+FFFD.
|
||||
- Microsoft's CP864 (DOS Arabic) maps `0x25` to ARABIC PERCENT SIGN — the low
|
||||
half is *not* pure ASCII, and `%` itself is unmappable. This is faithful to
|
||||
the source table.
|
||||
@@ -46,20 +56,23 @@ assert_eq!(DosEncoding::Cp874.decode_lossy(&[0xFF]), "\u{FFFD}");
|
||||
character (932's NEC/IBM extension overlap, 950's duplicated box-drawing
|
||||
rows). All of them decode; re-encoding picks the code Windows picks, as
|
||||
resolved from Microsoft's own `bestfit932/950.txt` WCTABLE (vendored in
|
||||
`data/`, used *only* for duplicate resolution — no lossy best-fit mappings
|
||||
are imported).
|
||||
`data/bestfit/`, used *only* for duplicate resolution — no lossy best-fit
|
||||
mappings are imported).
|
||||
- Double-byte decode adds two failure modes: a lead byte at end of input
|
||||
(`TruncatedPair`) and an undefined lead/trail pair (`UndefinedPair`).
|
||||
`decode_lossy` emits one U+FFFD per failed pair.
|
||||
- The full decode direction of all four CJK pages agrees exactly with
|
||||
Python's `cp932`/`cp936`/`cp949`/`cp950` codecs (59k+ pairs, verified
|
||||
during development).
|
||||
- Microsoft publishes the identical Thai table for both families, so
|
||||
`DosEncoding::Cp874` and `WindowsEncoding::Cp874` agree byte-for-byte
|
||||
(pinned by a test).
|
||||
|
||||
## Regenerating tables
|
||||
|
||||
Everything in `src/tables/` except `mod.rs` is generated from the vendored
|
||||
mapping files in `data/`
|
||||
(needs Python ≥ 3.10 and `rustfmt` on `PATH`):
|
||||
mapping files in `data/` (`pc/` and `windows/` mapping tables, `bestfit/`
|
||||
for duplicate resolution; needs Python ≥ 3.10 and `rustfmt` on `PATH`):
|
||||
|
||||
```sh
|
||||
uv run tools/gen_tables.py
|
||||
@@ -69,7 +82,8 @@ uv run tools/gen_tables.py
|
||||
|
||||
Text-encoding conversion only. Filename conventions (8.3 space padding,
|
||||
case folding, the 0x05/0xE5 deleted-entry dance) are deliberately **out of
|
||||
scope** and left to consumers.
|
||||
scope** and left to consumers, as is any Vietnamese (CP1258) combining-mark
|
||||
normalization.
|
||||
|
||||
## Consumers
|
||||
|
||||
|
||||
Reference in New Issue
Block a user