movq/msdos-encodings
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add the four double-byte CJK pages behind an on-by-default dbcs feature

Browse Source 
Code pages 932 (Shift-JIS), 936 (GBK), 949 (Unified Hangul), and 950
(Big5), generated from the Unicode Consortium's Microsoft WINDOWS
tables. Each page carries a three-state single-byte table (map / lead /
undefined) plus sorted pair arrays binary-searched in both directions.

Where several codes decode to one scalar (932's NEC/IBM overlap, 950's
duplicated box-drawing rows), the encoder's winner is resolved from
Microsoft's own bestfit WCTABLE — used only for duplicate resolution,
never to import lossy best-fit mappings. The full decode direction was
cross-validated against Python's cp932/936/949/950 codecs: zero
mismatches over 60k+ pairs.

Decode gains TruncatedPair and UndefinedPair failure modes; decode_lossy
emits one U+FFFD per failed pair. Both feature configurations are
clippy-clean and tested.
This commit is contained in:
Claude Fable 5
2026-07-19 12:28:33 -05:00
parent 948135af72
commit 518250da2c
18 changed files with 327372 additions and 154 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
Cargo.toml
View File

@@ -5,6 +5,12 @@ edition = "2021"
description = "Bidirectional, emulator-grade conversion between MS-DOS code pages and Unicode"
license = "MIT"
[features]
default = ["dbcs"]
# The double-byte CJK pages (932/936/949/950). Their tables are large;
# disable default features for a lean single-byte-only build.
dbcs = []
[dependencies]
thiserror = "2"

25
README.md
View File

@@ -28,22 +28,37 @@ assert_eq!(DosEncoding::Cp874.decode_lossy(&[0xFF]), "\u{FFFD}");
- **All sixteen single-byte DOS pages** — implemented, decode + encode:
437, 737, 775, 850, 852, 855, 857, 860, 861, 862, 863, 864, 865, 866, 869, 874.
- **Double-byte CJK pages** (932 Shift-JIS, 936 GBK, 949 Korean, 950 Big5) —
planned, to be codegen'd the same way and feature-gated.
- **All four double-byte CJK pages** — implemented, decode + encode:
932 (Shift-JIS), 936 (GBK), 949 (Unified Hangul), 950 (Big5). They sit
behind the on-by-default `dbcs` feature; build with
`default-features = false` for a lean single-byte-only crate.
## Fidelity notes
- Every table is a verified bijection: anything a page decodes re-encodes to
the identical bytes.
- Every single-byte table is a verified bijection: anything a page decodes
re-encodes to the identical bytes.
- Code pages 857, 864, 869, and 874 leave some byte values undefined, so
`decode` returns a `Result`; `decode_lossy` substitutes U+FFFD.
- Microsoft's CP864 (DOS Arabic) maps `0x25` to ARABIC PERCENT SIGN — the low
half is *not* pure ASCII, and `%` itself is unmappable. This is faithful to
the source table.
- CP932 and CP950 each have a handful of byte pairs that decode to the same
character (932's NEC/IBM extension overlap, 950's duplicated box-drawing
rows). All of them decode; re-encoding picks the code Windows picks, as
resolved from Microsoft's own `bestfit932/950.txt` WCTABLE (vendored in
`data/`, used *only* for duplicate resolution — no lossy best-fit mappings
are imported).
- Double-byte decode adds two failure modes: a lead byte at end of input
(`TruncatedPair`) and an undefined lead/trail pair (`UndefinedPair`).
`decode_lossy` emits one U+FFFD per failed pair.
- The full decode direction of all four CJK pages agrees exactly with
Python's `cp932`/`cp936`/`cp949`/`cp950` codecs (59k+ pairs, verified
during development).
## Regenerating tables
`src/tables.rs` is generated from the vendored mapping files in `data/`
Everything in `src/tables/` except `mod.rs` is generated from the vendored
mapping files in `data/`
(needs Python ≥ 3.10 and `rustfmt` on `PATH`):
```sh

7998
data/CP932.TXT Normal file
View File

File diff suppressed because it is too large Load Diff

22065
data/CP936.TXT Normal file
View File

File diff suppressed because it is too large Load Diff

17322
data/CP949.TXT Normal file
View File

File diff suppressed because it is too large Load Diff

13777
data/CP950.TXT Normal file
View File

File diff suppressed because it is too large Load Diff

19493
data/bestfit932.txt Normal file
View File

File diff suppressed because it is too large Load Diff

48948
data/bestfit936.txt Normal file
View File

File diff suppressed because it is too large Load Diff

35519
data/bestfit949.txt Normal file
View File

File diff suppressed because it is too large Load Diff

40567
data/bestfit950.txt Normal file
View File

File diff suppressed because it is too large Load Diff

552
src/lib.rs
View File

@@ -1,20 +1,24 @@
//! Bidirectional conversion between MS-DOS code pages and Unicode.
//!
//! MS-DOS and the FAT family of filesystems stored text in one of a family of
//! single-byte OEM code pages selected by country/locale. This crate converts
//! those bytes to and from Unicode with emulator-grade fidelity: it owns the
//! canonical tables rather than linking ICU, so it stays small, self-contained,
//! and cross-compiles cleanly.
//! OEM code pages selected by country/locale. This crate converts those bytes
//! to and from Unicode with emulator-grade fidelity: it owns the canonical
//! tables rather than linking ICU, so it stays small, self-contained, and
//! cross-compiles cleanly.
//!
//! The tables are **Microsoft's** mappings as published by the Unicode
//! Consortium (`VENDORS/MICSFT/PC/`) — not the IBM or Oracle variants, which
//! differ in a handful of slots. All sixteen single-byte DOS pages are
//! implemented; the double-byte CJK pages (932/936/949/950) are future work.
//! Consortium (`VENDORS/MICSFT/`) — not the IBM or Oracle variants, which
//! differ in a handful of slots. All sixteen single-byte DOS pages are always
//! available; the double-byte CJK pages (932 Shift-JIS, 936 GBK, 949 Korean,
//! 950 Big5) sit behind the on-by-default `dbcs` feature because their tables
//! are large.
//!
//! Decoding is fallible because four pages (857, 864, 869, 874) leave some
//! byte values undefined; [`DosEncoding::decode_lossy`] substitutes U+FFFD
//! instead. Every implemented table is a bijection, so any string a page can
//! decode re-encodes to the identical bytes.
//! Decoding is fallible: several pages leave byte values undefined, and the
//! double-byte pages add truncated/undefined pair failures.
//! [`DosEncoding::decode_lossy`] substitutes U+FFFD instead. Anything a page
//! decodes re-encodes to the same bytes, except the handful of CP932/CP950
//! codes that share a scalar — those re-encode to Microsoft's preferred code,
//! exactly as Windows does.
//!
//! ```
//! use msdos_encodings::DosEncoding;
@@ -30,11 +34,13 @@ use std::fmt;
mod tables;
#[cfg(feature = "dbcs")]
use tables::{DbcsPage, SingleEntry};
/// An MS-DOS (OEM) code page.
///
/// Non-exhaustive: more code pages (the double-byte CJK family) will be added
/// without it being a breaking change, so external matches must include a
/// wildcard arm.
/// Non-exhaustive: more code pages may be added without it being a breaking
/// change, so external matches must include a wildcard arm.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)]
#[non_exhaustive]
pub enum DosEncoding {
@@ -71,11 +77,32 @@ pub enum DosEncoding {
Cp869,
/// Code page 874 — DOS Thai.
Cp874,
/// Code page 932 — Japanese Shift-JIS (double-byte).
#[cfg(feature = "dbcs")]
Cp932,
/// Code page 936 — Simplified Chinese GBK (double-byte).
#[cfg(feature = "dbcs")]
Cp936,
/// Code page 949 — Korean Unified Hangul (double-byte).
#[cfg(feature = "dbcs")]
Cp949,
/// Code page 950 — Traditional Chinese Big5 (double-byte).
#[cfg(feature = "dbcs")]
Cp950,
}
/// How one code page turns bytes into characters.
enum Codec {
/// Every byte stands alone, looked up in a 256-entry table.
Single(&'static [Option<char>; 256]),
/// Lead bytes pull in a trail byte; pairs are binary-searched.
#[cfg(feature = "dbcs")]
Dbcs(&'static DbcsPage),
}
impl DosEncoding {
/// Every code page this crate implements, in numeric order.
pub const ALL: [DosEncoding; 16] = [
pub const ALL: &'static [DosEncoding] = &[
Self::Cp437,
Self::Cp737,
Self::Cp775,
@@ -92,6 +119,14 @@ impl DosEncoding {
Self::Cp866,
Self::Cp869,
Self::Cp874,
#[cfg(feature = "dbcs")]
Self::Cp932,
#[cfg(feature = "dbcs")]
Self::Cp936,
#[cfg(feature = "dbcs")]
Self::Cp949,
#[cfg(feature = "dbcs")]
Self::Cp950,
];
/// Select an encoding from its numeric code page identifier (as found in
@@ -100,7 +135,8 @@ impl DosEncoding {
#[must_use]
pub fn from_code_page(code_page: u16) -> Option<Self> {
Self::ALL
.into_iter()
.iter()
.copied()
.find(|enc| enc.code_page() == code_page)
}
@@ -124,6 +160,14 @@ impl DosEncoding {
Self::Cp866 => 866,
Self::Cp869 => 869,
Self::Cp874 => 874,
#[cfg(feature = "dbcs")]
Self::Cp932 => 932,
#[cfg(feature = "dbcs")]
Self::Cp936 => 936,
#[cfg(feature = "dbcs")]
Self::Cp949 => 949,
#[cfg(feature = "dbcs")]
Self::Cp950 => 950,
}
}
@@ -147,6 +191,14 @@ impl DosEncoding {
Self::Cp866 => "DOS Cyrillic Russian",
Self::Cp869 => "DOS Greek 2",
Self::Cp874 => "DOS Thai",
#[cfg(feature = "dbcs")]
Self::Cp932 => "Japanese Shift-JIS",
#[cfg(feature = "dbcs")]
Self::Cp936 => "Simplified Chinese GBK",
#[cfg(feature = "dbcs")]
Self::Cp949 => "Korean Unified Hangul",
#[cfg(feature = "dbcs")]
Self::Cp950 => "Traditional Chinese Big5",
}
}
@@ -156,38 +208,48 @@ impl DosEncoding {
///
/// # Errors
///
/// Returns [`DecodeError::Undefined`] for the first byte value this code
/// page leaves unmapped (possible only under 857, 864, 869, and 874).
/// Returns [`DecodeError::Undefined`] for a byte value the code page
/// leaves unmapped; the double-byte pages can also return
/// [`DecodeError::TruncatedPair`] and [`DecodeError::UndefinedPair`].
pub fn decode(self, bytes: &[u8]) -> Result<String, DecodeError> {
let table = self.table();
bytes
.iter()
.map(|&byte| {
table[usize::from(byte)].ok_or(DecodeError::Undefined {
byte,
encoding: self,
match self.codec() {
Codec::Single(table) => bytes
.iter()
.map(|&byte| {
table[usize::from(byte)].ok_or(DecodeError::Undefined {
byte,
encoding: self,
})
})
})
.collect()
.collect(),
#[cfg(feature = "dbcs")]
Codec::Dbcs(page) => self.decode_dbcs(page, bytes),
}
}
/// Decode like [`decode`](Self::decode), substituting U+FFFD REPLACEMENT
/// CHARACTER for any byte the code page leaves undefined.
/// CHARACTER for anything undecodable: an undefined byte, a lead byte
/// with no trail, or an undefined pair (one U+FFFD per failed pair, both
/// bytes consumed).
#[must_use]
pub fn decode_lossy(self, bytes: &[u8]) -> String {
let table = self.table();
bytes
.iter()
.map(|&byte| table[usize::from(byte)].unwrap_or('\u{FFFD}'))
.collect()
match self.codec() {
Codec::Single(table) => bytes
.iter()
.map(|&byte| table[usize::from(byte)].unwrap_or(REPLACEMENT))
.collect(),
#[cfg(feature = "dbcs")]
Codec::Dbcs(page) => Self::decode_dbcs_lossy(page, bytes),
}
}
/// Decode up to (but not including) the first `NUL` byte — the convention
/// for fixed-width name fields in DOS-era structures.
/// for fixed-width name fields in DOS-era structures. Safe for the
/// double-byte pages too: `0x00` never occurs as a trail byte.
///
/// # Errors
///
/// Returns [`DecodeError::Undefined`] as [`decode`](Self::decode) does.
/// Returns [`DecodeError`] as [`decode`](Self::decode) does.
pub fn decode_cstr(self, bytes: &[u8]) -> Result<String, DecodeError> {
let end = bytes.iter().position(|&b| b == 0).unwrap_or(bytes.len());
self.decode(&bytes[..end])
@@ -201,39 +263,152 @@ impl DosEncoding {
/// representation in this code page (e.g. `é` under DOS Greek, or `%`
/// under DOS Arabic, whose `0x25` is ARABIC PERCENT SIGN).
pub fn encode(self, text: &str) -> Result<Vec<u8>, EncodeError> {
let table = self.table();
text.chars()
.map(|ch| {
table
.iter()
.position(|&slot| slot == Some(ch))
.map(|i| i as u8)
.ok_or(EncodeError::Unmappable { ch, encoding: self })
})
.collect()
}
/// The full byte-to-scalar table for this code page.
fn table(self) -> &'static [Option<char>; 256] {
match self {
Self::Cp437 => &tables::CP437,
Self::Cp737 => &tables::CP737,
Self::Cp775 => &tables::CP775,
Self::Cp850 => &tables::CP850,
Self::Cp852 => &tables::CP852,
Self::Cp855 => &tables::CP855,
Self::Cp857 => &tables::CP857,
Self::Cp860 => &tables::CP860,
Self::Cp861 => &tables::CP861,
Self::Cp862 => &tables::CP862,
Self::Cp863 => &tables::CP863,
Self::Cp864 => &tables::CP864,
Self::Cp865 => &tables::CP865,
Self::Cp866 => &tables::CP866,
Self::Cp869 => &tables::CP869,
Self::Cp874 => &tables::CP874,
match self.codec() {
Codec::Single(table) => text
.chars()
.map(|ch| {
table
.iter()
.position(|&slot| slot == Some(ch))
.map(|i| i as u8)
.ok_or(EncodeError::Unmappable { ch, encoding: self })
})
.collect(),
#[cfg(feature = "dbcs")]
Codec::Dbcs(page) => self.encode_dbcs(page, text),
}
}
/// Decode a double-byte page strictly.
#[cfg(feature = "dbcs")]
fn decode_dbcs(self, page: &DbcsPage, bytes: &[u8]) -> Result<String, DecodeError> {
let mut out = String::with_capacity(bytes.len());
let mut rest = bytes;
while let [byte, tail @ ..] = rest {
match page.single[usize::from(*byte)] {
SingleEntry::Map(ch) => {
out.push(ch);
rest = tail;
}
SingleEntry::Undefined => {
return Err(DecodeError::Undefined {
byte: *byte,
encoding: self,
})
}
SingleEntry::Lead => {
let [trail, tail @ ..] = tail else {
return Err(DecodeError::TruncatedPair {
lead: *byte,
encoding: self,
});
};
let ch =
lookup_pair(page, *byte, *trail).ok_or(DecodeError::UndefinedPair {
lead: *byte,
trail: *trail,
encoding: self,
})?;
out.push(ch);
rest = tail;
}
}
}
Ok(out)
}
/// Decode a double-byte page, substituting U+FFFD for failures.
#[cfg(feature = "dbcs")]
fn decode_dbcs_lossy(page: &DbcsPage, bytes: &[u8]) -> String {
let mut out = String::with_capacity(bytes.len());
let mut rest = bytes;
while let [byte, tail @ ..] = rest {
match page.single[usize::from(*byte)] {
SingleEntry::Map(ch) => {
out.push(ch);
rest = tail;
}
SingleEntry::Undefined => {
out.push(REPLACEMENT);
rest = tail;
}
SingleEntry::Lead => {
let [trail, tail @ ..] = tail else {
out.push(REPLACEMENT);
break;
};
out.push(lookup_pair(page, *byte, *trail).unwrap_or(REPLACEMENT));
rest = tail;
}
}
}
out
}
/// Encode to a double-byte page: single-byte table first, then the pair
/// table by binary search.
#[cfg(feature = "dbcs")]
fn encode_dbcs(self, page: &DbcsPage, text: &str) -> Result<Vec<u8>, EncodeError> {
let mut out = Vec::with_capacity(text.len());
for ch in text.chars() {
let single = page
.single
.iter()
.position(|&slot| slot == SingleEntry::Map(ch));
if let Some(byte) = single {
out.push(byte as u8);
} else if let Ok(i) = page.encode.binary_search_by_key(&ch, |&(u, _)| u) {
let (_, code) = page.encode[i];
out.extend_from_slice(&code.to_be_bytes());
} else {
return Err(EncodeError::Unmappable { ch, encoding: self });
}
}
Ok(out)
}
/// The lookup machinery for this code page.
fn codec(self) -> Codec {
match self {
Self::Cp437 => Codec::Single(&tables::single::CP437),
Self::Cp737 => Codec::Single(&tables::single::CP737),
Self::Cp775 => Codec::Single(&tables::single::CP775),
Self::Cp850 => Codec::Single(&tables::single::CP850),
Self::Cp852 => Codec::Single(&tables::single::CP852),
Self::Cp855 => Codec::Single(&tables::single::CP855),
Self::Cp857 => Codec::Single(&tables::single::CP857),
Self::Cp860 => Codec::Single(&tables::single::CP860),
Self::Cp861 => Codec::Single(&tables::single::CP861),
Self::Cp862 => Codec::Single(&tables::single::CP862),
Self::Cp863 => Codec::Single(&tables::single::CP863),
Self::Cp864 => Codec::Single(&tables::single::CP864),
Self::Cp865 => Codec::Single(&tables::single::CP865),
Self::Cp866 => Codec::Single(&tables::single::CP866),
Self::Cp869 => Codec::Single(&tables::single::CP869),
Self::Cp874 => Codec::Single(&tables::single::CP874),
#[cfg(feature = "dbcs")]
Self::Cp932 => Codec::Dbcs(&tables::cp932::CP932),
#[cfg(feature = "dbcs")]
Self::Cp936 => Codec::Dbcs(&tables::cp936::CP936),
#[cfg(feature = "dbcs")]
Self::Cp949 => Codec::Dbcs(&tables::cp949::CP949),
#[cfg(feature = "dbcs")]
Self::Cp950 => Codec::Dbcs(&tables::cp950::CP950),
}
}
}
/// U+FFFD REPLACEMENT CHARACTER, used by the lossy decoders.
const REPLACEMENT: char = '\u{FFFD}';
/// Look up a lead/trail pair in a DBCS page's sorted decode table.
#[cfg(feature = "dbcs")]
fn lookup_pair(page: &DbcsPage, lead: u8, trail: u8) -> Option<char> {
let code = u16::from_be_bytes([lead, trail]);
page.decode
.binary_search_by_key(&code, |&(c, _)| c)
.ok()
.map(|i| page.decode[i].1)
}
impl fmt::Display for DosEncoding {
@@ -242,10 +417,9 @@ impl fmt::Display for DosEncoding {
}
}
/// A byte could not be decoded from the source code page.
/// A byte sequence could not be decoded from the source code page.
///
/// Non-exhaustive: the future double-byte pages will add failure modes
/// (truncated pair, invalid lead byte) without a breaking change.
/// Non-exhaustive so future variants are not a breaking change.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[non_exhaustive]
pub enum DecodeError {
@@ -257,6 +431,24 @@ pub enum DecodeError {
/// The code page that leaves it undefined.
encoding: DosEncoding,
},
/// A lead byte of a double-byte page appeared with no trail byte after it.
#[error("lead byte {lead:#04x} at end of input in {encoding}")]
TruncatedPair {
/// The dangling lead byte.
lead: u8,
/// The double-byte code page being decoded.
encoding: DosEncoding,
},
/// A lead/trail pair is not a defined code of the page.
#[error("byte pair {lead:#04x} {trail:#04x} is undefined in {encoding}")]
UndefinedPair {
/// The lead byte of the pair.
lead: u8,
/// The trail byte of the pair.
trail: u8,
/// The double-byte code page being decoded.
encoding: DosEncoding,
},
}
/// A character could not be encoded into the target code page.
@@ -279,6 +471,15 @@ pub enum EncodeError {
mod tests {
use super::*;
/// The 256-entry table of a single-byte page, `None` for double-byte pages.
fn single_table(enc: DosEncoding) -> Option<&'static [Option<char>; 256]> {
match enc.codec() {
Codec::Single(table) => Some(table),
#[cfg(feature = "dbcs")]
Codec::Dbcs(_) => None,
}
}
#[test]
fn ascii_passthrough_cp437() {
let enc = DosEncoding::Cp437;
@@ -326,8 +527,8 @@ mod tests {
}
#[test]
fn every_defined_byte_round_trips_on_every_page() {
for enc in DosEncoding::ALL {
fn every_page_round_trips_every_single_byte() {
for &enc in DosEncoding::ALL {
for byte in 0..=0xFFu8 {
match enc.decode(&[byte]) {
Ok(decoded) => assert_eq!(
@@ -335,20 +536,22 @@ mod tests {
vec![byte],
"byte {byte:#04x} failed to round-trip under {enc}"
),
Err(DecodeError::Undefined { .. }) => {
assert_eq!(enc.decode_lossy(&[byte]), "\u{FFFD}");
}
Err(DecodeError::Undefined { .. } | DecodeError::TruncatedPair { .. }) => {}
Err(other) => panic!("unexpected error for {byte:#04x}: {other}"),
}
}
}
}
#[test]
fn every_table_is_a_bijection() {
fn single_byte_tables_are_bijections() {
// No duplicate scalars, else encode would be ambiguous.
for enc in DosEncoding::ALL {
for &enc in DosEncoding::ALL {
let Some(table) = single_table(enc) else {
continue;
};
let mut seen = std::collections::HashSet::new();
for slot in enc.table().iter().flatten() {
for slot in table.iter().flatten() {
assert!(seen.insert(*slot), "duplicate scalar {slot:?} in {enc}");
}
}
@@ -362,8 +565,11 @@ mod tests {
DosEncoding::Cp869,
DosEncoding::Cp874,
];
for enc in DosEncoding::ALL {
let undefined = enc.table().iter().filter(|slot| slot.is_none()).count();
for &enc in DosEncoding::ALL {
let Some(table) = single_table(enc) else {
continue;
};
let undefined = table.iter().filter(|slot| slot.is_none()).count();
if partial.contains(&enc) {
assert!(undefined > 0, "{enc} expected undefined slots");
} else {
@@ -399,11 +605,10 @@ mod tests {
#[test]
fn from_code_page_round_trips() {
for enc in DosEncoding::ALL {
for &enc in DosEncoding::ALL {
assert_eq!(DosEncoding::from_code_page(enc.code_page()), Some(enc));
}
assert_eq!(DosEncoding::from_code_page(1252), None); // Windows, not DOS
assert_eq!(DosEncoding::from_code_page(932), None); // CJK, not yet
}
#[test]
@@ -423,3 +628,188 @@ mod tests {
assert_eq!(DosEncoding::Cp437.decode(&bytes).unwrap(), art);
}
}
#[cfg(all(test, not(feature = "dbcs")))]
mod no_dbcs_tests {
use super::*;
#[test]
fn double_byte_pages_are_absent_without_the_feature() {
for code_page in [932, 936, 949, 950] {
assert_eq!(DosEncoding::from_code_page(code_page), None);
}
assert_eq!(DosEncoding::ALL.len(), 16);
}
}
#[cfg(all(test, feature = "dbcs"))]
mod dbcs_tests {
use super::*;
#[test]
fn golden_cjk_strings() {
// Golden values straight from the Microsoft tables.
assert_eq!(
DosEncoding::Cp932.encode("日本語").unwrap(),
b"\x93\xFA\x96\x7B\x8C\xEA"
);
assert_eq!(
DosEncoding::Cp936.encode("中文").unwrap(),
b"\xD6\xD0\xCE\xC4"
);
assert_eq!(
DosEncoding::Cp949.encode("한국").unwrap(),
b"\xC7\xD1\xB1\xB9"
);
assert_eq!(
DosEncoding::Cp950.encode("中文").unwrap(),
b"\xA4\xA4\xA4\xE5"
);
for (enc, bytes, text) in [
(
DosEncoding::Cp932,
&b"\x93\xFA\x96\x7B\x8C\xEA"[..],
"日本語",
),
(DosEncoding::Cp936, &b"\xD6\xD0\xCE\xC4"[..], "中文"),
(DosEncoding::Cp949, &b"\xC7\xD1\xB1\xB9"[..], "한국"),
(DosEncoding::Cp950, &b"\xA4\xA4\xA4\xE5"[..], "中文"),
] {
assert_eq!(enc.decode(bytes).unwrap(), text);
}
}
#[test]
fn cp932_halfwidth_katakana_is_single_byte() {
// 0xA1-0xDF are single-byte halfwidth katakana in Shift-JIS.
assert_eq!(DosEncoding::Cp932.decode(&[0xB1]).unwrap(), "\u{FF71}");
assert_eq!(DosEncoding::Cp932.encode("\u{FF71}").unwrap(), vec![0xB1]);
}
#[test]
fn ascii_mixes_with_pairs() {
let enc = DosEncoding::Cp932;
let bytes = b"C:\\\x93\xFA\x96\x7B\x8C\xEA\\README.TXT";
assert_eq!(enc.decode(bytes).unwrap(), "C:\\日本語\\README.TXT");
}
#[test]
fn duplicate_scalars_encode_as_windows_does() {
// CP932: NEC selection (0xED/0xEE leads) loses to everything else.
// Verified against Microsoft's bestfit932.txt WCTABLE.
let enc = DosEncoding::Cp932;
assert_eq!(enc.encode("\u{2170}").unwrap(), vec![0xFA, 0x40]); // not 0xEEEF
assert_eq!(enc.encode("\u{2160}").unwrap(), vec![0x87, 0x54]); // not 0xFA4A
assert_eq!(enc.encode("\u{2252}").unwrap(), vec![0x81, 0xE0]); // not 0x8790
// Both duplicate codes still decode.
assert_eq!(enc.decode(&[0xEE, 0xEF]).unwrap(), "\u{2170}");
assert_eq!(enc.decode(&[0xFA, 0x40]).unwrap(), "\u{2170}");
// CP950: the six known dups prefer the higher code, the box-drawing
// corners prefer the lower. Verified against bestfit950.txt.
let enc = DosEncoding::Cp950;
assert_eq!(enc.encode("\u{2550}").unwrap(), vec![0xF9, 0xF9]); // not 0xA2A4
assert_eq!(enc.encode("\u{256D}").unwrap(), vec![0xA2, 0x7E]); // not 0xF9FA
assert_eq!(enc.encode("\u{5341}").unwrap(), vec![0xA4, 0x51]); // not 0xA2CC
assert_eq!(enc.decode(&[0xA2, 0xA4]).unwrap(), "\u{2550}");
assert_eq!(enc.decode(&[0xF9, 0xF9]).unwrap(), "\u{2550}");
}
#[test]
fn truncated_pair_errors_and_lossy_substitutes() {
let enc = DosEncoding::Cp932;
assert_eq!(
enc.decode(b"abc\x93"),
Err(DecodeError::TruncatedPair {
lead: 0x93,
encoding: enc
})
);
assert_eq!(enc.decode_lossy(b"abc\x93"), "abc\u{FFFD}");
}
#[test]
fn undefined_pair_errors_and_lossy_consumes_both_bytes() {
let enc = DosEncoding::Cp932;
// 0x81 is a lead byte but 0x81 0x00 is not a defined pair.
assert_eq!(
enc.decode(b"\x81\x00"),
Err(DecodeError::UndefinedPair {
lead: 0x81,
trail: 0x00,
encoding: enc
})
);
assert_eq!(enc.decode_lossy(b"\x81\x00A"), "\u{FFFD}A");
}
#[test]
fn undefined_single_bytes_in_dbcs_pages() {
// 0x80 is undefined in CP932 (not a lead, not mapped).
let enc = DosEncoding::Cp932;
assert_eq!(
enc.decode(&[0x80]),
Err(DecodeError::Undefined {
byte: 0x80,
encoding: enc
})
);
assert_eq!(enc.decode_lossy(&[0x80]), "\u{FFFD}");
}
#[test]
fn every_defined_pair_decodes_and_round_trips_to_preferred_code() {
for enc in [
DosEncoding::Cp932,
DosEncoding::Cp936,
DosEncoding::Cp949,
DosEncoding::Cp950,
] {
let Codec::Dbcs(page) = enc.codec() else {
panic!("{enc} should be double-byte");
};
for &(code, ch) in page.decode {
let decoded = enc.decode(&code.to_be_bytes()).unwrap();
assert_eq!(decoded, ch.to_string(), "{enc} {code:#06x}");
// Re-encoding yields the Windows-preferred code for this char.
let reencoded = enc.encode(&decoded).unwrap();
let preferred = page
.encode
.binary_search_by_key(&ch, |&(u, _)| u)
.map(|i| page.encode[i].1)
.expect("every decodable char must be encodable");
assert_eq!(reencoded, preferred.to_be_bytes().to_vec());
}
}
}
#[test]
fn decode_and_encode_tables_are_sorted_and_unique() {
for enc in [
DosEncoding::Cp932,
DosEncoding::Cp936,
DosEncoding::Cp949,
DosEncoding::Cp950,
] {
let Codec::Dbcs(page) = enc.codec() else {
panic!("{enc} should be double-byte");
};
assert!(page.decode.windows(2).all(|w| w[0].0 < w[1].0));
assert!(page.encode.windows(2).all(|w| w[0].0 < w[1].0));
}
}
#[test]
fn dbcs_decode_cstr_stops_at_nul() {
let enc = DosEncoding::Cp932;
assert_eq!(enc.decode_cstr(b"\x93\xFA\x00garbage").unwrap(), "\u{65E5}");
}
#[test]
fn astral_characters_are_unmappable() {
// Nothing beyond the BMP exists in these pages; the encoder must not
// truncate the scalar when searching.
let err = DosEncoding::Cp936.encode("🐍").unwrap_err();
assert!(matches!(err, EncodeError::Unmappable { ch: '🐍', .. }));
}
}

15332
src/tables/cp932.rs Normal file
View File

File diff suppressed because it is too large Load Diff

43864
src/tables/cp936.rs Normal file
View File

File diff suppressed because it is too large Load Diff

34378
src/tables/cp949.rs Normal file
View File

File diff suppressed because it is too large Load Diff

27278
src/tables/cp950.rs Normal file
View File

File diff suppressed because it is too large Load Diff

39
src/tables/mod.rs Normal file
View File

@@ -0,0 +1,39 @@
//! Code page tables. Everything except this module file is generated by
//! `tools/gen_tables.py` from the vendored Microsoft mapping tables in
//! `data/` — regenerate rather than editing by hand.
pub(crate) mod single;
#[cfg(feature = "dbcs")]
pub(crate) mod cp932;
#[cfg(feature = "dbcs")]
pub(crate) mod cp936;
#[cfg(feature = "dbcs")]
pub(crate) mod cp949;
#[cfg(feature = "dbcs")]
pub(crate) mod cp950;
/// One slot of a double-byte page's single-byte table.
#[cfg(feature = "dbcs")]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum SingleEntry {
/// The byte decodes to this character on its own.
Map(char),
/// The byte is a DBCS lead byte; a trail byte must follow.
Lead,
/// The code page leaves the byte undefined.
Undefined,
}
/// A double-byte code page: the single-byte table plus sorted pair tables
/// for binary search in each direction.
#[cfg(feature = "dbcs")]
pub(crate) struct DbcsPage {
/// How each lone byte value behaves.
pub single: &'static [SingleEntry; 256],
/// Every defined pair, sorted by 16-bit code (`lead << 8 | trail`).
pub decode: &'static [(u16, char)],
/// One pair per scalar, sorted by scalar; where several codes decode to
/// one scalar, the winner matches Microsoft's bestfit WCTABLE.
pub encode: &'static [(char, u16)],
}

9
src/tables.rs → src/tables/single.rs
View File

@@ -1,10 +1,7 @@
//! Generated code page tables — do not edit by hand.
//! Generated by `tools/gen_tables.py` — do not edit by hand.
//!
//! Source: the Unicode Consortium's Microsoft mapping tables
//! (`VENDORS/MICSFT/PC/CP*.TXT`), vendored in `data/` and regenerated with
//! `tools/gen_tables.py`. Each table maps every byte value `0x00..=0xFF` to
//! its Unicode scalar, or `None` where the code page leaves the byte
//! undefined.
//! Source: the Unicode Consortium's Microsoft mapping tables, vendored in
//! `data/`.
/// Code page 437, byte value to Unicode scalar.
pub(crate) const CP437: [Option<char>; 256] = [

354
tools/gen_tables.py
View File

@@ -2,123 +2,353 @@
# /// script
# requires-python = ">=3.10"
# ///
"""Generate src/tables.rs from the Microsoft mapping tables in data/.
"""Generate src/tables/ from the Microsoft mapping tables in data/.
The inputs are the canonical Microsoft code page tables published by the
Unicode Consortium (https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/).
Each output table is a full 256-entry ``[Option<char>; 256]``: MS-DOS pages are
not uniformly ASCII in the low half (CP864 maps 0x25 to ARABIC PERCENT SIGN)
and several pages leave byte values undefined (CP857/864/869/874).
Unicode Consortium (https://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/,
``PC/`` for the single-byte DOS pages and ``WINDOWS/`` for the double-byte
CJK pages).
Single-byte pages become full 256-entry ``[Option<char>; 256]`` tables:
MS-DOS pages are not uniformly ASCII in the low half (CP864 maps 0x25 to
ARABIC PERCENT SIGN) and several pages leave byte values undefined
(CP857/864/869/874).
Double-byte pages become a 256-entry single-byte/lead-byte table plus a pair
of sorted ``(code, scalar)`` arrays for binary search. Where several codes
decode to one scalar (CP932's NEC/IBM overlap, CP950's box-drawing rows),
the encoder's choice is resolved from Microsoft's own ``bestfit*.txt``
WCTABLE — used *only* to pick among duplicate candidates, never to import
lossy best-fit mappings.
Run from the crate root: ``uv run tools/gen_tables.py``
(needs ``rustfmt`` on PATH).
"""
from __future__ import annotations
import re
import subprocess
from collections.abc import Sequence
from dataclasses import dataclass
from pathlib import Path
CODE_PAGES: Sequence[int] = (
SINGLE_BYTE_PAGES: Sequence[int] = (
437, 737, 775, 850, 852, 855, 857, 860,
861, 862, 863, 864, 865, 866, 869, 874,
)
DOUBLE_BYTE_PAGES: Sequence[int] = (932, 936, 949, 950)
DATA_DIR = Path(__file__).resolve().parent.parent / "data"
OUTPUT = Path(__file__).resolve().parent.parent / "src" / "tables.rs"
OUTPUT_DIR = Path(__file__).resolve().parent.parent / "src" / "tables"
HEADER = """\
//! Generated code page tables — do not edit by hand.
GENERATED_NOTE = """\
//! Generated by `tools/gen_tables.py` — do not edit by hand.
//!
//! Source: the Unicode Consortium's Microsoft mapping tables
//! (`VENDORS/MICSFT/PC/CP*.TXT`), vendored in `data/` and regenerated with
//! `tools/gen_tables.py`. Each table maps every byte value `0x00..=0xFF` to
//! its Unicode scalar, or `None` where the code page leaves the byte
//! undefined.
//! Source: the Unicode Consortium's Microsoft mapping tables, vendored in
//! `data/`.
"""
SURROGATE_RANGE = range(0xD800, 0xE000)
class TableError(Exception):
"""A mapping table failed to parse or violated an invariant."""
def parse_table(path: Path) -> list[int | None]:
"""Parse one CP*.TXT into a 256-entry byte-to-scalar table.
@dataclass
class CodePageTable:
"""One parsed Microsoft mapping table.
Args:
path: The mapping table file.
Attributes:
single: Byte value to Unicode scalar for single-byte codes.
leads: Byte values marked ``DBCS LEAD BYTE`` (empty for single-byte
pages).
pairs: 16-bit lead<<8|trail code to Unicode scalar.
listed_bytes: Every byte value the file listed, mapped or not.
"""
Returns:
A list indexed by byte value; ``None`` marks an undefined byte.
single: dict[int, int]
leads: set[int]
pairs: dict[int, int]
listed_bytes: set[int]
def read_mapping_lines(path: Path) -> list[str]:
"""Read a mapping file and return its data lines (those starting 0x).
Raises:
TableError: If the file is missing or non-ASCII, a byte is out of
range or listed twice, a byte is never listed, or two bytes map
to the same scalar (the encoder requires a bijection).
TableError: If the file is missing or non-ASCII (the lone stray byte
in canonical data is a DOS EOF marker, 0x1A, which is still ASCII).
"""
try:
# The lone stray byte in canonical data is a DOS EOF marker (0x1A),
# which is still ASCII; anything else should fail loudly.
text = path.read_text(encoding="ascii")
except (OSError, UnicodeDecodeError) as error:
raise TableError(f"{path.name}: unreadable mapping table") from error
table: list[int | None] = [None] * 256
return [line.strip() for line in text.splitlines() if line.strip().startswith("0x")]
def parse_table(path: Path) -> CodePageTable:
"""Parse one CP*.TXT into single-byte, lead-byte, and pair mappings.
Raises:
TableError: If a code is listed twice, a single byte is out of range,
a scalar is a surrogate or beyond the BMP, or byte 0x00 appears
as a DBCS trail (which would break NUL-terminated decoding).
"""
table = CodePageTable(single={}, leads=set(), pairs={}, listed_bytes=set())
seen: set[int] = set()
for raw_line in text.splitlines():
line = raw_line.strip()
if not line.startswith("0x"):
continue
for line in read_mapping_lines(path):
columns = line.split("\t")
byte = int(columns[0], 16)
if byte > 0xFF:
raise TableError(f"{path.name}: byte {byte:#x} out of range")
if byte in seen:
raise TableError(f"{path.name}: byte {byte:#04x} listed twice")
seen.add(byte)
if len(columns) >= 2 and columns[1].strip().startswith("0x"):
table[byte] = int(columns[1], 16)
if len(seen) != 256:
missing = sorted(set(range(256)) - seen)
raise TableError(f"{path.name}: bytes never listed: {missing}")
defined = [scalar for scalar in table if scalar is not None]
if len(defined) != len(set(defined)):
raise TableError(f"{path.name}: table is not a bijection")
code = int(columns[0], 16)
if code in seen:
raise TableError(f"{path.name}: code {code:#06x} listed twice")
seen.add(code)
if code > 0xFFFF:
raise TableError(f"{path.name}: code {code:#x} out of range")
if code <= 0xFF:
table.listed_bytes.add(code)
mapped = len(columns) >= 2 and columns[1].strip().startswith("0x")
if not mapped:
if code <= 0xFF and "DBCS LEAD BYTE" in line:
table.leads.add(code)
continue
scalar = int(columns[1], 16)
if scalar in SURROGATE_RANGE or scalar > 0xFFFF:
raise TableError(f"{path.name}: scalar {scalar:#06x} not a BMP char")
if code <= 0xFF:
table.single[code] = scalar
else:
if code & 0xFF == 0:
raise TableError(f"{path.name}: NUL trail byte in {code:#06x}")
table.pairs[code] = scalar
unmarked_leads = {code >> 8 for code in table.pairs} - table.leads
if unmarked_leads:
raise TableError(
f"{path.name}: pairs under bytes not marked DBCS LEAD BYTE: "
f"{[hex(lead) for lead in sorted(unmarked_leads)]}"
)
return table
def rust_entry(scalar: int | None) -> str:
"""Render one table slot as a Rust ``Option<char>`` literal."""
if scalar is None:
return "None"
def parse_wctable(path: Path) -> dict[int, int]:
"""Parse the WCTABLE (Unicode to codepage) section of a bestfit file.
Raises:
TableError: If the file is missing or has no WCTABLE section.
"""
try:
# Comments embed raw codepage-encoded glyph bytes; latin-1 accepts
# any byte and the data lines we match are pure ASCII.
text = path.read_text(encoding="latin-1")
except OSError as error:
raise TableError(f"{path.name}: unreadable bestfit table") from error
_, marker, rest = text.partition("WCTABLE")
if not marker:
raise TableError(f"{path.name}: no WCTABLE section")
section = rest.split("ENDCODEPAGE")[0]
entry = re.compile(r"^0x([0-9a-fA-F]{4})\s+0x([0-9a-fA-F]+)")
table: dict[int, int] = {}
for line in section.splitlines():
if match := entry.match(line.strip()):
table[int(match.group(1), 16)] = int(match.group(2), 16)
return table
def check_bijection(path_name: str, table: CodePageTable) -> None:
"""Require every scalar to have exactly one code (single-byte pages).
Raises:
TableError: If two codes map to the same scalar.
"""
scalars = list(table.single.values()) + list(table.pairs.values())
if len(scalars) != len(set(scalars)):
raise TableError(f"{path_name}: table is not a bijection")
def resolve_encode(
path_name: str, table: CodePageTable, wctable: dict[int, int]
) -> dict[int, int]:
"""Choose one code per scalar, resolving duplicates via the WCTABLE.
Args:
path_name: The mapping file name, for error messages.
table: The parsed decode-direction table.
wctable: Microsoft's Unicode-to-codepage table for this page.
Returns:
Scalar to pair-code for the double-byte portion only (single-byte
codes never collide with pairs in these tables, which is asserted).
Raises:
TableError: If a duplicate set is not settled by the WCTABLE, mixes
single-byte and pair codes (the Rust encoder tries the single
table first, so a pair winner would be unreachable), or has a
single-byte winner that is not the lowest byte (the Rust encoder
scans upward, so any other winner would be shadowed).
"""
candidates: dict[int, list[int]] = {}
for code, scalar in list(table.single.items()) + list(table.pairs.items()):
candidates.setdefault(scalar, []).append(code)
encode: dict[int, int] = {}
for scalar, codes in candidates.items():
if len(codes) == 1:
chosen = codes[0]
else:
if any(c <= 0xFF for c in codes) and any(c > 0xFF for c in codes):
raise TableError(
f"{path_name}: U+{scalar:04X} mixes single and pair codes "
f"({[hex(c) for c in codes]})"
)
chosen = wctable.get(scalar, -1)
if chosen not in codes:
raise TableError(
f"{path_name}: WCTABLE does not settle duplicate scalar "
f"U+{scalar:04X} (candidates {[hex(c) for c in codes]})"
)
if chosen <= 0xFF and chosen != min(codes):
raise TableError(
f"{path_name}: U+{scalar:04X} single-byte winner "
f"{chosen:#04x} is not the lowest candidate"
)
if chosen <= 0xFF:
continue # single-byte encoding is derived from the single table
encode[scalar] = chosen
return encode
def rust_char(scalar: int) -> str:
"""Render a Unicode scalar as a Rust char literal."""
ch = chr(scalar)
if ch.isprintable() and ch not in ("'", "\\"):
return f"Some('{ch}')"
return f"Some('\\u{{{scalar:04X}}}')"
return f"'{ch}'"
return f"'\\u{{{scalar:04X}}}'"
def render_table(code_page: int, table: Sequence[int | None]) -> str:
"""Render one code page as a Rust constant, eight entries per line."""
def render_single_byte_page(code_page: int, table: CodePageTable) -> str:
"""Render one single-byte code page as an ``[Option<char>; 256]``."""
lines = [
f"/// Code page {code_page}, byte value to Unicode scalar.",
f"pub(crate) const CP{code_page}: [Option<char>; 256] = [",
]
for row_start in range(0, 256, 8):
row = ", ".join(rust_entry(table[i]) for i in range(row_start, row_start + 8))
lines.append(f" {row}, // {row_start:#04x}-{row_start + 7:#04x}")
entries = []
for byte in range(row_start, row_start + 8):
scalar = table.single.get(byte)
entries.append("None" if scalar is None else f"Some({rust_char(scalar)})")
lines.append(f" {', '.join(entries)}, // {row_start:#04x}-{row_start + 7:#04x}")
lines.append("];\n")
return "\n".join(lines)
def render_pair_array(name: str, doc: str, pairs: Sequence[tuple[int, int]]) -> str:
"""Render a sorted ``[(u16, char); N]`` static, eight entries per line."""
lines = [f"/// {doc}", f"static {name}: [(u16, char); {len(pairs)}] = ["]
for row_start in range(0, len(pairs), 8):
row = pairs[row_start : row_start + 8]
rendered = ", ".join(f"({code:#06x}, {rust_char(scalar)})" for code, scalar in row)
lines.append(f" {rendered},")
lines.append("];\n")
return "\n".join(lines)
def render_encode_array(name: str, doc: str, pairs: Sequence[tuple[int, int]]) -> str:
"""Render a sorted ``[(char, u16); N]`` static, eight entries per line."""
lines = [f"/// {doc}", f"static {name}: [(char, u16); {len(pairs)}] = ["]
for row_start in range(0, len(pairs), 8):
row = pairs[row_start : row_start + 8]
rendered = ", ".join(f"({rust_char(scalar)}, {code:#06x})" for scalar, code in row)
lines.append(f" {rendered},")
lines.append("];\n")
return "\n".join(lines)
def render_double_byte_page(
code_page: int, table: CodePageTable, encode: dict[int, int]
) -> str:
"""Render one DBCS page as a module body with single/decode/encode tables."""
single_lines = [
"/// Single-byte slots: mapped char, DBCS lead byte, or undefined.",
"const SINGLE: [SingleEntry; 256] = [",
]
for row_start in range(0, 256, 8):
entries = []
for byte in range(row_start, row_start + 8):
if byte in table.leads:
entries.append("L")
elif (scalar := table.single.get(byte)) is not None:
entries.append(f"M({rust_char(scalar)})")
else:
entries.append("U")
single_lines.append(
f" {', '.join(entries)}, // {row_start:#04x}-{row_start + 7:#04x}"
)
single_lines.append("];\n")
decode_pairs = sorted(table.pairs.items())
encode_pairs = sorted(encode.items())
return "\n".join(
[
GENERATED_NOTE,
"use super::SingleEntry::{self, Lead as L, Map as M, Undefined as U};",
"use super::DbcsPage;",
"",
f"/// Code page {code_page}.",
f"pub(crate) static CP{code_page}: DbcsPage = DbcsPage {{",
" single: &SINGLE,",
" decode: &DECODE,",
" encode: &ENCODE,",
"};",
"",
"\n".join(single_lines),
render_pair_array(
"DECODE",
"Every defined pair, sorted by 16-bit code (lead<<8 | trail).",
decode_pairs,
),
render_encode_array(
"ENCODE",
"One pair per scalar, sorted by scalar; duplicates resolved per "
"Microsoft's bestfit WCTABLE.",
encode_pairs,
),
]
)
def main() -> None:
"""Regenerate src/tables.rs from every table in CODE_PAGES."""
rendered = [HEADER]
for code_page in CODE_PAGES:
"""Regenerate src/tables/ from every configured code page."""
outputs: list[Path] = []
single_rendered = [GENERATED_NOTE]
for code_page in SINGLE_BYTE_PAGES:
table = parse_table(DATA_DIR / f"CP{code_page}.TXT")
rendered.append(render_table(code_page, table))
OUTPUT.write_text("\n".join(rendered), encoding="utf-8")
subprocess.run(["rustfmt", "--edition", "2021", str(OUTPUT)], check=True)
print(f"wrote {OUTPUT} ({len(CODE_PAGES)} tables)")
if table.listed_bytes != set(range(256)):
missing = sorted(set(range(256)) - table.listed_bytes)
raise TableError(f"CP{code_page}: bytes never listed: {missing}")
if table.leads or table.pairs:
raise TableError(f"CP{code_page}: unexpected DBCS content")
check_bijection(f"CP{code_page}", table)
single_rendered.append(render_single_byte_page(code_page, table))
single_path = OUTPUT_DIR / "single.rs"
single_path.write_text("\n".join(single_rendered), encoding="utf-8")
outputs.append(single_path)
for code_page in DOUBLE_BYTE_PAGES:
table = parse_table(DATA_DIR / f"CP{code_page}.TXT")
wctable = parse_wctable(DATA_DIR / f"bestfit{code_page}.txt")
encode = resolve_encode(f"CP{code_page}", table, wctable)
page_path = OUTPUT_DIR / f"cp{code_page}.rs"
page_path.write_text(
render_double_byte_page(code_page, table, encode), encoding="utf-8"
)
outputs.append(page_path)
subprocess.run(
["rustfmt", "--edition", "2021", *map(str, outputs)], check=True
)
for path in outputs:
print(f"wrote {path}")
if __name__ == "__main__":