Expand to twelve single-byte encodings via codegen; match msdos-encodings API
Rework the crate on the msdos-encodings design: vendored Unicode Consortium VENDORS/APPLE mapping files in data/apple/, a tools/gen_tables.py generator (PEP 723 / uv run / rustfmt) emitting full [Option<char>; 256] tables, and a fallible decode / decode_lossy / encode API with non_exhaustive thiserror errors. Encodings: Roman, Greek, Cyrillic, Central European Roman, Turkish, Croatian, Icelandic, Romanian, Celtic, Gaelic, Ukrainian (pre-9.0 Cyrillic variant), Inuit. The eight encodings Apple revised for the Euro carry a Revision payload (Classic = pre-8.5/9.0/9.2.2), generated from per-encoding deltas with invariant checks that fail regeneration if the vendored data drifts. Apple-specific API points beyond the msdos shape: from_text_encoding (kTextEncodingMac… base values, the HFS+ hint), from_script_code plus from_fd_script (bit-7 validity flag on the raw Finder Info byte), and decode_pstr for length-prefixed Str27/Str31 name fields with a TruncatedPascalString error for corrupt lengths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
84
README.md
84
README.md
@@ -4,28 +4,84 @@ Bidirectional, emulator-grade conversion between classic Mac OS text encodings
|
||||
and Unicode — without linking ICU. The crate owns the canonical Apple tables, so
|
||||
it stays small, self-contained, and cross-compiles cleanly.
|
||||
|
||||
```rust
|
||||
use apple_encodings::{AppleEncoding, MacRomanRevision};
|
||||
These are strictly **Apple's** mappings as published by the Unicode Consortium
|
||||
(`VENDORS/APPLE/`), vendored in `data/apple/`.
|
||||
|
||||
let enc = AppleEncoding::default(); // Mac OS Roman, post-8.5
|
||||
assert_eq!(enc.decode(b"Caf\x8e"), "Café");
|
||||
```rust
|
||||
use apple_encodings::{AppleEncoding, Revision};
|
||||
|
||||
let enc = AppleEncoding::default(); // Mac OS Roman, modern revision
|
||||
assert_eq!(enc.decode(b"Caf\x8e").unwrap(), "Café");
|
||||
assert_eq!(enc.encode("Café").unwrap(), b"Caf\x8e");
|
||||
|
||||
// Pick by the Finder Info `fdScript` byte:
|
||||
let enc = AppleEncoding::from_script_code(apple_encodings::SCRIPT_ROMAN).unwrap();
|
||||
// Pick by the HFS+ `textEncoding` hint (kTextEncodingMac… base values):
|
||||
let enc = AppleEncoding::from_text_encoding(7).unwrap();
|
||||
assert_eq!(enc.decode(&[0x80]).unwrap(), "А"); // Mac OS Cyrillic
|
||||
|
||||
// Revision matters for exactly one byte (0xDB):
|
||||
let classic = AppleEncoding::MacRoman(MacRomanRevision::Classic);
|
||||
assert_eq!(classic.decode(&[0xDB]), "¤"); // pre-8.5 currency sign, not €
|
||||
// Or by the raw HFS Finder Info `fdScript` byte (bit 7 flags validity;
|
||||
// a Cyrillic system writes 0x87). Script system, not table — the Roman
|
||||
// regional variants all report script 0:
|
||||
let enc = AppleEncoding::from_fd_script(0x87).unwrap();
|
||||
assert_eq!(enc.name(), "Mac OS Cyrillic");
|
||||
|
||||
// Classic Mac name fields are Pascal strings (Str27/Str31):
|
||||
let name = enc.decode_pstr(b"\x08\x84\xEE\xEA\xF3\xEC\xE5\xED\xF2").unwrap();
|
||||
assert_eq!(name, "Документ");
|
||||
|
||||
// The Euro rollout revised several tables, at different bytes per encoding;
|
||||
// pick the pre-Euro table for older-system fidelity:
|
||||
let classic = AppleEncoding::MacRoman(Revision::Classic);
|
||||
assert_eq!(classic.decode(&[0xDB]).unwrap(), "¤"); // pre-8.5, not €
|
||||
|
||||
// Decode is fallible for API uniformity (Greek pre-9.2.2 leaves 0xFF
|
||||
// undefined), or go lossy:
|
||||
let greek = AppleEncoding::MacGreek(Revision::Classic);
|
||||
assert!(greek.decode(&[0xFF]).is_err());
|
||||
assert_eq!(greek.decode_lossy(&[0xFF]), "\u{FFFD}");
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
- **Mac OS Roman** — implemented, both pre- and post-8.5 revisions, decode + encode.
|
||||
- **Regional single-byte** (Cyrillic, Greek, Turkish, …) and **CJK double-byte**
|
||||
(Japanese, Big5, GB, Korean) — planned, to be codegen'd from the Unicode
|
||||
Consortium `VENDORS/APPLE/*.TXT` tables. The double-byte tables will be
|
||||
feature-gated.
|
||||
- **Twelve single-byte encodings** — implemented, decode + encode: Roman,
|
||||
Greek, Cyrillic, Central European Roman, Turkish, Croatian, Icelandic,
|
||||
Romanian, Celtic, Gaelic, Ukrainian (the pre-9.0 Cyrillic variant), and
|
||||
Inuit.
|
||||
- **Pre-Euro revisions** — implemented for the eight encodings Apple revised:
|
||||
Roman/Croatian/Icelandic/Romanian/Celtic/Gaelic (`0xDB`, Mac OS 8.5),
|
||||
Cyrillic (`0xA2`/`0xB6`/`0xFF`, Mac OS 9.0), Greek (`0x9C`/`0xFF`,
|
||||
Mac OS 9.2.2).
|
||||
- **CJK double-byte** (Japanese, Chinese, Korean) and the
|
||||
**bidirectional/complex scripts** (Arabic, Hebrew, Thai, Indic — which need
|
||||
multi-scalar decode) — planned, behind the same API; the double-byte tables
|
||||
will be feature-gated.
|
||||
|
||||
## Fidelity notes
|
||||
|
||||
- Every table is a verified bijection: anything an encoding decodes re-encodes
|
||||
to the identical bytes.
|
||||
- Every table is total except Mac OS Greek at `Revision::Classic`, whose
|
||||
`0xFF` was undefined before Mac OS 9.2.2 — so `decode` returns a `Result`
|
||||
and `decode_lossy` substitutes U+FFFD, matching the `msdos-encodings`
|
||||
sister crate.
|
||||
- Apple's corporate Private Use Area mappings are preserved, not normalized:
|
||||
the Apple logo (`0xF0` → `U+F8FF` in Roman) and Mac OS Turkish's explicit
|
||||
"undefined character" slot (`0xF5` → `U+F8A0`) round-trip exactly.
|
||||
- The pre-Euro revision tables are generated from per-encoding deltas taken
|
||||
from each mapping file's own change history, with invariant checks that
|
||||
fail regeneration loudly if the vendored data ever disagrees — the Euro
|
||||
landed at different bytes per encoding, so nothing is hand-placed.
|
||||
- Mac OS Ukrainian is derived from `CYRILLIC.TXT` plus its documented delta
|
||||
(`UKRAINE.TXT` upstream is a notes-only stub: Mac OS 9.0 retired the
|
||||
separate Ukrainian character set).
|
||||
|
||||
## Regenerating tables
|
||||
|
||||
Everything in `src/tables/` except `mod.rs` is generated from the vendored
|
||||
mapping files in `data/apple/` (needs Python ≥ 3.10 and `rustfmt` on `PATH`):
|
||||
|
||||
```sh
|
||||
uv run tools/gen_tables.py
|
||||
```
|
||||
|
||||
## Scope
|
||||
|
||||
|
||||
Reference in New Issue
Block a user