Expand to twelve single-byte encodings via codegen; match msdos-encodings API

Rework the crate on the msdos-encodings design: vendored Unicode
Consortium VENDORS/APPLE mapping files in data/apple/, a
tools/gen_tables.py generator (PEP 723 / uv run / rustfmt) emitting full
[Option<char>; 256] tables, and a fallible decode / decode_lossy /
encode API with non_exhaustive thiserror errors.

Encodings: Roman, Greek, Cyrillic, Central European Roman, Turkish,
Croatian, Icelandic, Romanian, Celtic, Gaelic, Ukrainian (pre-9.0
Cyrillic variant), Inuit. The eight encodings Apple revised for the
Euro carry a Revision payload (Classic = pre-8.5/9.0/9.2.2), generated
from per-encoding deltas with invariant checks that fail regeneration
if the vendored data drifts.

Apple-specific API points beyond the msdos shape: from_text_encoding
(kTextEncodingMac… base values, the HFS+ hint), from_script_code plus
from_fd_script (bit-7 validity flag on the raw Finder Info byte), and
decode_pstr for length-prefixed Str27/Str31 name fields with a
TruncatedPascalString error for corrupt lengths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5
2026-07-19 14:07:44 -05:00
parent 4f1439af65
commit bbf6d6b2b9
19 changed files with 10193 additions and 205 deletions

View File

@@ -4,28 +4,84 @@ Bidirectional, emulator-grade conversion between classic Mac OS text encodings
and Unicode — without linking ICU. The crate owns the canonical Apple tables, so
it stays small, self-contained, and cross-compiles cleanly.
```rust
use apple_encodings::{AppleEncoding, MacRomanRevision};
These are strictly **Apple's** mappings as published by the Unicode Consortium
(`VENDORS/APPLE/`), vendored in `data/apple/`.
let enc = AppleEncoding::default(); // Mac OS Roman, post-8.5
assert_eq!(enc.decode(b"Caf\x8e"), "Café");
```rust
use apple_encodings::{AppleEncoding, Revision};
let enc = AppleEncoding::default(); // Mac OS Roman, modern revision
assert_eq!(enc.decode(b"Caf\x8e").unwrap(), "Café");
assert_eq!(enc.encode("Café").unwrap(), b"Caf\x8e");
// Pick by the Finder Info `fdScript` byte:
let enc = AppleEncoding::from_script_code(apple_encodings::SCRIPT_ROMAN).unwrap();
// Pick by the HFS+ `textEncoding` hint (kTextEncodingMac… base values):
let enc = AppleEncoding::from_text_encoding(7).unwrap();
assert_eq!(enc.decode(&[0x80]).unwrap(), "А"); // Mac OS Cyrillic
// Revision matters for exactly one byte (0xDB):
let classic = AppleEncoding::MacRoman(MacRomanRevision::Classic);
assert_eq!(classic.decode(&[0xDB]), "¤"); // pre-8.5 currency sign, not €
// Or by the raw HFS Finder Info `fdScript` byte (bit 7 flags validity;
// a Cyrillic system writes 0x87). Script system, not table — the Roman
// regional variants all report script 0:
let enc = AppleEncoding::from_fd_script(0x87).unwrap();
assert_eq!(enc.name(), "Mac OS Cyrillic");
// Classic Mac name fields are Pascal strings (Str27/Str31):
let name = enc.decode_pstr(b"\x08\x84\xEE\xEA\xF3\xEC\xE5\xED\xF2").unwrap();
assert_eq!(name, "Документ");
// The Euro rollout revised several tables, at different bytes per encoding;
// pick the pre-Euro table for older-system fidelity:
let classic = AppleEncoding::MacRoman(Revision::Classic);
assert_eq!(classic.decode(&[0xDB]).unwrap(), "¤"); // pre-8.5, not €
// Decode is fallible for API uniformity (Greek pre-9.2.2 leaves 0xFF
// undefined), or go lossy:
let greek = AppleEncoding::MacGreek(Revision::Classic);
assert!(greek.decode(&[0xFF]).is_err());
assert_eq!(greek.decode_lossy(&[0xFF]), "\u{FFFD}");
```
## Status
- **Mac OS Roman** — implemented, both pre- and post-8.5 revisions, decode + encode.
- **Regional single-byte** (Cyrillic, Greek, Turkish, …) and **CJK double-byte**
(Japanese, Big5, GB, Korean) — planned, to be codegen'd from the Unicode
Consortium `VENDORS/APPLE/*.TXT` tables. The double-byte tables will be
feature-gated.
- **Twelve single-byte encodings** — implemented, decode + encode: Roman,
Greek, Cyrillic, Central European Roman, Turkish, Croatian, Icelandic,
Romanian, Celtic, Gaelic, Ukrainian (the pre-9.0 Cyrillic variant), and
Inuit.
- **Pre-Euro revisions** — implemented for the eight encodings Apple revised:
Roman/Croatian/Icelandic/Romanian/Celtic/Gaelic (`0xDB`, Mac OS 8.5),
Cyrillic (`0xA2`/`0xB6`/`0xFF`, Mac OS 9.0), Greek (`0x9C`/`0xFF`,
Mac OS 9.2.2).
- **CJK double-byte** (Japanese, Chinese, Korean) and the
**bidirectional/complex scripts** (Arabic, Hebrew, Thai, Indic — which need
multi-scalar decode) — planned, behind the same API; the double-byte tables
will be feature-gated.
## Fidelity notes
- Every table is a verified bijection: anything an encoding decodes re-encodes
to the identical bytes.
- Every table is total except Mac OS Greek at `Revision::Classic`, whose
`0xFF` was undefined before Mac OS 9.2.2 — so `decode` returns a `Result`
and `decode_lossy` substitutes U+FFFD, matching the `msdos-encodings`
sister crate.
- Apple's corporate Private Use Area mappings are preserved, not normalized:
the Apple logo (`0xF0``U+F8FF` in Roman) and Mac OS Turkish's explicit
"undefined character" slot (`0xF5``U+F8A0`) round-trip exactly.
- The pre-Euro revision tables are generated from per-encoding deltas taken
from each mapping file's own change history, with invariant checks that
fail regeneration loudly if the vendored data ever disagrees — the Euro
landed at different bytes per encoding, so nothing is hand-placed.
- Mac OS Ukrainian is derived from `CYRILLIC.TXT` plus its documented delta
(`UKRAINE.TXT` upstream is a notes-only stub: Mac OS 9.0 retired the
separate Ukrainian character set).
## Regenerating tables
Everything in `src/tables/` except `mod.rs` is generated from the vendored
mapping files in `data/apple/` (needs Python ≥ 3.10 and `rustfmt` on `PATH`):
```sh
uv run tools/gen_tables.py
```
## Scope