[cyfung1031]

Summary

A ToUnicode CMap's sequential bfrange is now kept as one record instead of one HashMap entry per code. The parser's 2^20 cap used to count expanded codes and stop parsing, so a valid CMap with more codes than that silently lost its later mappings. The cap now counts mappings (an explicit entry counts 1, a range counts 2 because it covers at least two codes), so every CMap with at most 2^20 codes is kept whole. Lookups below code 65 536 read a table; higher codes use a binary search.

Problem and root cause

parse_to_unicode expanded every code of every sequential bfrange into a HashMap<u32, String> entry and stopped at MAX_CMAP_ENTRIES (1 << 20) expanded codes. The cap protected that expansion by discarding valid mappings: 17 ranges of 65 536 codes (about 1.1 million codes from 752 bytes of CMap) lost the last range, and the same expansion cost about 120 MiB for a 6 KB CMap. The guard was applied to the expanded representation rather than to the compact PDF one.

Scope and non-goals

In scope:

• crates/fonts/src/pdf.rs: sequential ToUnicode ranges become BfRange records. A table gives the covering record for codes below 65 536, and a sorted cover index serves higher codes. Explicit bfchar and array-form entries keep their strings, and so does a one-code range. The later declaration still wins where entries overlap. MAX_CMAP_ENTRIES now counts mappings (explicit = 1, range = 2).
• Metrics::text_of returns Option<Cow<str>> (it returned Option<&str>). The one caller, in crates/edit/src/text.rs, changes push_str(t) to push_str(&t).
• encode finds the lowest code per character from explicit entries, the encoding and each record, with no per-code expansion.

Non-goals (unchanged): codespace and CID range parsing and their MAX_CMAP_RANGES limit; the 65 536-code width limit per bfrange (wider ranges are still ignored, as before, because a lifted limit would need a bounded encode search); Symbol and ZapfDingbats policy, glyph names, shaping, font substitution; stream decode limits; page-content decoding and its 256 MiB and 512 MiB limits (no vendor/ or crates/render change).

Before and after behaviour

• Regression test sequential_ranges_past_the_entry_cap_still_map (17 ranges of 65 536 codes; code 0x100019 must read "Z"). Before: assertion left: None, right: Some("Z"). After: passes.
• Unchanged for CMaps with at most 2^20 codes: every mapping, the precedence rule and every encode result. Checked by the randomized test lookups_match_the_expanded_mappings and by the harness hashes below.
• Changed only where the old cap truncated: codes whose mapping came after the first 2^20 expanded codes now decode, and encode can return them. Where a later range overrode an earlier mapping after the old cut, the later one now wins.
• Performance trade-off: text_of builds a new string for each code that a sequential range maps (the old code returned a stored string). Ordinary decode is faster overall. A CMap in which every code sits in a range (the "dense" row) decodes more slowly per lookup, but uses about half the memory.

Time and memory

Harness: a local cmap_bench test binary (not committed) that builds Metrics from one synthetic ToUnicode CMap and then runs the workload. Decode scenarios perform 65 536 text_of lookups per pass (or 40 000 / 3 400 for the range-only scenarios) over 20 passes. Encode scenarios call encode on 300 synthetic strings over 20 passes. "Before" is the base commit 5d590604867fdac7ed71c555c40477ae3331a60a built with the same harness. Medians of 5 rounds. Peak footprint is the macOS "peak memory footprint" of a single invocation; net is peak minus the peak of the same binary loading the same input with no work.

scenario	command (harness, input)	before: time; peak (net)	after: time; peak (net)	output hash match
ordinary-decode	CMAP_BENCH_SCENARIO=ordinary-decode, ordinary.cmap (97,680 bytes: 6,000 bfchar, 500 bfrange)	24.5 ms; 2.8 MiB (1.7)	12.5 ms; 2.6 MiB (1.5)	match (7,902 codes found)
ordinary-encode	CMAP_BENCH_SCENARIO=ordinary-encode, ordinary.cmap	3.27 s; 5.4 MiB (4.4)	0.47 s; 2.3 MiB (1.3)	match (20 of 300 strings encodable)
dense-decode	CMAP_BENCH_SCENARIO=dense-decode, dense.cmap (681,180 bytes: 32,000 two-code ranges)	43.0 ms; 11.6 MiB (10.1)	61.5 ms; 5.7 MiB (4.1)	match (60,359 codes found)
dense-encode	CMAP_BENCH_SCENARIO=dense-encode, dense.cmap	20.9 s; 11.8 MiB (10.2)	1.44 s; 5.8 MiB (4.2)	match (240 of 300 strings encodable)
wide-decode	CMAP_BENCH_SCENARIO=wide-decode, wide200.cmap (6,083 bytes: 200 ranges of 65 536 codes)	283 ms; 123.8 MiB (122.9)	56.5 ms; 1.5 MiB (0.6)	mismatch, expected: 3,200 codes found before, 40,000 after
seventeen-decode	CMAP_BENCH_SCENARIO=seventeen-decode, seventeen.cmap (752 bytes: 17 ranges of 65 536 codes)	274.5 ms; 122.6 MiB (121.7)	6.0 ms; 1.3 MiB (0.4)	mismatch, expected: 3,200 found before, 3,400 after
adv-decode	CMAP_BENCH_SCENARIO=adv-decode, adv.cmap (29,280,226 bytes: 1,000,000 heavily overlapping two-code ranges)	0.39 s; 66.2 MiB (37.4)	0.30 s; 91.4 MiB (62.6)	match

Rounds and input sizes: one warm-up run per side, then 5 timed rounds per side with the order alternating each round. Decode runs time the mean of 20 back-to-back invocations, because a single invocation takes 10 to 40 ms, close to the 10 ms clock resolution. Encode and adversarial runs time one invocation. Footprint and hashes come from a separate single invocation in the same round. All runs were accepted: the 1-minute load stayed between 3.84 and 6.10, below 16, and nothing was discarded. The whole run of all seven scenarios, warm-ups included, took 4 min 25 s.

The adversarial row is the worst shape for the new representation. The old expansion kept only the distinct codes of overlapping ranges (about 65 thousand), while the new code keeps one record per range up to the budget (500 thousand here). Its peak rises by about 25 MiB and stays bounded by the budget; time still falls.

Output identity

Hashes are FNV-1a (64-bit) over the UTF-8 text of each looked-up code (a 0xFF byte after each text, 0xFE for a missing code); for encode, over the produced bytes. Matching hashes: ordinary-decode 25132a3b221f69d3, ordinary-encode 9538fbf21c3cfcc3, dense-decode c34995344cd3b397, dense-encode edbca6d32d0be03a, adv-decode 680c008c1dfb3a70. The two mismatching rows differ only in codes past the old cut, as described above.

The randomized test lookups_match_the_expanded_mappings builds 300 random ToUnicode CMaps (1 to 12 mappings each, with overlapping ranges, explicit overrides, array-form entries, wrap-around destinations, lone surrogates, surrogate pairs and U+FFFD). It compares text_of for codes 0 to 320, and encode for every character involved, against the expansion in declaration order.

Local checks run

All commands ran from the worktree on the final code (commit on this branch). Results are literal.

1. cargo fmt -p pdfcraft-fonts, then cargo fmt --all -- --check: exit 0.
2. cargo clippy -p pdfcraft-fonts --all-targets -- -D warnings (through the build slot): exit 0.
3. cargo test -p pdfcraft-fonts (through the build slot): 33 passed, 0 failed, exit 0. The new tests are a_range_is_one_record_however_many_codes, explicit_and_range_mappings_override_the_encoding, sequential_ranges_count_against_the_budget, lookups_match_the_expanded_mappings, fffd_lookups_skip_overridden_surrogates, hostile_ranges_stay_bounded, low_codes_resolve_to_the_covering_range, and the regression test sequential_ranges_past_the_entry_cap_still_map.
4. cargo clippy -p pdfcraft-edit --all-targets -- -D warnings: exit 0.
5. cargo test -p pdfcraft-edit -p pdfcraft-redact: edit 47 passed, redact 21 passed, 0 failed, exit 0.
6. Base before the change (base plus the new regression test only), cargo test -p pdfcraft-fonts: 25 passed, 1 failed (sequential_ranges_past_the_entry_cap_still_map, left: None, right: Some("Z")).
7. Release builds of the local harness against the base and against this change (cargo test --release -p pdfcraft-fonts --test cmap_bench --no-run), then the timed runs above.

Checks deferred to CI

The full workspace (cargo test --workspace, cargo clippy --workspace -- -D warnings), cargo xtask ci (assets, parity, layers, WASM check), cargo xtask fuzz, and the Windows, Linux, FreeBSD and WASM builds were not run locally. Local disk and CPU were shared and RULES §3 limits the runs to targeted crates. The CMap parser is a candidate for the fuzzer.

Malformed and hostile input safety

• Budget: explicit entries and ranges share MAX_CMAP_ENTRIES. The result is at most 2^20 explicit entries or 2^19 range records, plus a 256 KiB table for codes below 65 536 (only when some range exists) and an 8 KiB bit set for explicit codes below 65 536.
• Ranges wider than 65 536 codes, reversed ranges and maximal one-code ranges are ignored or mapped only where valid (hostile_ranges_stay_bounded). Malformed hex, surrogates and overflowing code counters are covered by the existing malformed_cmaps_never_panic.
• Lookups below 65 536 are one table read; higher codes are a binary search over the cover index. No lookup walks ranges.
• encode costs O(explicit entries + records × distinct wanted characters). The U+FFFD search checks at most 2,049 surrogate offsets per record.
• Non-test code uses no unwrap, expect, panic!, unreachable!, todo!, unimplemented! or unsafe. Indexing goes through get and get_mut. Arithmetic on input-derived numbers is checked or saturating. The #![deny] clippy lints in crates/fonts/src/lib.rs pass.
• Measured hostile case: adv-decode (29 MB CMap, one million records) peaks at 91 MiB against 66 MiB before, within the budget.

Independence

• Base: origin/main at 5d590604867fdac7ed71c555c40477ae3331a60a, checked again before pushing. The branch is one commit on that base.
• No dependency on the other planned PRs; nothing was cherry-picked or imported.
• M2: stream page content past the 256 MiB cap (item 1 of storytold/pdfcraft#623) #797 overlap gate: gh pr view 797 --json files lists ten paths (crates/render/src/raster.rs and vendor/…). git diff --name-only origin/main...HEAD lists crates/edit/src/text.rs and crates/fonts/src/pdf.rs. There is no common path. M2: stream page content past the 256 MiB cap (item 1 of storytold/pdfcraft#623) #797 changes page-content decoding limits; this change touches only font CMap parsing and lookup.
• Other open PRs touching the same files (informational, not blocking): M7: Arabic and right-to-left text with installed fonts (Add text, Edit text, rendering) #827 and F2: read glyph names by the naming algorithm, and stop reading Symbol as Latin #610 and F1 + F2: exact standard-14 metrics, and code-to-Unicode completeness #357 and Enable clippy::pedantic across the workspace #252 (both files); Arena/753d47d5 linkco pdf #717 and M7: Hebrew in Edit text (reading order, writing order, Hebrew glyphs) #521 (crates/edit/src/text.rs); Fix non-WinAnsi Latin diacritics (ă, ș, ț) in form appearances and support page-level widget annotations #447 (crates/fonts/src/pdf.rs).
• Textual conflicts: none against the current base. If an earlier-merged PR changes unicode_map (encoding inserts) or the text_of callers, a rebase may be needed.

Environment

macOS (Darwin 23.6.0), 8 cores shared with other sessions, rustc 1.95.0. 1-minute load averages during the timed runs were 3.84 to 6.10 (below 16). About 9 GiB of disk was free on / at the start.

🤖 Generated with Claude Code

[echelon]

Thank you @cyfung1031, this is merged!

One change before merging: the below-64k owner table and bit set are now sized to the highest code a font covers, rather than a fixed 256 KB per font. This keeps pages with many small subset fonts cheap. There's a test for it.

If you haven't already, come join us on Discord: https://discord.gg/artcraft