Skip to content

In-house Thumb decoder; capstone becomes a dev dependency - #125

Closed
BrianPugh wants to merge 1 commit into
mainfrom
thumb-decoder
Closed

BrianPugh wants to merge 1 commit into
mainfrom
thumb-decoder

Conversation

@BrianPugh

@BrianPugh BrianPugh commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Second step toward shipping the patcher in gnwmanager without extra native dependencies (#124 is the first; this PR is independent of it). capstone was a runtime dependency because the recursive descent in patches/xrefs.py disassembles the stock firmware, and relocate.py/xip.py re-descend the patched image at patch time, so caching the stock map couldn't remove it.

The decoder

patches/thumb_disasm.py decodes ARMv7E-M + FPv5 Thumb-2 into an Insn with only the fields the descent reads: size, branch target, PC-relative load (displacement and size) or address (adr, addw/subw rd, pc), the registers the pc-offset (ldr rX, [pc]; add rX, pc; adds rX, #adj) and table-branch (cmp rX, #n; tbb) patterns track, IT slots, and whether it writes or mentions PC. There's no assembly text. xrefs.Code._run keeps every rule it had; they read these fields now instead of capstone operands. Code.insns holds Insn objects instead of (size, mnemonic, op_str) tuples.

The fields follow capstone's operand model on purpose, quirks included (e.g. a strd [pc] counts as a word load, blo/bls start with bl), so the map can't change.

How it's checked

decode_capstone() derives the same fields from capstone using the old rules verbatim. tests/test_thumb_disasm.py compares the 2:

  • Every 16-bit encoding: identical wherever capstone decodes (1 extra the other way, 0xDEF9, which capstone rejects).
  • 100k random 32-bit encodings: identical, except an allowlist of encodings outside ARMv7E-M (A-profile hvc/smc, ARMv8-M tt/stl*/lda*/*_ns, the MP extension's pldw, NEON, FP16, non-FP coprocessors) and 2 capstone quirks (udf.w with any second halfword; rev/rbit/clz with mismatched Rm fields). I also ran 900k more by hand with no other differences.
  • The whole stock map of both models: instructions, branches, PC-relative refs, tables, pc-offsets, refs, byte classification, units and external refs are identical. The descent never meets an encoding capstone rejects there (0 of 111,644 instructions), so validity differences on garbage can't matter.

The golden hashes are unchanged, also with capstone blocked at import (checked zelda_xip_relocate, which exercises relocate.py and xip.py). The descent is ~1.5x faster (Zelda's stock map: 1.8 s vs 2.5 s).

capstone is dev-only now

  • requirements.txt drops it; the new requirements-dev.txt adds capstone and pytest.
  • tests/test_thumb_disasm.py's comparisons, tests/test_thumb_asm.py (capstone's text is its oracle) and tests/test_match_roms.py skip without capstone.
  • tools/re/match_roms.py signs functions by instruction text, so it runs the analysis with decode_capstone.
  • tests/test_relocate.py checks relocate's branch and PC-relative encoders with the in-house decoder.

No output changes, so no hardware check needed.

🤖 Generated with Claude Code

The recursive descent in patches/xrefs.py (the stock map, and the
re-descents relocate.py and xip.py run on the patched image) used
capstone at patch time. gnwmanager will ship the patcher, and it
shouldn't need capstone.

patches/thumb_disasm.py decodes ARMv7E-M/FPv5 Thumb-2 into only the
fields the descent reads: size, branch target, PC-relative load and
address, the registers the pc-offset and table-branch patterns track,
and whether it writes PC. Code._run keeps its rules, now on those fields.

decode_capstone() derives the same fields from capstone with the old
rules. tests/test_thumb_disasm.py compares the 2 over every 16-bit
encoding, 100k random 32-bit ones (allowing only encodings outside
ARMv7E-M and a few capstone quirks), and the whole stock map of both
models, which is identical. The golden hashes are unchanged, and the
descent is ~1.5x faster.

capstone moves to requirements-dev.txt: the decoder tests,
test_thumb_asm.py and tools/re/match_roms.py (which signs functions by
instruction text) skip or need it; test_relocate.py checks the branch
encoders with the in-house decoder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BrianPugh

Copy link
Copy Markdown
Owner Author

Closing this one. capstone is actively maintained and ships prebuilt wheels for the common platforms (keystone was the real problem, and that's already gone), so I don't think there's a strong enough reason to own a Thumb decoder. The web builder doesn't use Pyodide, which was the other main motivation. The branch stays around in case that changes.

@BrianPugh BrianPugh closed this Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant