[Feature] Adding Zero Knowledge Proof of Exploit (ZKPoX) - #470
Open
unprovable wants to merge 93 commits into
Open
[Feature] Adding Zero Knowledge Proof of Exploit (ZKPoX) #470unprovable wants to merge 93 commits into
unprovable wants to merge 93 commits into
Conversation
…sclosure envelope
Throwaway prototype tree under docs/research/zkpox-phase0/ plus a
standalone findings deliverable (zkpox-phase0-findings.md). All four
go/no-go questions from the proposal answered yes:
1. SP1 generates a verifying proof of a real freestanding C stack-BOF
(targets/01-stack-bof.c) in 15–52 s CPU-only on Apple Silicon,
2.65 MB core proof; Groth16 wrap to ~256 B is supported but unwired.
2. C-fidelity met via C-via-FFI: the bug lives in actual C source
cross-compiled to RISC-V64 by Homebrew clang from harness/guest/
build.rs and linked into the SP1 Rust guest. The true LLVM-IR
interpreter inside SP1 is documented as Phase-1.x research.
3. Disclosure envelope (AES-256-GCM + age + Drand tlock via tle CLI)
round-trips both vendor and time-lock paths.
4. Two gadgets (crash_only / oob_write) share one redzone primitive
in guest/src/redzone.rs; -fn canarymatch witnesses cleanly
distinguish them.
Findings doc enumerates the 7 issues caught and resolved during the
spike (each a deferred Phase-1 surprise) and a ranked Phase-1
follow-on list starting with the Groth16 wrap.
Phase 0 deliberately omits: any RAPTOR command/skill/persona surface,
real CBOR bundle, Sigstore anchoring, EVM/embedded/x86-64 modes, real
CVE demo, second target corpus, CI integration. Those are Phase 1+.
…+ CBOR bundle Builds on the Phase 0 spike (3f22b60). Spike tree at docs/research/zkpox-phase0/ moved into core/zkpox/ (Rust workspace) and packages/zkpox/ (Python orchestration) via git mv (history preserved). Phase 0 findings doc unchanged at docs/research/zkpox-phase0-findings.md. Phase 1.1 — Productionize the harness * core/zkpox/ Cargo workspace: members guest/prover/verifier; default-members=[prover, verifier] excludes guest from `cargo build` (guest is exclusively `cargo prove build`'d to RISC-V). * packages/zkpox/ matches packages/exploitation/ shape: envelope.py refactored from spike script into seal()/open_via_vendor()/ open_via_tlock() API; prove.py wraps the Rust binary with structured ProveResult/Verdicts dataclasses; tests/ holds 17 passing pytests plus one network-gated skip. Phase 1.2 — Groth16 wrap (wiring; bench deferred) * prover gains --wrap=core|groth16 (default core), --proof-out for persistent artefacts, wrap field in the JSON record. * core branch unchanged from Phase 0; groth16 branch uses client.prove(&pk, stdin).groth16().run() per SP1's fibonacci EVM example. First invocation triggers SP1's multi-GB circuit-artifacts download + ~5-10 min prove; bench captured in a follow-up run. Phase 1.3 — CBOR disclosure bundle * packages/zkpox/bundle.py implements proposal §8 with one deviation: vendor_envelope surfaces the three layered ciphertexts (aes_blob, ct_K_age, ct_K_tlock) directly rather than collapsing into a single opaque field. More honest about the actual scheme. * core/zkpox/verifier/ standalone Rust binary: parses CBOR, validates structural invariants (vendor_pubkey_fingerprint == sha256(vendor_ pubkey), proof.bytes non-empty, vk hash form). End-to-end: 3.6 KB bundle round-trips; tampered fingerprint correctly rejected. * STARK verification on the Rust side stubbed — wiring sp1-sdk into the verifier lands in 1.3.x / 1.5 alongside /verify-exploit-proof. Deliberately deferred: * Sigstore Rekor anchoring (timestamp field is omitted in 1.3, lands in 1.4). * /prove-exploit + /verify-exploit-proof commands, skill, persona (Phase 1.5). * zlib CVE-2018-25032 end-to-end demo (Phase 1.7).
…endix
Phase 1.2 bench surfaced an 18-min cliff: SP1 v6's .groth16().run()
fails with "artifact not found" if its 6.2 GB BN254 trusted-setup
artifacts aren't already at ~/.sp1/circuits/groth16/<ver>/. The SDK
exposes sp1_sdk::install::try_install_circuit_artifacts but never
calls it automatically. The first attempt produced a partially-
downloaded tarball that SP1's artifact loader treated as "already
installed, skipping" — fatally inconsistent.
Fix: prover calls try_install_circuit_artifacts("groth16") via a
single-threaded Tokio runtime before any .groth16().run(). Idempotent;
subsequent runs hit the cache in ms. Without this, every first-time
user hits the same 18-min cliff.
Bench numbers (Mac, CPU only, first run incl. download):
Total wall-clock: 24 m 06 s
.groth16().run() in-process: 17 m 11 s
Gnark wrap step alone: 4 m 59 s
Constraints: 15,972,262
Verifier (in-process): 4.8 ms
Peak RSS: 2.89 GB
Proof bytes (SDK saved): 1,704
verified: ✓
Proposal's "~256 B proof" needs a footnote: that's the bare BN254
on-chain Groth16 proof. The SDK-saved artifact (with public values
+ VK reference for self-contained verification) is 1,704 B — and
that's the number the CBOR disclosure bundle stores. Either way the
core STARK → wrap compression is ~1,560× (2.65 MB → 1.7 KB).
Findings doc: appended "Appendix: Phase 1.2 Groth16 bench" with the
full numbers, proposal-vs-actual reconciliation, and the captured
gotcha as Issue gadievron#8.
Adds an apples-to-apples table at the top of the bench section covering all three execution modes the prover supports as of Phase 1.2: --execute, --prove --wrap=core, --prove --wrap=groth16. Same target, same crash witness, same Mac host. Headline tradeoff visible at-a-glance: core → groth16 is roughly 60× wall-clock penalty for ~1,560× proof-size reduction (2.65 MB → 1.7 KB). Plus a one-time 6.2 GB circuit-artifact download on the first groth16 run; the prover's try_install_circuit_artifacts() call makes this idempotent across reruns. Detailed Gnark-stage breakdown still lives in the appendix at the bottom of the doc — the new top-of-bench table is the at-a-glance summary readers want before they dive into stage timings.
Adds the Timestamp field of proposal §8's bundle schema. The bundle's disclosure now binds to a moment in Rekor's append-only log, giving verifiers a transparency-log timestamp without trusting any single notary. Phase 1.4a — packages/zkpox/anchor.py * ed25519 keypair generation, SubjectPublicKeyInfo PEM serialisation, sign() helper. * hashedrekord/0.0.1 POST to rekor.sigstore.dev (overridable via ZKPOX_REKOR_URL); response parsed into a typed Timestamp. * confirm_anchor_matches() GETs the entry back and asserts the recorded hash equals the locally-recomputed bundle hash. Phase 1.4b — packages/zkpox/bundle.py * New Timestamp dataclass (rekor_log_index, log_id, integrated_time, entry_uuid, inclusion_proof_root_hash, tree_size, hashes). * bundle_hash_pre_timestamp(bundle) → 32-byte canonical-CBOR sha256 of the bundle with timestamp=None. This is the hash Rekor binds: computing it pre-anchor lets the timestamp be attached post-hoc without invalidating the binding. Two invariants guarded by tests. * with_timestamp(bundle, ts) functional update for frozen Bundle. Phase 1.4c — core/zkpox/verifier/ * Reads the optional timestamp field; renders log index, log id, entry uuid, integrated time, tree size, path length. * Full Merkle inclusion proof + STH verification deferred to 1.4.x (lands alongside sp1-sdk wiring for STARK verify in 1.5). Phase 1.4d — Tests + findings doc * 9 new tests across test_anchor.py and test_bundle.py covering keypair flow, PEM round-trip, hashedrekord schema shape, env override, Timestamp round-trip, and the two pre-anchor-hash invariants. 27/29 pass, 2 skip pending RAPTOR_NET_TESTS=1. * Findings appendix with the anchor flow diagram, what's deferred to 1.4.x, and ergonomic/network cost notes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Makes the zkpox feature visible inside RAPTOR. The Rust workspace (Phase 1.1) and Python orchestration (1.1–1.4) now drive through two new slash commands plus a skill, a violation-gadget catalogue, and a new persona. Phase 1.5a — raptor_zkpox.py * Standalone driver, analogous to raptor_codeql.py / raptor_fuzzing.py. * `prove` subcommand: end-to-end orchestration — call zkpox-prove, optionally seal vendor envelope, assemble CBOR bundle, optionally anchor to Sigstore Rekor, persist bundle.cbor + proof.bin + prove-record.json to --out. * `verify` subcommand: delegate to the standalone zkpox-verify binary. Phase 1.5b — raptor.py * mode_prove_exploit / mode_verify_exploit_proof handlers added. * mode_handlers + mode_scripts + top-of-file docstring + _HELP_EPILOG all carry the new mode names so --help, help <mode>, and direct invocation all work consistently. * mode_prove_exploit uses _run_with_lifecycle (writes a real run directory); mode_verify_exploit_proof bypasses lifecycle (read-only). Phase 1.5c — slash commands * .claude/commands/prove-exploit.md and verify-exploit-proof.md. * Both registered in the harness's skill list (confirmed mid-session). Phase 1.5d — .claude/skills/zkpox/ * SKILL.md per proposal §11 sketch — when to use / when NOT to use / workflow / trust model / file outputs. * violation-gadgets/crash-only.md and memory-safety-oob-write.md — small catalogue users pick from for --gadget-id. Phase 1.5e — tiers/personas/disclosure_engineer.md * The 10th persona. CVD framework selection (ISO 29147, Project Zero, ZDI, EU CRA Art. 13), bundle composition decisions (--tlock-duration / --vendor-pubkey / --no-anchor / leaked_fields), vendor + public advisory templates, safe-harbor citations, dual-use checks. Phase 1.5f — CLAUDE.md + smoke test * COMMANDS section advertises the two new commands. * PROGRESSIVE LOADING references the zkpox skill + disclosure_engineer persona. * New ZKPOX section spelling out skill / persona / substrate paths. * Dispatch verified: \`python3 raptor.py prove-exploit --help\` → raptor_zkpox prove subparser; same for verify. 27/27 pytest still green; no regression in packages/zkpox/tests/. Deferred to Phase 1.5.x: * STARK verification in core/zkpox/verifier/ (wiring sp1-sdk). * Sigstore Rekor Merkle-inclusion verification. * Real \`proof.verifier_key_hash\` (currently a placeholder). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…dispatch Lands a concrete second target to back Phase 0's claim that the redzone+pattern primitive generalises across bug shapes. Same gadget infrastructure handles both; the only target-specific code is the C victim and a one-line dispatch entry. Phase 1.6a — target gadievron#2 + multi-target build * core/zkpox/targets/02-off-by-one.c — classic `i <= buf_size` bug. * core/zkpox/guest/build.rs now compiles a Vec<(src, obj)> targets list into one static lib (libzkpox_targets.a). Adding target N is one new line + a new C file. Phase 1.6b — guest dispatch + public-values schema bump * Witness layout becomes [target_id: u8] || raw_bytes. Guest peels the byte, dispatches to the right extern "C" victim, falls back to target 01 on any unknown / missing prefix (backward compat). * Public-values schema is now 5 fields, target_id committed first as u32. Existing 4 fields unchanged in order, so the bundle's STARK verifier just appends one read. Phase 1.6c — host --target flag * prover/src/main.rs gains --target {01,02} (default 01) and prepends the byte to the witness bytes before stdin.write(). * Verdicts struct + JSON record + packages/zkpox/prove.py Verdicts dataclass all surface target_id. Phase 1.6d — witness corpus for target 02 * 12 new 02-*.bin witnesses (5 benign / 6 crash / 1 canarymatch-fn). * generate.py's collision check is target-aware: for target 02 it only looks at byte[buf_size] (the off-by-one writes exactly one byte), so a wider sanity net catches a smaller failure mode. Phase 1.6e — regression harness * run-tests.sh parses target prefix from filename (\`01-*\`, \`02-*\`), passes --target, asserts target_id round-trip in addition to the verdicts. 25/25 pass across both targets. * packages/zkpox/tests/test_regression.py: pytest wrapper that subprocesses run-tests.sh. Gated behind RAPTOR_SLOW_TESTS=1 (~10 min wall — SP1 SDK startup x 25 invocations). Default pytest run is 27 passes + 3 skips + 0.6 s. Phase 1.6f — findings appendix * docs/research/zkpox-phase0-findings.md: per-target cycle table, side-by-side oob_count comparison (32-byte crash → target 01 reports 16, target 02 reports 1 — the off-by-one signature), schema-bump notes, soundness-probe results, what 1.6 deliberately didn't do. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Lands a slice of a real, public, low-severity CVE — the xmlSnprintfElementContent stale-`len` bug fixed in GNOME/libxml2 commit 932cc9896ab4 — to back the gadget-stability claim with a bug shape that wasn't designed around the redzone primitive. Phase 1.7a — target gadievron#3 source * core/zkpox/targets/03-libxml2-cve-2017-9047.c — freestanding extraction of the pre-patch XML_ELEMENT_CONTENT_ELEMENT branch. Inline z03_strlen / z03_strcat_n (no libc in SP1 freestanding). int-typed `size` and `len` mirror libxml2's upstream signature. Witness encoding: [prefix_len:u8][name_len:u8][prefix...][name...]. Phase 1.7b — per-target buffer geometry * guest/src/main.rs: BUF_SIZE_DEFAULT=16 (targets 01,02), BUF_SIZE_T03=32 (the bypass window of CVE-2017-9047 needs prefix_len > 8 to exceed the size-check's safety margin of 10 — doesn't fit a 16-byte buffer). Dispatch returns (victim, buf_size) tuple per target; LEADING and MIN_TRAILING redzones stay shared. * guest/build.rs: adds 03-libxml2-cve-2017-9047.c to the cross-compiled targets list. * prover/src/main.rs: TargetSel::T03 (clap name "03", id_byte 0x03). No public-values schema change — Phase 1.6's 5-field layout covers target 03 unchanged. Phase 1.7c — witness corpus (15 witnesses, 9 benign / 6 crash / 0 fn) * Benign sweep: empty, no-prefix, both-small, prefix-only, borderline (sum=30 exactly fills buf), bypass-fails on either side. * Crash sweep: prefix_len ∈ {9,10,15,22} with name_len=22 → overflow widths 1, 2, 7, 14 bytes respectively; plus 0xFF fill and zero-name-with-nonzero-prefix variants for byte-pattern coverage. * No -fn (canarymatch) witness: strcat-style overflows always end in a NUL byte, so an all-0xA5 OOB is structurally impossible. Documented in generate.py's _assert_not_canarymatch arm. Phase 1.7d — soundness probe surfaced during regression * 03-nulprefix-benign.bin: prefix bytes all 0x00. The first appended byte becomes the buffer's NUL terminator, so subsequent strcat sees strlen(buf)=0 and the stale-`len` lag never opens. Real property of CVE-2017-9047 in production libxml2 — bug requires non-NUL prefix. * 03-nullname-crash.bin: prefix=0x01*15, name=0x00*22. Bug fires with non-NUL prefix; OOB bytes are 0x00 (≠ 0xA5 canary, so detected). Phase 1.7e — regression harness * run-tests.sh accepts "03" target prefix in case arm. * packages/zkpox/tests/test_regression.py: asserts " t=3 " present, bumps min pass count 25 → 40. 40/40 execute-mode passes on this Mac. Phase 1.7f — findings appendix * docs/research/zkpox-phase0-findings.md: CVE selection rationale, bypass/overflow window math, buffer geometry override design, why -fn is structurally impossible for strcat overflows, oob_count identity table across the crash corpus, soundness probe write-up, what 1.7 deliberately didn't do. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The proposal RAPTOR's zkpox build-out has been implementing all along. Was sitting untracked locally; belongs in the tree alongside the findings doc and the implementation phases so reviewers and future maintainers have access to the original scope and design rationale. Covers: the pitch, why this belongs in RAPTOR, scope/roadmap (MVP through v0.4), architecture inside RAPTOR, command surface, violation gadget catalogue, disclosure bundle format, trust model, performance realism, SKILL/command sketches, implementation phases and effort estimate, open design questions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ages/zkpox
Adds a path-filtered CI job that builds the SP1 guest + host prover
and runs the full witness-corpus sweep in execute mode. Catches the
kind of breakage that surfaced during Phase 1.7 development — a
"crash" witness whose bytes interact with strcat semantics in a way
the bug doesn't fire on. Pure unit tests can't reach that.
* compute_filters.py: new `zkpox` filter, globs on
core/zkpox/**, packages/zkpox/**, and the workflow file itself.
* test_filter_coverage.py: new SUBSYSTEM entry pinned at
packages/zkpox. Import-coverage check passes trivially because
packages/zkpox has no external core.* / packages.* imports.
* tests.yml: new `zkpox-regression` job, gated on
`needs.changes.outputs.zkpox == 'true'` (or `force_full` on the
nightly cron / merge_group / workflow_dispatch tiers).
Job steps: checkout → rust toolchain (stable) → apt-get clang lld
llvm jq → cache ~/.sp1 + ~/.cargo/{registry,git} + core/zkpox/target
keyed by Cargo.lock hash → sp1up install → cargo build --release →
python3 generate.py → bash run-tests.sh.
Scope deliberately execute-mode only. `--prove --wrap=core` is
~17 s/witness × 40 ≈ 11 min just for proves; `--wrap=groth16` is
~17 min/witness plus a one-time 6.2 GB circuit-artifact download.
Either is wrong for a PR gate — they belong on the schedule/nightly
tier or workflow_dispatch. See docs/research/zkpox-phase0-findings.md
"Phase 1.2 Groth16 bench" appendix for the timing details.
Expected wall-clock:
* Cold cache: ~10-15 min (sp1up download + cold cargo build + 40
SP1 SDK startups in the sweep).
* Warm cache: ~3-5 min (mostly the 40 SP1 SDK startups, each
~5-10 s).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
sp1-prover-types v6.1.0's build.rs invokes `prost-build`, which needs protoc on PATH. Ubuntu-latest doesn't ship it, so the cargo build aborted in CI with "Could not find `protoc`". Adding `golang-go` at the same time — sp1-recursion-gnark-ffi pulls go transitively even in execute-only flows. Caught during the Phase 0 spike (findings doc "Issues discovered" row 1); ubuntu-latest doesn't ship a go toolchain either. Both go into the existing apt-get step rather than a new SP1 toolchain installer step, to keep the cold-cache build path as short as possible. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`envelope_ct_K_age_len` / `envelope_ct_K_tlock_len` violated rustc's default `non_snake_case` lint and showed up as two warnings on every build (including CI). Renamed to lowercase k. The wire-format CBOR keys `"ct_K_age"` / `"ct_K_tlock"` (lines 125-126) are deliberately preserved — those are bundle-format identifiers consumed across the Python (packages/zkpox/bundle.py) and Rust boundaries, and uppercase K matches the cryptographic convention "ciphertext of K under X". Only the Rust struct field names (which become JSON keys in `zkpox-verify --json` output) are changed; no downstream consumer keys off those names. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
sp1-build v6.1.0 prints `cargo:warning=rustc +succinct --version: ...` and `cargo:warning=<package> built at <timestamp>` on every build. Cargo renders both as yellow "warning:" lines, but neither is actionable — they're informational notices the upstream maintainers chose `cargo:warning=` for because it's the only build-script channel cargo surfaces to the user. No env var disables them. Filter at the CI invocation with a narrow awk regex that strips only those two exact signatures. Real cargo warnings (including any future zkpox-prover@VERSION warning that isn't one of these two) still pass through. Verified locally: printf '...' | awk '!/<pattern>/' drops the two noise lines and preserves both standard rustc warnings and unrelated build-script `cargo:warning=` outputs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous job was running long enough to hit the 25-min timeout
on cold-cache runs. Two cuts:
1. Build only `zkpox-prove` (the binary the sweep actually needs),
not the whole workspace. Skips the `zkpox-verify` link step. The
`python-unit-tests-fast` tier still covers `packages/zkpox/`'s
Python verifier wrapper, so this isn't a coverage gap — just
moves zkpox-verify's compile-error catching to a future
first-class regression job if it ever needs one.
2. New --ci-subset flag in run-tests.sh that runs 8 representative
witnesses instead of all 40. Each SP1 SDK startup is ~20-25 s in
CI, so cutting 32 witnesses saves ~10-13 min of wall-clock. The
8 picked still cover, end-to-end:
* each target's basic crash detection (T01 BOF, T02 off-by-one,
T03 CVE-2017-9047)
* the uniform-canary blind spot covered by the position-varying
gadget (T01, T02 -canarymatch-fn)
* T03's "bug inert without prefix" property
* T03's "bug requires non-NUL prefix" soundness probe (the
finding that surfaced during Phase 1.7 dev)
* T03's byte-pattern coverage via zero-name + non-NUL prefix
The full 40-witness corpus runs on the schedule /
workflow_dispatch / merge_group tier (where `force_full=true`),
gated by an `if` in the workflow that dispatches to either
`run-tests.sh` or `run-tests.sh --ci-subset`.
Coverage rationale lives inline in core/zkpox/test/run-tests.sh
(one comment line per witness) so future operators editing the
subset have the reasoning right next to the list.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…you trust
A non-cryptographer security reviewer asked for "one paragraph that
says explicitly: MVP gadget = memory-safety violation predicate; full
exploit-impact gadgets are in proposal §7 but not in this PR." That
paragraph turned into a one-page scope doc covering:
* What this MVP proves (memory-safety violation predicate; the five
public values committed by the SP1 guest; what a verifier learns).
* What this MVP does NOT prove (control-flow hijack, RCE, info-leak
with attacker-chosen target, exploit reliability, vulnerability
class label).
* What you trust when you trust a bundle (SP1, the compiled guest,
age, Drand, Sigstore Rekor, local clock at anchor time) — each
with current-MVP caveats spelled out.
* Failure modes (vendor loses age sk, Drand unavailable, vendor
patches early, vendor refuses to patch, sp1-sdk soundness bug,
producer lies about target_id in plaintext metadata).
* The redzone gadget's FP/FN profile — uniform 0xA5 vs
position-varying pattern, with explicit `(1/256)^N` for the
latter and a note that adversarial-witness defeat is what the
Phase 1.x shadow-allocation work eliminates.
* Roadmap for the next vulnerability classes by name.
* How to read a bundle as a non-cryptographer, including the
`--strict` flag's role.
The doc is the single source of truth for the scope claim. README
and the prove-exploit / verify-exploit-proof command docs are
updated to link to it as the first thing a reviewer should read.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ct, version binding
Five small but load-bearing fixes that turn "looks verifiable but
binds to nothing" footguns into honest beta tooling. Each addresses
a concrete concern from the non-cryptographer review.
1. raptor_zkpox.py refuses to write a bundle without the explicit
--allow-placeholder-hashes opt-in. Phase 1.5 produces placeholder
verifier_key_hash and harness.hash that do NOT bind to the real
SP1 verifying key or the harness binary — silently shipping that
in a "verifiable" bundle was the worst dishonesty in the MVP.
Opt-in via flag, with a loud warning emitted on every prove and
referenced in docs/zkpox-scope.md.
2. Vulnerability.gadget_hash → gadget_id_hash (bundle schema +
verifier). The field hashes the gadget identifier string (e.g.
"memory-safety::oob-write@0.1.0"), not the gadget source code,
so the original name was misleading. Phase 1.5.x will add a
separate gadget_code_hash over the implementation files.
3. zkpox-verify --strict flag. Exits non-zero on any DEFERRED check
(currently both STARK proof verification and Rekor Merkle
inclusion). Default off in 1.5 so operators can inspect bundles
produced today; becomes the default in 1.5.x. Loud "MODE:
structural-only" banner in non-strict, non-JSON mode so a casual
reader doesn't mistake exit 0 for "the bundle is verified."
4. anchor.py raises on missing integratedTime instead of falling
back to time.time(). Substituting the local clock would make the
bundle's recorded timestamp disagree with what's actually in
Rekor's log, breaking the verification path. Fail loud, not
silent.
5. Verifier checks vendor_envelope.scheme starts with the supported
v1 prefix (or the no-envelope sentinel). Without this a future
v2 envelope could be replayed under v1 framing — the AAD
mismatch would surface as a confusing AES-GCM authentication
failure only on decrypt, far too late for the structural pass to
catch.
Plus polish:
* Stale "build it with: (cd harness/host && ...)" message in
run-tests.sh replaced with the canonical
`cargo build --release --manifest-path core/zkpox/Cargo.toml`.
* prove-exploit.md and verify-exploit-proof.md updated for the new
flags + the docs/zkpox-scope.md reference.
Local validation:
* `cargo build --release` clean (workspace; verifier rebuilt).
* 27 passed / 2 skipped across packages/zkpox/tests/.
* `run-tests.sh --ci-subset` 8/8 pass — the schema rename hasn't
broken the SP1 wire format.
* Filter coverage test passes; gadget_id_hash CBOR encode/decode
round-trips byte-exact.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
Code-side fixes (
|
Post-merge of main into zkpox-dev, test_filter_coverage caught that
packages/codeql/ now imports core.dataflow.{evidence_renderer,
finding, llm_bridge, sanitizer_evidence, validator}. These modules
landed in main via the cocci-hunt / iris SMT work; the codeql filter
in compute_filters.py wasn't updated alongside.
Without this glob, a PR touching core/dataflow/ would not retrigger
the python-unit-tests-codeql job — exactly the indirect-breakage
scenario test_filter_coverage exists to catch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Collaborator
There was a problem hiding this comment.
Three items worth addressing before merge, just keeping in with how the safety angle of RAPTOR works:
1. sys.path safety rule
raptor_zkpox.py does:
sys.path.insert(0, str(Path(__file__).resolve().parent))The established RAPTOR pattern (enforced in #312) is:
sys.path.insert(0, os.environ["RAPTOR_DIR"])The hard KeyError on an unset RAPTOR_DIR is intentional, it fails loudly rather than silently falling back to a cwd that might not be the repo root. In practice this script lives at the repo root so the behaviour is equivalent, but it's inconsistent with every other entry point and will silently do the wrong thing if the script is ever symlinked or invoked from a different working directory. Easy one-line fix. shows we care and take this seriously
2. No explicit get_safe_env() in nested subprocess calls
packages/zkpox/prove.py (subprocess.run(cmd, ...)) and the age/tle calls in packages/zkpox/envelope.py don't pass env=RaptorConfig.get_safe_env(). Functionally this is fine raptor.py launches raptor_zkpox.py via _run_with_lifecycle which already calls get_safe_env() at line 485, so the nested processes inherit a clean environment. Buuuut the RAPTOR convention is to be explicit at each subprocess boundary so a future caller that invokes these functions directly (tests, integration scripts) can't accidentally pass a dirty env through. A comment at each subprocess.run() call noting the inherited-sanitised-env assumption would also serve as documentation.
3. Placeholder hashes worth a CHANGELOG note
The verifier_key_hash and harness.hash fields in Phase 1.5 bundles are explicitly placeholders (they hash the string "placeholder-vk-1.5" rather than the real SP1 verifying key). This is correctly gated behind --allow-placeholder-hashes with loud warnings, and documented in docs/zkpox-scope.md. No code change needed here, just flagging it so reviewers are aware that bundles produced with this merge are not suitable for real disclosure until Phase 1.5.x. Worth a note in the CHANGELOG or PR description to make that explicit for anyone who merges and tries to use it immediately.
None of these block the merge, which is solid. the isolation is clean and existing functionality is untouched. Items 1 and 2 are worth a follow-up commit before the branch lands if Mark has bandwidth. Thank you @unprovable for this, very sexy time
Collaborator
|
I'm leaning more towards a Single PR if im honest |
Contributor
Author
|
Ok, to be explicit here in the PR: Bundles produced by this branch contain placeholder verifier_key_hash and harness.hash fields — they hash the literal strings "placeholder-vk-1.5" and "harness-1.5", not the real SP1 verifying key or harness binary.
NB - See
Do we want to add a changelog file formally? Else, see edit above. |
…y spawn Review follow-up before merge of gadievron#470. raptor_zkpox.py: - sys.path.insert(0, os.environ["RAPTOR_DIR"]) per the project rule (CLAUDE.md / gadievron#312 lineage). Hard KeyError on unset is intentional — fail loudly rather than silently falling back to a cwd or __file__-relative walk that breaks under symlinks / non-repo cwd. - subprocess.run() calls in cmd_prove / cmd_verify now pass env=RaptorConfig.get_safe_env() explicitly. packages/zkpox/prove.py, packages/zkpox/envelope.py: - All subprocess.run sites (zkpox-prove, age-keygen, age, tle) now pass env=RaptorConfig.get_safe_env() explicitly. raptor.py's _run_with_lifecycle already sanitises the parent env, so this is belt-and-braces today, but the convention is "explicit at every spawn site" so a future direct caller (tests, integration scripts) cannot accidentally leak a parent's dirty env through these helpers. .github/scripts/compute_filters.py: - zkpox filter now covers core/config.py + core/config/** to track the new RaptorConfig import in packages/zkpox/. Validated by .github/tests/test_filter_coverage.py. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
|
OK, I think this starts to answer your review - thank you so much, @danielcuthbert . Also incoming in the hour should be a warning banner that says "this is still experimental". Experimental but this is the bleeding edge of disclosure, I think. Thanks for the engagement, too! :D |
Two layers, so an operator who skipped the slash-command help can't
miss the beta status:
raptor_zkpox.py:
- cmd_prove and cmd_verify each emit a 6-line stderr banner at the
top: "EXPERIMENTAL (beta) — Phase 1.5. Subject to change ... NOT
for real CVE disclosure. Scope: docs/zkpox-scope.md". Fires
unconditionally — distinct from the placeholder-hash warning,
which only fires when --allow-placeholder-hashes flips the gate.
Banner goes to stderr so stdout JSON stays clean.
packages/zkpox/bundle.py:
- Bundle gains `experimental: bool = True`. Always serialised at top
level of the CBOR. _replace_timestamp preserves it across the
timestamp-mutation path.
- _from_dict defaults to True on absent — a bundle without the field
is treated as experimental rather than silently dropping the
marker. Safer to false-warn than to false-clear once a future
post-stable verifier looks at an old bundle.
core/zkpox/verifier/src/main.rs:
- Summary gains an `experimental: bool` field, serialised as
`"experimental": true` in JSON mode so machine consumers can
branch on it.
- Human-readable summary line gains a "[EXPERIMENTAL]" tag.
- Non-JSON output adds a 4-line stderr banner ("EXPERIMENTAL
BUNDLE — produced by Phase 1.5 (beta) zkpox..."), fires whenever
experimental == true. Always-on (even with --strict) — strict is
about deferred crypto checks, beta-status is orthogonal.
- optional_bool_at helper; mirrors the producer's default-true-on-
absent rule so verifier reads of an old bundle still warn.
packages/zkpox/tests/test_bundle.py:
- Four new round-trip tests: default-construct, true round-trip,
false round-trip (for future Phase 1.6+ flip), absent-field
defaults to True.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…/schedule
zkpox-regression was the long pole on PR CI at ~16.5 min wall-clock
(sweep 11 min + apt 2 min + cargo build 2 min) while every other job
finished in ~3 min — a 5× drag on every zkpox-touching PR.
Skip it on `pull_request` events. Post-merge coverage stays intact:
* merge_group — runs before the PR actually lands
* push to main — belt-and-braces after merge (zkpox-gated)
* schedule — nightly full 40-witness corpus
* workflow_dispatch — author can trigger manually for a PR
Author-side fast loop unchanged:
cargo build --release --manifest-path core/zkpox/Cargo.toml \\
&& bash core/zkpox/test/run-tests.sh --ci-subset
~5 min on a warm local toolchain — equivalent to what CI ran on PR-tier
Contributor
Author
|
ok, also added a fix to make the tests faster as ~16"30s per run is way way way too long! |
Contributor
Author
Not planning on making more changes without a real pipeline to run it off :P so sounds like it's good to keep alive for when |
The SCA PR gate failed on a new high-severity regression: the zkpox dev deps introduced cryptography==45.0.4, whose PyPI wheels statically link a vulnerable OpenSSL (GHSA-537c-gmf6-5ccf, CVSS 7.5 — out-of-bounds read → DoS). Fixed upstream in 48.0.1. Bump the active pin in requirements-dev.txt and the commented mirror in requirements.txt. cbor2 unchanged.
Contributor
Author
|
This bug is hilarious... given we use pycrypto in this PR, updated accordingly: GHSA-537c-gmf6-5ccf |
grokjc
added a commit
that referenced
this pull request
Jun 25, 2026
) Workspace members (guest/, prover/, verifier/) don't carry their own Cargo.lock — only the workspace root does. Discovery found each Cargo.toml independently, so each member spawned a standalone `cargo update` copying only its own manifest without the workspace root — wrong dep graph and wasted CI time (3 redundant sandboxed cargo invocations on PR #470). Walk parent dirs for a [workspace] Cargo.toml with a sibling Cargo.lock; drop member entries whose root lockfile is already parsed by the pipeline.
Collaborator
|
Issues seem to be vulnerable quinn-proto and p3-challenger packages. |
Collaborator
|
Have dug into this further - the SCA gate is failing on one fixable high-severity finding:
There are also 12 new medium-severity Cargo advisories introduced by this PR. Most are structural - transitive deps of the SP1 prover SDK (
The rest have no patched version and are waiting on SP1 to update their deps. None of these block the gate (threshold is high severity). |
Resolves the SCA PR-gate high-severity regression on PR gadievron#470. quinn-proto <0.11.15 is vulnerable to remote memory exhaustion via unbounded out-of-order stream reassembly (CVSS 7.5). Semver-compatible patch bump; pulled in transitively via reqwest→quinn. cargo update also reconciled stale zkpox-prover/zkpox-verify lock entries with their manifests (lock-only, no manifest changes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
|
7ca15d7 will fix your tests. |
Collaborator
|
SCA CI is highlighting:
d4cee71 is your friend. |
pip 26.1.1 carries GHSA-wf93-45jw-7689 (path traversal in console_scripts/gui_scripts entry-point names). The SCA workflows pinned and installed that exact version, so the PR gate flagged a vulnerability its own setup introduced. 26.1.2 is the fixed release (verified clean via OSV). Applied to all 9 workflows sharing the pin. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…HSAs pnpm 11.3.0 is affected by 15 advisories (9 high + 6 moderate), detected by the SCA inline-installs parser from the `npm install -g` line. 11.12.0 (current latest) clears the whole cluster — highest required fix on the 11.x line is 11.8.0; verified 0 advisories against 11.12.0 via OSV. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Soryr for the delay - think this is done now, @grokjc |
Collaborator
|
689a453 needed ... (no rush!) |
# Conflicts: # raptor.py
Resolve conflicts from main's CI test-dispatch refactor (static glob lists
in compute_filters.py → import-graph analysis in test_scope.py), which
collided with the zkpox regression job that was wired into the old mechanism.
- .github/scripts/compute_filters.py: take main's codeql-only version;
subsystem dispatch (incl. the zkpox filter) moved out to test_scope.py.
- .github/tests/test_filter_coverage.py: accept main's deletion; the
filter-coverage concept is obsolete under import-graph dispatch.
- .github/workflows/tests.yml: take main's scope-based changes-job outputs,
re-adding the `zkpox` gate and `force_full` output that the
zkpox-regression job depends on.
- .github/scripts/test_scope.py: add a `zkpox` tier (outside_graph, triggers
on core/zkpox + packages/zkpox) so steps.scope.outputs.zkpox is emitted —
without it the regression job's gate would be empty and never fire.
- .github/tests/test_test_scope.py: add zkpox dispatch tests to replace the
coverage lost with test_filter_coverage.py.
Contributor
Author
|
Done! Apologies for the delay! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Much of the hype around AI Agents finding bugs stems from the fact that scaled vulnerability disclosure and verification is hard. Part of this is the necessity to share deep technical details in a way that might risk exposure of a critical flaw - something seen as 'too hard' and 'risky'.
Adding Zero-Knowledge Proofs of eXploit (ZKPoX) allows RAPTOR to find an exploit and prove publicly that the exploit is valid without disclosing the exploit except to those who may be charged with the fix. Allowing such functionality permits the risk to be checked and validated, without a working and potentially dangerous exploit being made public without a fix being in place and available.
Whilst this may be controversial, it is certainly experimental and is included here as a proof of concept for reviewer deliberation (@grokjc @danielcuthbert @thomasdullien ) over providing a full assured zero knowledge disclosure pipeline. Google recently improved quantum factoring and hid the algorithm in a ZKP in a similar way to show that it was a) done work, and b) not ready for disclosure: https://research.google/blog/safeguarding-cryptocurrency-by-disclosing-quantum-vulnerabilities-responsibly/
Responsible disclosure is changing, and agents that speak the math are probably the right place to trial these methods. Especially those that might, like raptor, find issues at scale. Enclosed is an example, from CVE-2017-9047 (see below) that gives a demonstration of the workflow proposed.
What this is
Lands the ZKPoX MVP — zero-knowledge proof-of-exploit. Convert a working exploit witness into a publicly-verifiable disclosure bundle: the witness stays private, the proof is public, the vendor gets full details immediately via an age-encrypted envelope, and the public-witness-decrypt time is enforced cryptographically by a Drand time-lock. Anchored in Sigstore Rekor for tamper-evident timestamping.
Design proposal lives in this PR (
docs/proposals/raptor-zkpox-design.md), spec → implementation has been a single-author build-out over a few weeks; commits track the phase plan from the proposal's §13.What's in here
Rust workspace under
core/zkpox/:i <= buf_size)xmlSnprintfElementContentstale-len(real, public, low-CVSS bug, fixed in commit932cc989)zkpox-prove) —--execute/--provewith--wrap={core,groth16}, target dispatch, JSON bench recordszkpox-verify) — standalone CBOR-bundle verificationcrash-only(uniform canary) +memory-safety::oob-write(position-varying pattern with structured count/offset outputs)Python orchestration under
packages/zkpox/:prove.run()wrapper around the host proverbundle.py), age vendor envelope + Drand tlock (envelope.py), Rekor anchoring (anchor.py)RAPTOR_SLOW_TESTS=1)Command surface:
/prove-exploit— produce a disclosure bundle/verify-exploit-proof— verify one.claude/skills/zkpox/SKILL.md+ violation-gadget skill filestiers/personas/disclosure_engineer.mdTests:
run-tests.shregression harness — 40/40 pass execute-mode on Macpackages/zkpox/tests/Python wrapper + slow integration testzkpox-regression— path-filtered oncore/zkpox/**/packages/zkpox/**; builds the SP1 cross-compile pipeline and runs the execute-mode sweep on every relevant PRBenchmarks (Apple Silicon, single witness, target #1)
--execute--prove --wrap=core--prove --wrap=groth16The Groth16 wrap pays a 60× wall-clock penalty for a 1,560× proof-size reduction — the right shape for a CBOR disclosure bundle. Full breakdown in
docs/research/zkpox-phase0-findings.md.Test plan
python3 -m pytest packages/zkpox/tests/ -v— fast unit tests pass, regression test correctly skips withoutRAPTOR_SLOW_TESTS=1cargo build --release --manifest-path core/zkpox/Cargo.toml(needs SP1 toolchain:curl -L https://sp1up.succinct.xyz | bash)bash core/zkpox/test/run-tests.sh— 40/40 passzkpox-regressionruns on this PR and passesdocs/proposals/raptor-zkpox-design.md(§1–3 for the pitch, §7 for the gadget catalogue, §8 for the bundle format)docs/research/zkpox-phase0-findings.mdfor empirical numbers and recorded architectural decisionsScope: what's NOT in this PR
Explicitly deferred per the design doc's §4 roadmap:
--wrap=corewould add ~11 min/run and--wrap=groth16would add hours + 6.2 GB artifact download. Both belong on a nightly / dispatch tier.Caveats
Bundles produced by this branch contain placeholder verifier_key_hash and harness.hash fields — they hash the literal strings "placeholder-vk-1.5" and "harness-1.5", not the real SP1 verifying key or harness binary.
NB - See
docs/zkpox-scope.mdfor the full statement of what this MVP proves and doesn't prove.Open questions worth reviewer input
References
docs/proposals/raptor-zkpox-design.mddocs/research/zkpox-phase0-findings.md932cc989: https://gitlab.gnome.org/GNOME/libxml2/-/commit/932cc9896ab41475d4aa429c27d9afd175959d74