Circom auditor

PLAN.md

@@ -0,0 +1,229 @@
+# LLM Auditor Run Plan — circom zkbugs
+
+Covers both `circom_auditor_claude` and `circom_auditor_codex` across all 70
+runnable circom bugs in `config.toml`, using `--zkbugs-mode both` (direct +
+original entrypoints). Bugs are split into 6 shards to stay under API rate
+limits.
+
+**Why 70 and not 72?** `config.toml` tracks 72 circom bugs total: 70 in
+`[circom].bugs` (runnable — these are analyzed here) and 2 in
+`[circom].unreproducible_bugs` (not present in the dataset at all:
+`ProcessMessages` and `Potentially_Easy_to_Misuse_Interface`).
+
+---
+
+## Prerequisites
+
+Verify each item before starting.
+
+```bash
+# 1. Verify the zkbugs dataset is present next to this repo
+ls ../zkbugs/dataset/circom   # expected: org directories (0xbok, iden3, ...)
+
+# 2. (Optional) Download original codebases for original-mode runs
+bash ../zkbugs/scripts/download_sources.sh
+
+# 3. Claude Code CLI
+claude --version
+echo $CLAUDE_PLUGIN_DIR         # must be set
+ls $CLAUDE_PLUGIN_DIR/skills/circom-auditor/SKILL.md   # must exist
+
+# 4. OpenAI Codex CLI
+codex --version
+echo $OPENAI_API_KEY            # must be set
+# CODEX_PLUGIN_DIR defaults to CLAUDE_PLUGIN_DIR if unset (same skill dir works)
+
+# 5. Optional: increase include-closure cap for large circuits (default 5000)
+# export CIRCOM_AUDITOR_MAX_LINES=8000
+```
+
+---
+
+## Step 1 — Split bugs into shards
+
+Run once. Produces `shards/shard_{1..6}.txt`, each with ~11 bugs.
+
+```bash
+python scripts/split_bugs.py --shards 6 --output-dir shards/
+```
+
+---
+
+## Step 2 — Run each shard
+
+Run shards **sequentially** to respect rate limits (each shard takes ~2-3 h).
+Adjacent shards (1+4, 2+5, 3+6) can run concurrently in separate terminals —
+they have no shared state.
+
+Each command runs both tools on both modes (direct + original) and writes
+results under `output/shard_N/`.
+
+```bash
+# Shard 1
+uv run python -m zkhydra.main zkbugs \
+  --dataset ../zkbugs/dataset/circom \
+  --tools circom_auditor_claude,circom_auditor_codex \
+  --bugs-file shards/shard_1.txt \
+  --zkbugs-mode both \
+  --output output/shard_1
+
+# Shard 2
+uv run python -m zkhydra.main zkbugs \
+  --dataset ../zkbugs/dataset/circom \
+  --tools circom_auditor_claude,circom_auditor_codex \
+  --bugs-file shards/shard_2.txt \
+  --zkbugs-mode both \
+  --output output/shard_2
+
+# Shard 3
+uv run python -m zkhydra.main zkbugs \
+  --dataset ../zkbugs/dataset/circom \
+  --tools circom_auditor_claude,circom_auditor_codex \
+  --bugs-file shards/shard_3.txt \
+  --zkbugs-mode both \
+  --output output/shard_3
+
+# Shard 4
+uv run python -m zkhydra.main zkbugs \
+  --dataset ../zkbugs/dataset/circom \
+  --tools circom_auditor_claude,circom_auditor_codex \
+  --bugs-file shards/shard_4.txt \
+  --zkbugs-mode both \
+  --output output/shard_4
+
+# Shard 5
+uv run python -m zkhydra.main zkbugs \
+  --dataset ../zkbugs/dataset/circom \
+  --tools circom_auditor_claude,circom_auditor_codex \
+  --bugs-file shards/shard_5.txt \
+  --zkbugs-mode both \
+  --output output/shard_5
+
+# Shard 6
+uv run python -m zkhydra.main zkbugs \
+  --dataset ../zkbugs/dataset/circom \
+  --tools circom_auditor_claude,circom_auditor_codex \
+  --bugs-file shards/shard_6.txt \
+  --zkbugs-mode both \
+  --output output/shard_6
+```
+
+**Resuming a failed shard**: re-run the same command. Bugs that already have
+`results.json` are not re-executed (pass `--vanilla` to just re-parse existing
+raw output without re-running tools).
+
+---
+
+## Step 3 — Merge shards into one combined run
+
+Stitches the six shard output dirs into a single `output/llm_combined/` that
+mirrors a normal `--zkbugs-mode both` run structure (`direct/` + `original/`).
+
+```bash
+python scripts/merge_shards.py \
+  output/shard_1 output/shard_2 output/shard_3 \
+  output/shard_4 output/shard_5 output/shard_6 \
+  --output output/llm_combined
+```
+
+Expected layout after merge:
+
+```
+output/llm_combined/
+  summary.json
+  direct/
+    summary.json
+    <bug_name>/
+      ground_truth.json
+      circom_auditor_claude/  raw.txt  parsed.json  results.json  evaluation.json
+      circom_auditor_codex/   ...
+  original/
+    summary.json
+    <bug_name>/   # only bugs with a distinct Original Entrypoint
+      ...
+```
+
+---
+
+## Step 4 — Triage Undecided verdicts
+
+Claude automatically resolves verdicts that were marked Undecided by the
+auto-evaluator (e.g. class match but different line number). Rewrites
+`evaluation.json` in place; preserves the original at `evaluation.original.json`.
+
+Run for both modes:
+
+```bash
+python scripts/triage_zkbugs_run.py output/llm_combined/direct \
+  --dataset ../zkbugs/dataset/circom \
+  --auto --update-evaluation --jobs 4
+
+# Only if output/llm_combined/original/ exists and is non-empty:
+python scripts/triage_zkbugs_run.py output/llm_combined/original \
+  --dataset ../zkbugs/dataset/circom \
+  --auto --update-evaluation --jobs 4
+```
+
+Review the triage summary:
+
+```bash
+cat output/llm_combined/direct/triage_summary.json
+```
+
+---
+
+## Step 5 — Merge into the remote zkbugs run
+
+Copies the two LLM tool result dirs into the existing multi-tool remote run.
+Run all four commands (two tools × two modes):
+
+```bash
+# direct mode
+python scripts/merge_tool_run.py \
+  --source output/llm_combined/direct \
+  --target output/zkbugs-remote/direct \
+  --tool circom_auditor_claude
+
+python scripts/merge_tool_run.py \
+  --source output/llm_combined/direct \
+  --target output/zkbugs-remote/direct \
+  --tool circom_auditor_codex
+
+# original mode (skip if output/llm_combined/original/ is empty)
+python scripts/merge_tool_run.py \
+  --source output/llm_combined/original \
+  --target output/zkbugs-remote/original \
+  --tool circom_auditor_claude
+
+python scripts/merge_tool_run.py \
+  --source output/llm_combined/original \
+  --target output/zkbugs-remote/original \
+  --tool circom_auditor_codex
+```
+
+---
+
+## Step 6 — Generate summary tables
+
+```bash
+# Summary table + bug-tool matrix for the combined LLM run:
+python scripts/process_zkbugs_results.py output/llm_combined/direct
+
+# Full remote run (all tools):
+python scripts/process_zkbugs_results.py output/zkbugs-remote/direct
+
+# Optional: LaTeX/PDF report:
+python scripts/process_zkbugs_results.py output/zkbugs-remote/direct \
+  --latex output/zkbugs-remote/report.pdf
+```
+
+---
+
+## Summary of output directories
+
+| Path | Contents |
+|------|----------|
+| `shards/shard_N.txt` | Bug selectors for shard N |
+| `output/shard_N/` | Raw shard run (direct/ + original/) |
+| `output/llm_combined/` | Merged LLM run (all 70 bugs) |
+| `output/zkbugs-remote/` | Full multi-tool run (all tools including LLM) |

README.md

@@ -237,6 +237,90 @@ uv run python -m zkhydra.main zkbugs \
 - **Picus** - Symbolic execution via Rosette
 - **EcneProject** - Julia-based circuit analysis
 - **zkFuzz** - Fuzzing-based bug detection
+- **circom_auditor_claude / circom_auditor_codex** - 17-agent parallel LLM audit via [zksecurity/zk-skills](https://github.com/zksecurity/zk-skills) — **native-only, not bundled in Docker**; ~3-10 min per circuit
+
+### circom-auditor — native-only
+
+> ⚠️ **`circom_auditor_claude` and `circom_auditor_codex` only run in native mode.** They are *not* installed in the zkhydra Docker image. If you try to invoke them inside `docker-compose run`, the tool plugin will exit with an error pointing back here.
+
+`circom_auditor_claude` invokes the [zksecurity/zk-skills](https://github.com/zksecurity/zk-skills) `circom-auditor` Claude Code skill. `circom_auditor_codex` invokes the same skill through Codex's native skill discovery (`.agents/skills`, user skills, admin skills, or installed plugins). The Codex wrapper prebuilds the delegated worker bundles and forbids local fallback: if Codex subagents are unavailable, the run fails instead of doing a single-agent audit. The skill spawns specialist sub-agents in parallel and produces a deduplicated, gate-validated security report.
+
+The reason for native-only: LLM CLIs may store subscription/OAuth credentials in the host OS keychain (macOS Keychain / libsecret on Linux / DPAPI on Windows). Those credentials cannot be mounted into a Linux container, so a containerised version would force everyone onto API-key billing. By keeping these tools native, you can use the host CLI auth already configured for Claude or Codex.
+
+#### One-time host setup
+
+```bash
+# 1. Install Claude Code CLI
+npm install -g @anthropic-ai/claude-code
+
+# 2. Authenticate — pick one
+claude login                       # subscription, opens browser, stores in OS keychain
+# OR
+export ANTHROPIC_API_KEY=sk-ant-... # API key, pay per token
+
+# 3. Install zk-skills (the repo doubles as a plugin dir thanks to its
+#    committed `skills/circom-auditor` symlink — no extra scaffolding needed)
+git clone https://github.com/zksecurity/zk-skills.git ~/zk-skills
+
+# 4. Tell zkhydra where the plugin dir is (add this to your shell rc)
+export CLAUDE_PLUGIN_DIR=~/zk-skills
+
+# Optional Codex variant
+npm install -g @openai/codex
+codex login                      # or export CODEX_API_KEY=...
+mkdir -p ~/.agents/skills
+ln -s ~/zk-skills/skills/circom-auditor ~/.agents/skills/circom-auditor
+```
+
+#### Run it
+
+From a checkout of zkhydra, **without Docker**:
+
+```bash
+uv run python -m zkhydra.main analyze \
+  --input examples/test_bug/circuits/circuit.circom \
+  --tools circom_auditor_claude \
+  --timeout 600
+```
+
+#### Mixed sweep — fast static tools in Docker, LLM auditor natively
+
+```bash
+# 1. Static tools in the container (fast, deterministic, no auth)
+docker-compose run --rm zkhydra uv run python -m zkhydra.main zkbugs \
+  --dataset zkbugs/dataset/circom \
+  --tools circomspect,circom_civer,picus,zkfuzz \
+  --bugs daira_hopwood_darkforest_v0_3_missing_bit_length_check \
+  --output output/static-only
+
+# 2. LLM auditor natively, against the same bug
+uv run python -m zkhydra.main zkbugs \
+  --dataset zkbugs/dataset/circom \
+  --tools circom_auditor_claude \
+  --bugs daira_hopwood_darkforest_v0_3_missing_bit_length_check \
+  --output output/llm-only
+
+# 3. Merge per-bug findings.json files manually if you want a unified view
+```
+
+#### Caveats
+
+- Each `circom_auditor_claude` / `circom_auditor_codex` run spawns up to 17 parallel specialist sub-agents and takes 3-10 minutes wall-clock on a small bundle (1-5 templates / a few hundred lines). On larger scopes the wall-clock grows non-linearly — the sub-agents each have to ingest the full bundle before producing findings.
+- **Use a per-bug timeout of `1800` (30 min), not 24h.** A bug that doesn't finish in 30 min is hung — fail it and move on. The skill is designed for the 2-5 templates a developer is actively touching, not monorepo-sized audits.
+- **Bundle-size guard:** the tool plugin refuses to launch on scratch dirs above 30 `.circom` files or 5 000 lines of source (override via `CIRCOM_AUDITOR_MAX_FILES` / `CIRCOM_AUDITOR_MAX_LINES` env vars). zkbugs reproducers from large monorepos like Panther transitively pull in ~200 files / ~50K lines via `-l` link flags; those will be skipped with a clear "bundle too large" failure rather than hanging the run.
+- Cost: subscription quota / account quota or per-token API spend, depending on the CLI auth you use. Pair with `--tools` and `--bugs` filtering to avoid running it on every bug in a large dataset sweep unless that's what you want.
+- The skill follows Circom `include` chains (so wrapper-only zkbugs reproducers see the actual buggy template body via the `-l` link flag → scratch-dir trick the tool plugin handles automatically).
+- Output is rich markdown — the tool plugin parses the `## Findings` and `## Leads` sections into zkhydra's standardized `Finding` schema.
+
+#### Eval-mode sandboxing (zkbugs honesty guarantees)
+
+When you point `circom_auditor_claude` or `circom_auditor_codex` at a zkbugs reproducer, the bug folder ships sidecar files that contain the literal answer key — `README.md` lists the vulnerability class, root cause, location, and proposed mitigation; `zkbugs_config.json` carries the same structured data. To prevent the LLM from "auditing" by reading the answer, the tool plugin runs every audit inside a fresh tmp dir and keeps the answer key out of it. Three layers of defence:
+
+1. **Filesystem isolation** — the scratch dir contains *only* `.circom` source: the wrapper, any sibling `.circom` files at the top level of the bug dir, and symlinks to the linked codebase's source subdirectories. Excluded by name: `README*`, `zkbugs_config.json`, `zkbugs_*.sh`, `input.json`, `direct_input.json`. Excluded by directory blocklist: `test`, `tests`, `doc`, `docs`, `client`, `examples`, `node_modules`, hidden dirs, and a few other common project-noise names that could leak per-bug hints.
+2. **CLI restrictions** — the Claude variant blocks web/search tools and skips the user/project/local Claude settings stack; the Codex variant prebuilds delegated bundles, runs `codex exec` in read-only sandbox mode from the scratch directory, and refuses local fallback.
+3. **Plain-text instruction** — both variants inject an explicit "sandboxed eval mode: no web, no external context, audit constraint logic only" note that every sub-agent reads.
+
+Layer 1 is the load-bearing one; 2 and 3 are belt-and-suspenders. Net effect: when you run an LLM auditor on `dataset/circom/.../daira_hopwood_..._missing_bit_length_check`, the CLI sees a tmp dir with `circuit.circom` and a `circuits/` symlink — nothing that names the bug, no reference to the audit report, no exploit witness.
 
 ## Usage Modes
 
@@ -366,6 +450,27 @@ docker-compose run --rm zkhydra uv run python -m zkhydra.main zkbugs \
   --timeout 120
 ```
 
+### circom-auditor (native only — single circuit)
+
+> See the **circom-auditor — native-only** section above for the one-time host setup. Briefly: install Claude Code and/or Codex, authenticate the CLI, clone zk-skills, and expose the skill through `CLAUDE_PLUGIN_DIR` or `.agents/skills`.
+
+```bash
+uv run python -m zkhydra.main analyze \
+  --input examples/test_bug/circuits/circuit.circom \
+  --tools circom_auditor_claude \
+  --timeout 600
+```
+
+### circom-auditor on a single zkbugs reproducer (native)
+
+```bash
+uv run python -m zkhydra.main zkbugs \
+  --dataset zkbugs/dataset/circom \
+  --tools circom_auditor_claude \
+  --bugs veridise_decoder_accepting_bogus_output_signal \
+  --timeout 600
+```
+
 ## Output Structure
 
 ```