diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..e31d422 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,229 @@ +# LLM Auditor Run Plan — circom zkbugs + +Covers both `circom_auditor_claude` and `circom_auditor_codex` across all 70 +runnable circom bugs in `config.toml`, using `--zkbugs-mode both` (direct + +original entrypoints). Bugs are split into 6 shards to stay under API rate +limits. + +**Why 70 and not 72?** `config.toml` tracks 72 circom bugs total: 70 in +`[circom].bugs` (runnable — these are analyzed here) and 2 in +`[circom].unreproducible_bugs` (not present in the dataset at all: +`ProcessMessages` and `Potentially_Easy_to_Misuse_Interface`). + +--- + +## Prerequisites + +Verify each item before starting. + +```bash +# 1. Verify the zkbugs dataset is present next to this repo +ls ../zkbugs/dataset/circom # expected: org directories (0xbok, iden3, ...) + +# 2. (Optional) Download original codebases for original-mode runs +bash ../zkbugs/scripts/download_sources.sh + +# 3. Claude Code CLI +claude --version +echo $CLAUDE_PLUGIN_DIR # must be set +ls $CLAUDE_PLUGIN_DIR/skills/circom-auditor/SKILL.md # must exist + +# 4. OpenAI Codex CLI +codex --version +echo $OPENAI_API_KEY # must be set +# CODEX_PLUGIN_DIR defaults to CLAUDE_PLUGIN_DIR if unset (same skill dir works) + +# 5. Optional: increase include-closure cap for large circuits (default 5000) +# export CIRCOM_AUDITOR_MAX_LINES=8000 +``` + +--- + +## Step 1 — Split bugs into shards + +Run once. Produces `shards/shard_{1..6}.txt`, each with ~11 bugs. + +```bash +python scripts/split_bugs.py --shards 6 --output-dir shards/ +``` + +--- + +## Step 2 — Run each shard + +Run shards **sequentially** to respect rate limits (each shard takes ~2-3 h). +Adjacent shards (1+4, 2+5, 3+6) can run concurrently in separate terminals — +they have no shared state. + +Each command runs both tools on both modes (direct + original) and writes +results under `output/shard_N/`. + +```bash +# Shard 1 +uv run python -m zkhydra.main zkbugs \ + --dataset ../zkbugs/dataset/circom \ + --tools circom_auditor_claude,circom_auditor_codex \ + --bugs-file shards/shard_1.txt \ + --zkbugs-mode both \ + --output output/shard_1 + +# Shard 2 +uv run python -m zkhydra.main zkbugs \ + --dataset ../zkbugs/dataset/circom \ + --tools circom_auditor_claude,circom_auditor_codex \ + --bugs-file shards/shard_2.txt \ + --zkbugs-mode both \ + --output output/shard_2 + +# Shard 3 +uv run python -m zkhydra.main zkbugs \ + --dataset ../zkbugs/dataset/circom \ + --tools circom_auditor_claude,circom_auditor_codex \ + --bugs-file shards/shard_3.txt \ + --zkbugs-mode both \ + --output output/shard_3 + +# Shard 4 +uv run python -m zkhydra.main zkbugs \ + --dataset ../zkbugs/dataset/circom \ + --tools circom_auditor_claude,circom_auditor_codex \ + --bugs-file shards/shard_4.txt \ + --zkbugs-mode both \ + --output output/shard_4 + +# Shard 5 +uv run python -m zkhydra.main zkbugs \ + --dataset ../zkbugs/dataset/circom \ + --tools circom_auditor_claude,circom_auditor_codex \ + --bugs-file shards/shard_5.txt \ + --zkbugs-mode both \ + --output output/shard_5 + +# Shard 6 +uv run python -m zkhydra.main zkbugs \ + --dataset ../zkbugs/dataset/circom \ + --tools circom_auditor_claude,circom_auditor_codex \ + --bugs-file shards/shard_6.txt \ + --zkbugs-mode both \ + --output output/shard_6 +``` + +**Resuming a failed shard**: re-run the same command. Bugs that already have +`results.json` are not re-executed (pass `--vanilla` to just re-parse existing +raw output without re-running tools). + +--- + +## Step 3 — Merge shards into one combined run + +Stitches the six shard output dirs into a single `output/llm_combined/` that +mirrors a normal `--zkbugs-mode both` run structure (`direct/` + `original/`). + +```bash +python scripts/merge_shards.py \ + output/shard_1 output/shard_2 output/shard_3 \ + output/shard_4 output/shard_5 output/shard_6 \ + --output output/llm_combined +``` + +Expected layout after merge: + +``` +output/llm_combined/ + summary.json + direct/ + summary.json + / + ground_truth.json + circom_auditor_claude/ raw.txt parsed.json results.json evaluation.json + circom_auditor_codex/ ... + original/ + summary.json + / # only bugs with a distinct Original Entrypoint + ... +``` + +--- + +## Step 4 — Triage Undecided verdicts + +Claude automatically resolves verdicts that were marked Undecided by the +auto-evaluator (e.g. class match but different line number). Rewrites +`evaluation.json` in place; preserves the original at `evaluation.original.json`. + +Run for both modes: + +```bash +python scripts/triage_zkbugs_run.py output/llm_combined/direct \ + --dataset ../zkbugs/dataset/circom \ + --auto --update-evaluation --jobs 4 + +# Only if output/llm_combined/original/ exists and is non-empty: +python scripts/triage_zkbugs_run.py output/llm_combined/original \ + --dataset ../zkbugs/dataset/circom \ + --auto --update-evaluation --jobs 4 +``` + +Review the triage summary: + +```bash +cat output/llm_combined/direct/triage_summary.json +``` + +--- + +## Step 5 — Merge into the remote zkbugs run + +Copies the two LLM tool result dirs into the existing multi-tool remote run. +Run all four commands (two tools × two modes): + +```bash +# direct mode +python scripts/merge_tool_run.py \ + --source output/llm_combined/direct \ + --target output/zkbugs-remote/direct \ + --tool circom_auditor_claude + +python scripts/merge_tool_run.py \ + --source output/llm_combined/direct \ + --target output/zkbugs-remote/direct \ + --tool circom_auditor_codex + +# original mode (skip if output/llm_combined/original/ is empty) +python scripts/merge_tool_run.py \ + --source output/llm_combined/original \ + --target output/zkbugs-remote/original \ + --tool circom_auditor_claude + +python scripts/merge_tool_run.py \ + --source output/llm_combined/original \ + --target output/zkbugs-remote/original \ + --tool circom_auditor_codex +``` + +--- + +## Step 6 — Generate summary tables + +```bash +# Summary table + bug-tool matrix for the combined LLM run: +python scripts/process_zkbugs_results.py output/llm_combined/direct + +# Full remote run (all tools): +python scripts/process_zkbugs_results.py output/zkbugs-remote/direct + +# Optional: LaTeX/PDF report: +python scripts/process_zkbugs_results.py output/zkbugs-remote/direct \ + --latex output/zkbugs-remote/report.pdf +``` + +--- + +## Summary of output directories + +| Path | Contents | +|------|----------| +| `shards/shard_N.txt` | Bug selectors for shard N | +| `output/shard_N/` | Raw shard run (direct/ + original/) | +| `output/llm_combined/` | Merged LLM run (all 70 bugs) | +| `output/zkbugs-remote/` | Full multi-tool run (all tools including LLM) | diff --git a/README.md b/README.md index adda005..24b691f 100644 --- a/README.md +++ b/README.md @@ -237,6 +237,90 @@ uv run python -m zkhydra.main zkbugs \ - **Picus** - Symbolic execution via Rosette - **EcneProject** - Julia-based circuit analysis - **zkFuzz** - Fuzzing-based bug detection +- **circom_auditor_claude / circom_auditor_codex** - 17-agent parallel LLM audit via [zksecurity/zk-skills](https://github.com/zksecurity/zk-skills) — **native-only, not bundled in Docker**; ~3-10 min per circuit + +### circom-auditor — native-only + +> ⚠️ **`circom_auditor_claude` and `circom_auditor_codex` only run in native mode.** They are *not* installed in the zkhydra Docker image. If you try to invoke them inside `docker-compose run`, the tool plugin will exit with an error pointing back here. + +`circom_auditor_claude` invokes the [zksecurity/zk-skills](https://github.com/zksecurity/zk-skills) `circom-auditor` Claude Code skill. `circom_auditor_codex` invokes the same skill through Codex's native skill discovery (`.agents/skills`, user skills, admin skills, or installed plugins). The Codex wrapper prebuilds the delegated worker bundles and forbids local fallback: if Codex subagents are unavailable, the run fails instead of doing a single-agent audit. The skill spawns specialist sub-agents in parallel and produces a deduplicated, gate-validated security report. + +The reason for native-only: LLM CLIs may store subscription/OAuth credentials in the host OS keychain (macOS Keychain / libsecret on Linux / DPAPI on Windows). Those credentials cannot be mounted into a Linux container, so a containerised version would force everyone onto API-key billing. By keeping these tools native, you can use the host CLI auth already configured for Claude or Codex. + +#### One-time host setup + +```bash +# 1. Install Claude Code CLI +npm install -g @anthropic-ai/claude-code + +# 2. Authenticate — pick one +claude login # subscription, opens browser, stores in OS keychain +# OR +export ANTHROPIC_API_KEY=sk-ant-... # API key, pay per token + +# 3. Install zk-skills (the repo doubles as a plugin dir thanks to its +# committed `skills/circom-auditor` symlink — no extra scaffolding needed) +git clone https://github.com/zksecurity/zk-skills.git ~/zk-skills + +# 4. Tell zkhydra where the plugin dir is (add this to your shell rc) +export CLAUDE_PLUGIN_DIR=~/zk-skills + +# Optional Codex variant +npm install -g @openai/codex +codex login # or export CODEX_API_KEY=... +mkdir -p ~/.agents/skills +ln -s ~/zk-skills/skills/circom-auditor ~/.agents/skills/circom-auditor +``` + +#### Run it + +From a checkout of zkhydra, **without Docker**: + +```bash +uv run python -m zkhydra.main analyze \ + --input examples/test_bug/circuits/circuit.circom \ + --tools circom_auditor_claude \ + --timeout 600 +``` + +#### Mixed sweep — fast static tools in Docker, LLM auditor natively + +```bash +# 1. Static tools in the container (fast, deterministic, no auth) +docker-compose run --rm zkhydra uv run python -m zkhydra.main zkbugs \ + --dataset zkbugs/dataset/circom \ + --tools circomspect,circom_civer,picus,zkfuzz \ + --bugs daira_hopwood_darkforest_v0_3_missing_bit_length_check \ + --output output/static-only + +# 2. LLM auditor natively, against the same bug +uv run python -m zkhydra.main zkbugs \ + --dataset zkbugs/dataset/circom \ + --tools circom_auditor_claude \ + --bugs daira_hopwood_darkforest_v0_3_missing_bit_length_check \ + --output output/llm-only + +# 3. Merge per-bug findings.json files manually if you want a unified view +``` + +#### Caveats + +- Each `circom_auditor_claude` / `circom_auditor_codex` run spawns up to 17 parallel specialist sub-agents and takes 3-10 minutes wall-clock on a small bundle (1-5 templates / a few hundred lines). On larger scopes the wall-clock grows non-linearly — the sub-agents each have to ingest the full bundle before producing findings. +- **Use a per-bug timeout of `1800` (30 min), not 24h.** A bug that doesn't finish in 30 min is hung — fail it and move on. The skill is designed for the 2-5 templates a developer is actively touching, not monorepo-sized audits. +- **Bundle-size guard:** the tool plugin refuses to launch on scratch dirs above 30 `.circom` files or 5 000 lines of source (override via `CIRCOM_AUDITOR_MAX_FILES` / `CIRCOM_AUDITOR_MAX_LINES` env vars). zkbugs reproducers from large monorepos like Panther transitively pull in ~200 files / ~50K lines via `-l` link flags; those will be skipped with a clear "bundle too large" failure rather than hanging the run. +- Cost: subscription quota / account quota or per-token API spend, depending on the CLI auth you use. Pair with `--tools` and `--bugs` filtering to avoid running it on every bug in a large dataset sweep unless that's what you want. +- The skill follows Circom `include` chains (so wrapper-only zkbugs reproducers see the actual buggy template body via the `-l` link flag → scratch-dir trick the tool plugin handles automatically). +- Output is rich markdown — the tool plugin parses the `## Findings` and `## Leads` sections into zkhydra's standardized `Finding` schema. + +#### Eval-mode sandboxing (zkbugs honesty guarantees) + +When you point `circom_auditor_claude` or `circom_auditor_codex` at a zkbugs reproducer, the bug folder ships sidecar files that contain the literal answer key — `README.md` lists the vulnerability class, root cause, location, and proposed mitigation; `zkbugs_config.json` carries the same structured data. To prevent the LLM from "auditing" by reading the answer, the tool plugin runs every audit inside a fresh tmp dir and keeps the answer key out of it. Three layers of defence: + +1. **Filesystem isolation** — the scratch dir contains *only* `.circom` source: the wrapper, any sibling `.circom` files at the top level of the bug dir, and symlinks to the linked codebase's source subdirectories. Excluded by name: `README*`, `zkbugs_config.json`, `zkbugs_*.sh`, `input.json`, `direct_input.json`. Excluded by directory blocklist: `test`, `tests`, `doc`, `docs`, `client`, `examples`, `node_modules`, hidden dirs, and a few other common project-noise names that could leak per-bug hints. +2. **CLI restrictions** — the Claude variant blocks web/search tools and skips the user/project/local Claude settings stack; the Codex variant prebuilds delegated bundles, runs `codex exec` in read-only sandbox mode from the scratch directory, and refuses local fallback. +3. **Plain-text instruction** — both variants inject an explicit "sandboxed eval mode: no web, no external context, audit constraint logic only" note that every sub-agent reads. + +Layer 1 is the load-bearing one; 2 and 3 are belt-and-suspenders. Net effect: when you run an LLM auditor on `dataset/circom/.../daira_hopwood_..._missing_bit_length_check`, the CLI sees a tmp dir with `circuit.circom` and a `circuits/` symlink — nothing that names the bug, no reference to the audit report, no exploit witness. ## Usage Modes @@ -366,6 +450,27 @@ docker-compose run --rm zkhydra uv run python -m zkhydra.main zkbugs \ --timeout 120 ``` +### circom-auditor (native only — single circuit) + +> See the **circom-auditor — native-only** section above for the one-time host setup. Briefly: install Claude Code and/or Codex, authenticate the CLI, clone zk-skills, and expose the skill through `CLAUDE_PLUGIN_DIR` or `.agents/skills`. + +```bash +uv run python -m zkhydra.main analyze \ + --input examples/test_bug/circuits/circuit.circom \ + --tools circom_auditor_claude \ + --timeout 600 +``` + +### circom-auditor on a single zkbugs reproducer (native) + +```bash +uv run python -m zkhydra.main zkbugs \ + --dataset zkbugs/dataset/circom \ + --tools circom_auditor_claude \ + --bugs veridise_decoder_accepting_bogus_output_signal \ + --timeout 600 +``` + ## Output Structure ``` diff --git a/config.toml b/config.toml index dc527ac..daa4d37 100644 --- a/config.toml +++ b/config.toml @@ -28,7 +28,9 @@ tools = [ "circomspect", "EcneProject", "Picus", - "zkFuzz" + "zkFuzz", + "circom_auditor_claude", + "circom_auditor_codex", ] # List of bugs to analyze @@ -76,23 +78,67 @@ bugs = [ "bugs/zkbugs/dataset/circom/succinctlabs/telepathy-circuits/veridise_zero_padding_for_sha256_in_ExpandMessageXMD_is_vulnerable_to_an_overflow", ## tangle-network "bugs/zkbugs/dataset/circom/tangle-network/protocol-solidity/veridise_incorrect_initialization_in_membership_circuits", + ## personaelabs (cont.) + "bugs/zkbugs/dataset/circom/personaelabs/spartan-ecdsa/yacademy_Knowledge_of_any_member_signature_allow_to_generate_proof_of_membership", + ## privacy-scaling-explorations + "bugs/zkbugs/dataset/circom/privacy-scaling-explorations/maci/hashcloak_data_are_not_fully_verified_during_state_update", + ## reclaimprotocol (cont.) + "bugs/zkbugs/dataset/circom/reclaimprotocol/circom-chacha20/zksecurity_Unsound_Addition_Gadget", + "bugs/zkbugs/dataset/circom/reclaimprotocol/circom-chacha20/zksecurity_Unsound_XOR_gadget", ## zkopru-network + "bugs/zkbugs/dataset/circom/zkopru-network/zkopru/leastauthority_Circuit_Does_Not_Check_the_ERC_20_Sum_Correctly_", "bugs/zkbugs/dataset/circom/zkopru-network/zkopru/leastauthority_previously_correct_ownership_proof_disabled_via_code_changes", + ## aptos-labs + "bugs/zkbugs/dataset/circom/aptos-labs/keyless-zk-proofs/koukyosyumei_unconstrained_base64_decoded_len", + ## Arianee + "bugs/zkbugs/dataset/circom/Arianee/arianee-sdk/veridise_creditnoteproofs_can_be_stolen", + "bugs/zkbugs/dataset/circom/Arianee/arianee-sdk/veridise_ownershipproofs_could_identify_issuers", + ## banyancomputer + "bugs/zkbugs/dataset/circom/banyancomputer/hot-proofs-blake3-circom/koukyosyumei_checkdepth_comparator_overflow", + ## inference-labs-inc + "bugs/zkbugs/dataset/circom/inference-labs-inc/subnet-2-circom/koukyosyumei_clamp_comparator_overflow", + "bugs/zkbugs/dataset/circom/inference-labs-inc/subnet-2-circom/koukyosyumei_subtract_unconstrained_multiplier", + ## Moonsong-Labs + "bugs/zkbugs/dataset/circom/Moonsong-Labs/zksync-social-login-circuit/openzeppelin_mismatched_base64url_decoding_may_break_completeness", + "bugs/zkbugs/dataset/circom/Moonsong-Labs/zksync-social-login-circuit/openzeppelin_non_determinism_of_some_inputs", + ## pantherfoundation + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_babyjubjub_suborder_constraints_not_applied_correctly", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_blacklist_states_not_representable_in_field", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_bypassing_internal_transfer_limits_via_swaps", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_data_escrow_encrypted_message_wrong_input", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_fortxreward_abstracted_away_in_reward_calc", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_kyt_signature_verification_fails_nonzero_hash", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_nullifier_verification_can_be_disabled", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_unsafe_num2bits_254_blacklist_leaf", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_zaccount_renewal_kyc_expiry_not_validated", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_zaccount_renewal_multiple_nullifiers_same_utxo", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_zone_id_inclusion_prover_bypass", + "bugs/zkbugs/dataset/circom/pantherfoundation/panther-core/veridise_zone_limits_bypass_zswapv1", + ## rarimo + "bugs/zkbugs/dataset/circom/rarimo/passport-zk-circuits/koukyosyumei_under_constrained_date_encoder", + ## Rate-Limiting-Nullifier + "bugs/zkbugs/dataset/circom/Rate-Limiting-Nullifier/circom-rln/veridise_spammers_may_slash_themselves", + ## sismo-core + "bugs/zkbugs/dataset/circom/sismo-core/hydra-s2-zkps/veridise_private_information_leakage", + ## siv-org + "bugs/zkbugs/dataset/circom/siv-org/verifiable-private-overrides/koukyosyumei_emitifinrange_lessthan_overflow", + "bugs/zkbugs/dataset/circom/siv-org/verifiable-private-overrides/koukyosyumei_extractstringfrompoint_shiftedfirstbyte_unconstrained", + ## worm-privacy + "bugs/zkbugs/dataset/circom/worm-privacy/proof-of-burn/koukyosyumei_spend_missing_range_check", + ## zkemail + "bugs/zkbugs/dataset/circom/zkemail/ether-email-auth/matterlabs_emailauth_fails_with_overlapping_invitation_code_regex", + "bugs/zkbugs/dataset/circom/zkemail/zk-email-verify/matterlabs_underconstrained_fpmul_circuit", + "bugs/zkbugs/dataset/circom/zkemail/zk-email-verify/zksecurity_sha256_templates_return_zero_on_arbitrary_inputs", + "bugs/zkbugs/dataset/circom/zkemail/zk-regex/matterlabs_email_spoofing_via_manipulated_from_header", + "bugs/zkbugs/dataset/circom/zkemail/zk-regex/matterlabs_fromaddrregex_allows_email_address_spoofing", ] -# List of unreproducible bugs +# List of unreproducible bugs (not present in the dataset) unreproducible_bugs = [ - ## personaelabs - "bugs/zkbugs/dataset/circom/personaelabs/spartan-ecdsa/yacademy_Knowledge_of_any_member_signature_allow_to_generate_proof_of_membership", ## privacy-scaling-explorations - "bugs/zkbugs/dataset/circom/privacy-scaling-explorations/maci/hashcloak_data_are_not_fully_verified_during_state_update", "bugs/zkbugs/dataset/circom/privacy-scaling-explorations/maci/ProcessMessages_circuit_to_prevent_message_censorship_by_the_coordinator", # reclaimprotocol "bugs/zkbugs/dataset/circom/reclaimprotocol/circom-chacha20/zksecurity_Potentially_Easy_to_Misuse_Interface", - "bugs/zkbugs/dataset/circom/reclaimprotocol/circom-chacha20/zksecurity_Unsound_Addition_Gadget", - "bugs/zkbugs/dataset/circom/reclaimprotocol/circom-chacha20/zksecurity_Unsound_XOR_gadget", - # zkopru-network - "bugs/zkbugs/dataset/circom/zkopru-network/zkopru/leastauthority_Circuit_Does_Not_Check_the_ERC_20_Sum_Correctly_", ] [pil] diff --git a/scripts/merge_shards.py b/scripts/merge_shards.py new file mode 100644 index 0000000..37a463a --- /dev/null +++ b/scripts/merge_shards.py @@ -0,0 +1,230 @@ +#!/usr/bin/env python3 +"""Merge N partial shard run dirs into one combined run directory. + +Each shard ran on a non-overlapping subset of the bugs. This script +stitches the individual shard outputs into a single directory that looks +like a full zkhydra zkbugs run. + +For --zkbugs-mode both runs (which produce direct/ and original/ subdirs) +each sub-run is merged independently. For flat runs (direct or original +mode only, bug dirs at the top level) the bug dirs are merged directly. + +Usage: + python scripts/merge_shards.py \\ + output/shard_1 output/shard_2 output/shard_3 \\ + --output output/llm_combined + + # Then triage undecideds and merge into the remote run: + python scripts/triage_zkbugs_run.py output/llm_combined/direct \\ + --auto --update-evaluation --jobs 4 + python scripts/merge_tool_run.py \\ + --source output/llm_combined/direct \\ + --target output/zkbugs-remote/direct \\ + --tool circom_auditor_claude + python scripts/merge_tool_run.py \\ + --source output/llm_combined/direct \\ + --target output/zkbugs-remote/direct \\ + --tool circom_auditor_codex +""" + +import argparse +import json +import logging +import shutil +import sys +from pathlib import Path + + +def _is_bug_dir(d: Path) -> bool: + """True if this directory looks like a processed bug output dir.""" + if not d.is_dir(): + return False + return (d / "ground_truth.json").exists() or any( + c.is_dir() for c in d.iterdir() + ) + + +def _merge_summary_list(summaries: list[dict], mode: str) -> dict: + """Combine multiple shard summary.json files into one aggregate.""" + all_bugs: list[dict] = [] + total = processed = errors = skipped = 0 + + for s in summaries: + all_bugs.extend(s.get("bugs", [])) + total += s.get("total", 0) + processed += s.get("processed", 0) + errors += s.get("errors", 0) + skipped += s.get("skipped", 0) + + base = dict(summaries[0]) + base["bugs"] = all_bugs + base["total"] = total + base["processed"] = processed + base["errors"] = errors + base["skipped"] = skipped + base["mode"] = mode + # evaluation_counts will be stale after merging; remove so it's not misleading + base.pop("evaluation_counts", None) + return base + + +def _merge_dir( + shards: list[Path], + src_subdir: str | None, + dest: Path, + overwrite: bool, +) -> tuple[int, int]: + """Copy bug dirs from each shard's src_subdir into dest. + + Returns (copied, skipped_dup) counts. + """ + dest.mkdir(parents=True, exist_ok=True) + summaries: list[dict] = [] + copied = skipped_dup = 0 + + for shard in shards: + shard_root = shard / src_subdir if src_subdir else shard + + if not shard_root.is_dir(): + logging.info("Shard %s has no %s, skipping", shard.name, src_subdir or ".") + continue + + summary_path = shard_root / "summary.json" + if summary_path.is_file(): + try: + summaries.append( + json.loads(summary_path.read_text(encoding="utf-8")) + ) + except (json.JSONDecodeError, OSError) as exc: + logging.warning("Cannot read %s: %s", summary_path, exc) + + for bug_dir in sorted(shard_root.iterdir()): + if not _is_bug_dir(bug_dir): + continue + dest_bug = dest / bug_dir.name + if dest_bug.exists(): + if overwrite: + shutil.rmtree(dest_bug) + logging.warning( + "Overwriting duplicate bug in %s: %s", + src_subdir or ".", + bug_dir.name, + ) + else: + logging.warning( + "Duplicate bug %s in %s — skipped (use --overwrite to replace)", + bug_dir.name, + src_subdir or ".", + ) + skipped_dup += 1 + continue + shutil.copytree(bug_dir, dest_bug) + copied += 1 + + if summaries: + mode = src_subdir or "direct" + merged = _merge_summary_list(summaries, mode) + (dest / "summary.json").write_text( + json.dumps(merged, indent=2, ensure_ascii=False), encoding="utf-8" + ) + + return copied, skipped_dup + + +def main() -> int: + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(message)s", + datefmt="%H:%M:%S", + ) + p = argparse.ArgumentParser(description=__doc__) + p.add_argument( + "shards", + nargs="+", + type=Path, + help="Shard run dirs to merge (e.g. output/shard_1 output/shard_2)", + ) + p.add_argument( + "--output", "-o", + type=Path, + required=True, + help="Target combined run directory", + ) + p.add_argument( + "--overwrite", + action="store_true", + help="Replace duplicate bug dirs (default: skip and warn)", + ) + args = p.parse_args() + + for s in args.shards: + if not s.is_dir(): + logging.error("Shard not found: %s", s) + return 1 + + args.output.mkdir(parents=True, exist_ok=True) + + # Detect run structure: if any shard has a direct/ subdir it's a both-mode run. + is_both_mode = any((s / "direct").is_dir() for s in args.shards) + + if is_both_mode: + logging.info("Detected both-mode structure (direct/ + optional original/)") + + direct_dest = args.output / "direct" + copied, dups = _merge_dir(args.shards, "direct", direct_dest, args.overwrite) + logging.info("direct/ copied=%d skipped_dup=%d", copied, dups) + + has_original = any((s / "original").is_dir() for s in args.shards) + original_dest = args.output / "original" + if has_original: + copied, dups = _merge_dir( + args.shards, "original", original_dest, args.overwrite + ) + logging.info("original/ copied=%d skipped_dup=%d", copied, dups) + + combined = { + "mode": "both", + "merged_from": [str(s) for s in args.shards], + "output_root": str(args.output), + "modes": { + "direct": { + "ran": True, + "output_dir": str(direct_dest), + }, + "original": { + "ran": has_original, + "output_dir": str(original_dest) if has_original else None, + }, + }, + } + (args.output / "summary.json").write_text( + json.dumps(combined, indent=2, ensure_ascii=False), encoding="utf-8" + ) + else: + logging.info("Detected flat structure (bug dirs at top level)") + copied, dups = _merge_dir(args.shards, None, args.output, args.overwrite) + logging.info("copied=%d skipped_dup=%d", copied, dups) + + logging.info("Merge complete → %s", args.output) + + print(f"\nNext steps:") + run_dir = args.output / "direct" if is_both_mode else args.output + print(f" 1. Triage undecided verdicts:") + print( + f" python scripts/triage_zkbugs_run.py {run_dir}" + f" --auto --update-evaluation --jobs 4" + ) + print(f" 2. Merge into the remote run:") + for tool in ("circom_auditor_claude", "circom_auditor_codex"): + print( + f" python scripts/merge_tool_run.py" + f" --source {run_dir}" + f" --target output/zkbugs-remote/direct" + f" --tool {tool}" + ) + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/process_zkbugs_results.py b/scripts/process_zkbugs_results.py index 87d9cfd..cf57ead 100755 --- a/scripts/process_zkbugs_results.py +++ b/scripts/process_zkbugs_results.py @@ -18,6 +18,21 @@ from typing import Dict, List, Tuple +# All tools evaluated by zkhydra zkbugs sweeps. Order matters: it sets the +# column order in every printed table, the LaTeX report, and the bug-tool +# matrix. Add a tool here to surface it everywhere. +TOOLS = [ + "circomspect", + "circom_civer", + "picus", + "ecneproject", + "zkfuzz", + "conscs", + "circom_auditor_claude", + "circom_auditor_codex", +] + + def load_json(file_path: Path) -> dict: """Load JSON file safely.""" try: @@ -116,7 +131,7 @@ def collect_results( bug_dirs = sorted([d for d in results_dir.iterdir() if d.is_dir()]) # Tools to check - tools = ["circomspect", "circom_civer", "picus", "ecneproject", "zkfuzz", "conscs"] + tools = TOOLS for bug_dir in bug_dirs: bug_name = bug_dir.name @@ -155,7 +170,7 @@ def print_tool_summary_table(tool_stats: Dict[str, Dict[str, int]], tool_times: # Define columns and tools columns = ["TP", "FN", "Undecided", "Timeout", "Failure"] - tools = ["circomspect", "circom_civer", "picus", "ecneproject", "zkfuzz", "conscs"] + tools = TOOLS # Calculate column widths tool_width = max(len(tool) for tool in tools + ["TOTAL"]) @@ -222,7 +237,7 @@ def print_bug_tool_matrix( bug_tool_matrix: Dictionary mapping bug names to tool statuses (with asterisks) full_path: If True, print full bug names without truncation """ - tools = ["circomspect", "circom_civer", "picus", "ecneproject", "zkfuzz", "conscs"] + tools = TOOLS # Calculate column widths bug_width = max(len(bug) for bug in bug_tool_matrix.keys()) @@ -271,7 +286,7 @@ def print_execution_time_stats( print("EXECUTION TIME STATISTICS") print("=" * 100) - tools = ["circomspect", "circom_civer", "picus", "ecneproject", "zkfuzz", "conscs"] + tools = TOOLS # Calculate column widths tool_width = max(len(tool) for tool in tools) @@ -326,8 +341,8 @@ def print_statistics( print("STATISTICS") print("=" * 80) print(f"Total bugs processed: {bug_count}") - print(f"Total tools evaluated: 5") - print(f"Total possible evaluations: {bug_count * 5}") + print(f"Total tools evaluated: {len(TOOLS)}") + print(f"Total possible evaluations: {bug_count * len(TOOLS)}") # Count actual evaluations (excluding N/A and Unknown) actual_evals = sum(sum(counts.values()) for counts in tool_stats.values()) @@ -354,8 +369,8 @@ def print_statistics( print(f" Failures: {total_failure:3d} ({total_failure/evaluated*100:5.1f}%)") # Bug-level detection statistics - tools = ["circomspect", "circom_civer", "picus", "ecneproject", "zkfuzz", "conscs"] - tools_without_ecne = ["circomspect", "circom_civer", "picus", "zkfuzz"] + tools = TOOLS + tools_without_ecne = [t for t in TOOLS if t != "ecneproject"] # Count bugs where at least one tool detected the vulnerability bugs_detected_all = 0 @@ -400,7 +415,7 @@ def generate_latex_report( output_pdf: Path, ): """Generate LaTeX report with four tables.""" - tools = ["circomspect", "circom_civer", "picus", "ecneproject", "zkfuzz", "conscs"] + tools = TOOLS columns = ["TP", "FN", "Timeout", "Failure"] # Create bug ID mapping @@ -505,6 +520,16 @@ def generate_latex_report( f"{latex_tool_name} & 0 & --- & --- & --- & --- & {nr_timeout} " + r"\\" + "\n" ) + # Build dynamic LaTeX bits driven by `tools` so adding a tool here is a + # one-line change to TOOLS rather than four hand-edited table headers. + matrix_col_spec = "l|" + "c" * len(tools) + time_col_spec = "l|" + "r" * len(tools) + tool_headers = " & ".join( + rf"\textbf{{{t.replace('_', r'\_')}}}" for t in tools + ) + matrix_header_row = rf"\textbf{{Bug ID}} & {tool_headers} \\" + matrix_continued_cols = len(tools) + 1 + latex_content += r"""\bottomrule \end{tabular} \end{table} @@ -515,19 +540,19 @@ def generate_latex_report( \begin{landscape} \footnotesize -\begin{longtable}{l|ccccc} +""" + rf"""\begin{{longtable}}{{{matrix_col_spec}}} \toprule -\textbf{Bug ID} & \textbf{circomspect} & \textbf{circom\_civer} & \textbf{picus} & \textbf{ecneproject} & \textbf{zkfuzz} \\ +{matrix_header_row} \midrule \endfirsthead \toprule -\textbf{Bug ID} & \textbf{circomspect} & \textbf{circom\_civer} & \textbf{picus} & \textbf{ecneproject} & \textbf{zkfuzz} \\ +{matrix_header_row} \midrule \endhead \midrule -\multicolumn{6}{r}{\textit{Continued on next page}} \\ +\multicolumn{{{matrix_continued_cols}}}{{r}}{{\textit{{Continued on next page}}}} \\ \endfoot \bottomrule @@ -594,19 +619,19 @@ def generate_latex_report( \begin{landscape} \footnotesize -\begin{longtable}{l|rrrrr} +""" + rf"""\begin{{longtable}}{{{time_col_spec}}} \toprule -\textbf{Bug ID} & \textbf{circomspect} & \textbf{circom\_civer} & \textbf{picus} & \textbf{ecneproject} & \textbf{zkfuzz} \\ +{matrix_header_row} \midrule \endfirsthead \toprule -\textbf{Bug ID} & \textbf{circomspect} & \textbf{circom\_civer} & \textbf{picus} & \textbf{ecneproject} & \textbf{zkfuzz} \\ +{matrix_header_row} \midrule \endhead \midrule -\multicolumn{6}{r}{\textit{Continued on next page}} \\ +\multicolumn{{{matrix_continued_cols}}}{{r}}{{\textit{{Continued on next page}}}} \\ \endfoot \bottomrule diff --git a/scripts/split_bugs.py b/scripts/split_bugs.py new file mode 100644 index 0000000..6c9f022 --- /dev/null +++ b/scripts/split_bugs.py @@ -0,0 +1,154 @@ +#!/usr/bin/env python3 +"""Split the configured bug list into N shards for rate-limited LLM tools. + +Each shard is written as a --bugs-file compatible text file (one bug selector +per line). Bugs are distributed via round-robin so each shard is nearly the +same size and the last shard is never more than N-1 entries smaller than +the first. + +Usage: + # Split config.toml [circom].bugs into 4 shards: + python scripts/split_bugs.py --shards 4 --output-dir shards/ + + # Split an existing bugs file: + python scripts/split_bugs.py --bugs-file all_bugs.txt --shards 3 --output-dir shards/ + +After splitting, the script prints ready-to-run commands for each shard and +the merge command to combine them afterwards. +""" + +import argparse +import sys +import tomllib +from pathlib import Path + + +def read_bugs_from_config(config_path: Path) -> list[str]: + """Read [circom].bugs from config.toml.""" + with open(config_path, "rb") as f: + data = tomllib.load(f) + return data.get("circom", {}).get("bugs", []) + + +def read_bugs_from_file(bugs_file: Path) -> list[str]: + """Read one bug selector per line, ignoring blank lines and comments.""" + lines = [] + for raw in bugs_file.read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if line and not line.startswith("#"): + lines.append(line) + return lines + + +def split_round_robin(bugs: list[str], n: int) -> list[list[str]]: + """Distribute bugs across n shards by round-robin for even sizing.""" + shards: list[list[str]] = [[] for _ in range(n)] + for i, bug in enumerate(bugs): + shards[i % n].append(bug) + return shards + + +def main() -> int: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument( + "--shards", "-n", type=int, required=True, + help="Number of shards to create", + ) + p.add_argument( + "--output-dir", "-o", type=Path, required=True, + help="Directory where shard_1.txt ... shard_N.txt are written", + ) + + source = p.add_mutually_exclusive_group() + source.add_argument( + "--config", type=Path, default=None, + help="Path to config.toml (default: config.toml in cwd)", + ) + source.add_argument( + "--bugs-file", type=Path, default=None, + help="Existing --bugs-file to split (one selector per line)", + ) + + args = p.parse_args() + + if args.shards < 1: + print("Error: --shards must be >= 1", file=sys.stderr) + return 1 + + if args.bugs_file: + if not args.bugs_file.is_file(): + print(f"Error: --bugs-file not found: {args.bugs_file}", file=sys.stderr) + return 1 + bugs = read_bugs_from_file(args.bugs_file) + else: + config_path = args.config or Path("config.toml") + if not config_path.is_file(): + print(f"Error: config not found: {config_path}", file=sys.stderr) + return 1 + bugs = read_bugs_from_config(config_path) + + if not bugs: + print("Error: no bugs found in source", file=sys.stderr) + return 1 + + effective_shards = min(args.shards, len(bugs)) + if effective_shards < args.shards: + print( + f"Warning: only {len(bugs)} bug(s) — reducing to {effective_shards} shard(s)", + file=sys.stderr, + ) + + print(f"Found {len(bugs)} bug(s), splitting into {effective_shards} shard(s)") + + shards = split_round_robin(bugs, effective_shards) + + args.output_dir.mkdir(parents=True, exist_ok=True) + shard_paths: list[Path] = [] + for i, shard in enumerate(shards, 1): + shard_path = args.output_dir / f"shard_{i}.txt" + shard_path.write_text("\n".join(shard) + "\n", encoding="utf-8") + shard_paths.append(shard_path) + print(f" shard_{i}.txt: {len(shard)} bug(s)") + + print("\nReady-to-run commands (run each separately to respect rate limits):") + for i, path in enumerate(shard_paths, 1): + print(f"\n # Shard {i}/{effective_shards}:") + print( + f" uv run python -m zkhydra.main zkbugs" + f" --dataset bugs/zkbugs/dataset/circom" + f" --tools circom_auditor_claude,circom_auditor_codex" + f" --bugs-file {path}" + f" --zkbugs-mode both" + f" --output output/shard_{i}" + ) + + shard_dirs = " ".join(f"output/shard_{i}" for i in range(1, effective_shards + 1)) + print(f"\n # After all shards complete — merge into one combined run:") + print( + f" python scripts/merge_shards.py {shard_dirs}" + f" --output output/llm_combined" + ) + print( + f"\n # Triage undecided verdicts:" + ) + print( + f" python scripts/triage_zkbugs_run.py output/llm_combined/direct" + f" --auto --update-evaluation --jobs 4" + ) + print( + f"\n # Merge into the remote run:" + ) + print( + f" python scripts/merge_tool_run.py --source output/llm_combined/direct" + f" --target output/zkbugs-remote/direct --tool circom_auditor_claude" + ) + print( + f" python scripts/merge_tool_run.py --source output/llm_combined/direct" + f" --target output/zkbugs-remote/direct --tool circom_auditor_codex" + ) + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/zkhydra/core.py b/zkhydra/core.py index f36ba17..b03673e 100644 --- a/zkhydra/core.py +++ b/zkhydra/core.py @@ -102,6 +102,8 @@ def to_dict(self) -> dict: "picus", "ecneproject", "zkfuzz", + "circom_auditor_claude", + "circom_auditor_codex", ], "pil": ["pilspector"], "cairo": ["sierra-analyzer"], @@ -326,7 +328,12 @@ def load_bug_selectors( def _bug_matches_selectors( bug_dir: Path, dataset_dir: Path, selectors: list[str] ) -> bool: - """True if any selector is a substring of the bug name or relative path.""" + """True if any selector overlaps with the bug name or dataset-relative path. + + Supports both short selectors (e.g. "0xbok", "veridise_decoder") and full + config-file paths (e.g. "bugs/zkbugs/dataset/circom/org/repo/bug") by + checking in both directions. + """ if not selectors: return True name = bug_dir.name @@ -334,7 +341,10 @@ def _bug_matches_selectors( rel = str(bug_dir.resolve().relative_to(dataset_dir.resolve())) except ValueError: rel = str(bug_dir) - return any(sel in name or sel in rel for sel in selectors) + return any( + sel in name or sel in rel or name in sel or rel in sel + for sel in selectors + ) def discover_zkbugs( diff --git a/zkhydra/tools/circom_auditor_base.py b/zkhydra/tools/circom_auditor_base.py new file mode 100644 index 0000000..a2a8f66 --- /dev/null +++ b/zkhydra/tools/circom_auditor_base.py @@ -0,0 +1,783 @@ +""" +Shared sandbox-building and markdown-report-parsing logic for LLM-based +Circom auditors (Claude and Codex variants). + +Both CircomAuditorClaude and CircomAuditorCodex inherit from CircomAuditorBase. +The base class owns: + - Include-closure sandbox builder (_prepare_scratch_dir and helpers) + - Markdown report parser (_helper_parse_output and helpers) + - Uniform Finding converter (_helper_generate_uniform_results) + - zkbugs ground-truth evaluator (evaluate_zkbugs_ground_truth) + +Each subclass implements only _internal_execute, choosing how to invoke +its respective LLM CLI from the prepared scratch directory, and __init__, +which checks that the required binary and credentials are present. +""" + +import logging +import os +import re +import shutil +import tempfile +from abc import abstractmethod +from collections import deque +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +from zkhydra.tools.base import ( + AbstractTool, + AnalysisStatus, + Finding, + Input, + StandardizedBugCategory, + ToolOutput, +) + +# Map common bug-class slugs (kebab-case fragments from finding titles or +# dedup keys) to the standardized zkhydra category. Matched on the most +# reliable substring in the finding title. +BUG_CLASS_TO_STANDARD: List[Tuple[str, StandardizedBugCategory]] = [ + ( + "comparator-input-not-range-checked", + StandardizedBugCategory.UNDER_CONSTRAINED, + ), + ("missing-range-check", StandardizedBugCategory.UNDER_CONSTRAINED), + ("range-check", StandardizedBugCategory.UNDER_CONSTRAINED), + ("range checked", StandardizedBugCategory.UNDER_CONSTRAINED), + ("range checks", StandardizedBugCategory.UNDER_CONSTRAINED), + ("packbytes", StandardizedBugCategory.UNDER_CONSTRAINED), + ("num2bits", StandardizedBugCategory.UNDER_CONSTRAINED), + ("aliasing", StandardizedBugCategory.UNDER_CONSTRAINED), + ("assigned-but-unconstrained", StandardizedBugCategory.UNDER_CONSTRAINED), + ("assigned but unconstrained", StandardizedBugCategory.UNDER_CONSTRAINED), + ("unconstrained", StandardizedBugCategory.UNDER_CONSTRAINED), + ("under-constrained", StandardizedBugCategory.UNDER_CONSTRAINED), + ("under constrained", StandardizedBugCategory.UNDER_CONSTRAINED), + ("witness-only", StandardizedBugCategory.UNDER_CONSTRAINED), + ("decoder", StandardizedBugCategory.UNDER_CONSTRAINED), + ("one-sided", StandardizedBugCategory.UNDER_CONSTRAINED), + ("div-by-zero", StandardizedBugCategory.UNDER_CONSTRAINED), + ("division-by-zero", StandardizedBugCategory.UNDER_CONSTRAINED), + ("ec-edge-case", StandardizedBugCategory.UNDER_CONSTRAINED), + ("equal-x", StandardizedBugCategory.UNDER_CONSTRAINED), + ("intent-binding", StandardizedBugCategory.UNDER_CONSTRAINED), + ("replay", StandardizedBugCategory.UNDER_CONSTRAINED), + ("conditional-gate-collapse", StandardizedBugCategory.UNDER_CONSTRAINED), + ("selector", StandardizedBugCategory.UNDER_CONSTRAINED), + ("selector-not-boolean", StandardizedBugCategory.UNDER_CONSTRAINED), + ("limb-out-of-range", StandardizedBugCategory.UNDER_CONSTRAINED), + ("non-canonical", StandardizedBugCategory.UNDER_CONSTRAINED), + ("merkle", StandardizedBugCategory.UNDER_CONSTRAINED), + ("nullifier", StandardizedBugCategory.UNDER_CONSTRAINED), + ("over-constrained", StandardizedBugCategory.OVER_CONSTRAINED), + ("over constrained", StandardizedBugCategory.OVER_CONSTRAINED), + ("completeness", StandardizedBugCategory.OVER_CONSTRAINED), + ("regex-overlap", StandardizedBugCategory.COMPUTATIONAL_ISSUE), + ("base64", StandardizedBugCategory.COMPUTATIONAL_ISSUE), + ("hash-construction", StandardizedBugCategory.COMPUTATIONAL_ISSUE), + ("non-determinism", StandardizedBugCategory.COMPUTATIONAL_ISSUE), + ("non-deterministic", StandardizedBugCategory.COMPUTATIONAL_ISSUE), + ("computational", StandardizedBugCategory.COMPUTATIONAL_ISSUE), + ("privacy", StandardizedBugCategory.COMPUTATIONAL_ISSUE), + ("information-leak", StandardizedBugCategory.COMPUTATIONAL_ISSUE), + ("information leak", StandardizedBugCategory.COMPUTATIONAL_ISSUE), + ("shadowing", StandardizedBugCategory.WARNING), + ("bitwise-complement", StandardizedBugCategory.WARNING), + ("assertion-vs-constraint", StandardizedBugCategory.WARNING), + ("assert-vs-constraint", StandardizedBugCategory.WARNING), + ("slash-vs-backslash", StandardizedBugCategory.WARNING), +] + + +@dataclass +class _ScopeInfo: + """Description of the include closure materialised in the scratch dir.""" + + wrapper_name: str + file_count: int + total_lines: int + cap_lines: int + unresolved: List[str] = field(default_factory=list) + skipped_no_root: List[str] = field(default_factory=list) + + @property + def truncated(self) -> bool: + return any("truncated at budget" in u for u in self.unresolved) + + def summary_line(self) -> str: + truncated = " (truncated at budget)" if self.truncated else "" + return ( + f"include closure rooted at `{self.wrapper_name}` — " + f"{self.file_count} file(s) / {self.total_lines} lines" + f"{truncated}" + ) + + def manifest_markdown(self) -> str: + lines = [ + "# Audit scope", + "", + f"**Wrapper / entrypoint:** `{self.wrapper_name}`", + f"**Files in scope:** {self.file_count} `.circom` (transitive include closure of the wrapper)", + f"**Total source lines:** {self.total_lines:,} (cap: {self.cap_lines:,})", + "", + "Audit only what's reachable from the wrapper's `include` graph " + "as materialised in this directory. Do **not** speculate about " + "code that isn't here — assume out-of-scope templates are " + "outside the soundness boundary you're being asked to verify.", + ] + if self.truncated: + lines += [ + "", + "## Warning: Closure was truncated at the line budget", + "", + "The wrapper's full transitive closure exceeded the configured " + f"line cap ({self.cap_lines:,}). The bundle in this directory " + "is the **breadth-first prefix** that fit; deeper / later-in-BFS " + "deps were dropped.", + ] + if self.unresolved: + shown = [ + u for u in self.unresolved if "truncated at budget" not in u + ] + if shown: + lines += [ + "", + "## Unresolved includes", + "", + "These `include` specs could not be resolved against any link root:", + "", + ] + [f"- `{u}`" for u in shown[:32]] + return "\n".join(lines) + "\n" + + +@dataclass +class CircomAuditorIssue: + """One finding parsed out of the skill's markdown report.""" + + title: str + confidence: int + template: Optional[str] + file: Optional[str] + line: Optional[int] + line_end: Optional[int] + signal: Optional[str] + description: str + severity: str # "finding" or "lead" + agents: Optional[int] = ( + None # the [agents: N] convergence count, when present + ) + + def to_dict(self) -> Dict[str, Any]: + return { + "title": self.title, + "confidence": self.confidence, + "template": self.template, + "file": self.file, + "line": self.line, + "line_end": self.line_end, + "signal": self.signal, + "description": self.description, + "severity": self.severity, + "agents": self.agents, + } + + +@dataclass +class CircomAuditorParsed: + """Structured parsed output from a circom-auditor run.""" + + status: str # "success" | "timeout" | "error" | "no_findings" + issues: List[CircomAuditorIssue] = field(default_factory=list) + raw_report: str = "" + + def to_dict(self) -> Dict[str, Any]: + return { + "status": self.status, + "issues": [issue.to_dict() for issue in self.issues], + "statistics": { + "total_findings": sum( + 1 for i in self.issues if i.severity == "finding" + ), + "total_leads": sum( + 1 for i in self.issues if i.severity == "lead" + ), + }, + } + + +# Regex bank for extracting finding metadata from the report's per-finding +# header line (per circom-auditor/references/report-formatting.md): +# `TemplateName (file.circom:LL-LL) · signal: ` · Confidence: 95 +_HEADER_RE = re.compile( + r"`(?P