Feature Deep Verify

Feature: Deep Verify

What it is. The final, deepest check on a surviving finding: one full agentic session per finding — no batching, no excerpt budget, full Read/Grep/Glob access to the actual repo — that independently re-derives the finding from the source, then reasons across the whole survivor set at once. It runs after Docs & History Corroboration, before optional runtime/live verification.

This is the tool encoding a specific manual discipline: read every cited line yourself, trace every sibling function, and don't trust an earlier verdict just because it survived two prior passes.

Why a separate stage from validation

Adversarial Validation is skeptical, but its prompt explicitly forbids one finding's verdict from influencing another's — so it can never notice that finding A and finding B are the same bug reached two different ways, or that finding C is quietly bundling two independently-triggerable bugs under one description. It's also batched and excerpt-budgeted for throughput across every raw candidate.

Deep-verify inverts both trade-offs on purpose: it runs on the much smaller surviving set (already thinned by validation and corroboration), one uncapped session per finding, and is handed a compact summary of every other surviving finding so it can reason across the set.

The verdict space

Validation's verdict is essentially binary (confirmed / refuted). Deep-verify adds a real middle ground:

Verdict	Meaning	Effect
reconfirmed	Independently re-derived; stands as written	kept
corrected	Mechanism is real, one stated fact was wrong (wrong line, wrong field, wrong precondition)	kept, correction noted
split	This one finding is actually ≥2 independently-triggerable bugs	replaced by N independent sub-findings, original kept in a split_originals appendix
merged	Duplicates another survivor by root cause (not just by dedup key)	folded into that finding, kept in a merged_findings appendix
refuted	Deep re-derivation shows validation and corroboration were both wrong	moved to the normal dropped list
inconclusive	A genuine, thorough attempt couldn't settle it from source alone	kept, flagged

Downgrade-don't-delete still applies: only refuted removes a finding outright. split and merged never delete — the original is always kept in a report appendix, never silently gone.

Every verdict — including reconfirmed — requires an independent_derivation field: the actual file:line trail walked, the sibling/caller/callee functions opened, the struct/ABI/precondition detail confirmed. This is the audit trail that makes the pass trustworthy rather than just trusted.

Reachability against sibling consumers (added 2026-07-19)

A finding can correctly identify a real, unbounded allocation or loop and still overclaim how easy it is to trigger, because a single flagged code path is never the whole story: the same untrusted field or byte range is often read by other code too, and if one of those other readers runs earlier in the same pipeline and fails safely on malformed input, it can incidentally block the flagged path from ever being reached for the simplest malicious construction — without being any kind of deliberate validation of that field.

This surfaced from manual disclosure work, not from a deep-verify pipeline run: a hand-verified "tiny file instantly crashes the whole run" DoS finding didn't reproduce when an actual proof-of-concept was built and executed, because an unrelated dependency-scan pre-pass happened to walk the same bytes first, hit a plain caught exception on the malformed input, and flagged the item as already-erroneous before the real, unbounded allocation was ever attempted. The underlying missing bound was completely real; the specific reachability claim, as originally worded, was not — for that construction.

Deep-verify's prompt now makes this an explicit required step: grep the whole repo for every other reader of the same untrusted field/bytes, check whether any plausibly runs before the flagged code and would fail in a way that gates it out of reach, and — if so — downgrade to corrected rather than silently reconfirmed, folding the caveat into corrections even when the underlying defect still stands. This is exactly the kind of cross-call-site reasoning Adversarial Validation and Docs & History Corroboration cannot do from an excerpt in isolation — it needs deep-verify's unbounded repo access and one-session-per-finding budget.

How it's gated

• Opt-in, off by default — --verify/--no-verify on pipeline, or argo verify --run RUN_ID standalone. Off by default because it's the most expensive annotation stage: one full session per finding, never batched.
• Offline, like validation — full read-only repo access (Read/Grep/Glob/Write), no network tools. See Guardrails & Safety.
• --verify-max-findings N caps how many survivors get a session (cost control on a large survivor set); findings past the cap are kept, un-deep-verified, never dropped.
• Best-effort with one retry — a session failure (CLI/sandbox crash, no output, malformed JSON) retries once by default (verify_max_attempts, default 2) before falling back to inconclusive with an infra-failure rationale prefix, distinguishable from a genuine "couldn't tell". A bare retry is worth it here specifically: real sessions have run 1-4M input tokens and cost $1-4 each, unlike validation/corroboration's much cheaper per-session failures (no retry there). Same infra-vs-genuine split pattern as validation's needs_runtime_verification and corroboration's unknown.

First real-run results (authentik, 2026-07-18)

On a 22.4k-star identity provider, verify re-examined 20 survivors already passed by validation (and partly by corroboration): 8 reconfirmed with genuine file:line re-derivations, 2 corrected (a wrong reachability claim on a SCIM group-patch path; a WebAuthn finding's own overclaim against the project's documented partial-coverage caveat), 1 refuted, and 9 inconclusive from a Codex sandbox instability that hit verify's heavier, unbounded sessions harder than validation's lighter ones (motivating the retry above). The refuted case is the clearest evidence of this stage's value: validation had confirmed a SCIM JIT-provisioning finding, and corroboration couldn't reach a verdict (infra failure) — deep-verify re-derived the mechanism as real, then found the project's own SCIM source documentation defines that exact behavior as the intended high-trust provisioning model, and refuted it under the purpose-is-the-feature rule. Zero splits/merges were seen in this run; that machinery hasn't yet been exercised against a run known to contain redundant/bundled findings.

Related

• Adversarial Validation — the stage immediately before corroborate; the per-finding-isolated, batched pass this stage is not.
• Docs & History Corroboration — the networked cross-check that runs right before this one.
• Determinism & Anti-Hallucination — the downgrade-don't-delete philosophy this stage extends with split/merged appendices.