`results_verified.json` metrics recounts from verdict strings and has no bucket for errored or incomplete verifications

[gadievron]

Evidence provenance (added 2026-08-22). The run-derived figures in this issue come from a
private scan that a reader cannot reproduce (raptor-e2e-20260818, a 9.85-hour run against
a Python target). They are reported for completeness, not as independently checkable evidence.
Every code claim below is pinned to public source at b5019628 and is checkable; where a
conclusion rests on the private numbers rather than on the code, treat it as my report of what
that run showed. This label was missing when the issue was filed.

Summary

_write_verified_results re-derives the metrics block after Stage 2 by classifying each merged
result on its verdict string. Its only route into the errors bucket is elif r.get("verdict") == "ERROR" —
a value the verify error path never writes. An errored verification keeps its Stage-1
finding: "vulnerable" / verdict: "VULNERABLE", so it is counted as a confirmed vulnerability,
and errors stays 0. The correct buckets are already computed a few lines earlier by
_count_verification_outcomes, and are discarded rather than written to the file.

This is the root of the headline metric divergence. core/reporter.py:304 merely reads
experiment["metrics"] — it consumes what is already wrong. core/reporter.py:562-570 is a
second, different defect (filed separately) and fixing only the reporter would copy a wrong
number more elaborately.

Evidence

libs/openant-core/core/verifier.py:359-373 (HEAD b501962) — the recount:

    # Recount metrics after verification
    counts = {
        "vulnerable": 0, "bypassable": 0, "inconclusive": 0,
        "protected": 0, "safe": 0, "errors": 0,
    }
    for r in merged_results:
        # Canonical read: lowercase a PRESENT finding too (not only the
        # verdict/default), so a verdict-only result is classified correctly.
        finding = str(r.get("finding") or r.get("verdict") or "error").lower()
        if finding in counts:
            counts[finding] += 1
        elif r.get("verdict") == "ERROR":
            counts["errors"] += 1

    output["metrics"] = {"total": len(merged_results), **counts}

core/verifier.py:274-330 (_count_verification_outcomes) already buckets the same list correctly —
error_count on r.get("error"), needs_review on verification.incomplete — and its result goes
to verify.report.json but never into results_verified.json.

core/reporter.py:304:

    metrics = experiment.get("metrics", {})

The two blocks disagree on the same run

RUN=~/.openant/projects/gadievron/raptor-e2e-20260818/scans/7dbf9c691d7/python
python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['metrics'])" $RUN/results_verified.json
python3 -c "import json,sys; print(json.load(open(sys.argv[1]))['summary']['metrics'])" $RUN/scan.report.json

results_verified.json  {total 5475, vulnerable 183, bypassable 0, inconclusive 0,
                        protected 414, safe 4878, errors 0}
scan.report.json       {total 5475, vulnerable 1,   bypassable 0, inconclusive 0,
                        protected 408, safe 4884, errors 48,
                        verified 223, stage2_agreed 1, stage2_disagreed 40, needs_review 134}

Why errors is 0

Every errored verification carries a truthy error string and its Stage-1 verdict:

python3 - <<'EOF'
import json, collections
d = json.load(open("$RUN/results_verified.json"))
res = d["results"]
err = [r for r in res if r.get("error")]
print("records with .error :", len(err))
print("their verdict values:", collections.Counter(str(r.get("verdict")) for r in err))
print("their finding values:", collections.Counter(str(r.get("finding")) for r in err))
print("records with verdict == 'ERROR':", sum(1 for r in res if r.get("verdict") == "ERROR"))
print("records with verification.incomplete:",
      sum(1 for r in res if (r.get("verification") or {}).get("incomplete")))
EOF

records with .error : 48
their verdict values: Counter({'VULNERABLE': 48})
their finding values: Counter({'vulnerable': 48})
records with verdict == 'ERROR': 0
records with verification.incomplete: 182

So the elif at :370 is unreachable on this data, and all 48 errored units land in vulnerable.
The 183 reconciles exactly: 183 = 134 (incomplete, adjudication never completed) + 48 (errored) + 1 (Stage 2 agreed and confirmed).

The writer that produces those records is utilities/finding_verifier.py:755-758 (the comment continues to :762; :759-762 not shown), whose own comment
states the intended contract:

            # L4 (PR #69 round-5): record the error ON the result dict, not just
            # in the local ``detail``. The downstream counter (core/verifier.py)
            # buckets on ``r.get("error")``; without this the errored finding
            # falls through to "disagreed" and is folded into the ``safe`` count.

_count_verification_outcomes honours that contract. The recount at :359-373 does not.

Why it matters

results_verified.json is the file the reporting chain reads. Within this repo, anything downstream
of it inherits the wrong numbers: core/reporter.py:304 feeds pipeline_output.json, which feeds
report/SUMMARY_REPORT.md. On this run that surfaced as "183 vulnerable, 0 errors" for a scan in
which 182 of 223 Stage-2 candidates were never adjudicated and 48 hard-errored.

The direction is the dangerous one for a security scanner in both halves at once: an unverified
candidate is presented as a confirmed vulnerability (over-claim), while the fact that verification
failed at all is erased (errors: 0, no needs_review key). A consumer reading only this file
cannot distinguish "Stage 2 confirmed 183" from "Stage 2 confirmed 1 and could not finish 182".

Suggested fix

Write the buckets that are already computed instead of re-deriving them from verdict strings:

1. In _write_verified_results, classify on the same signals _count_verification_outcomes uses —
   r.get("error") first, then verification.get("incomplete") — before falling through to the
   verdict string. An errored or incomplete unit must not be counted as vulnerable.
2. Add needs_review (and, if useful, stage2_agreed/stage2_disagreed) to the metrics block so
   results_verified.json and scan.report.json carry the same schema and can be reconciled.
3. Keep total = the sum of the buckets, and add a test asserting that invariant on a fixture
   containing one errored and one incomplete result.
4. Consider replacing the recount entirely — have _write_verified_results take the counts from
   _count_verification_outcomes so there is one counter and one definition. If instead the recount
   is kept, its elif r.get("verdict") == "ERROR" branch must be retained and a verify-error
   branch added beside it (see the correction below).

Corrected 2026-08-21. The sentence that stood here called the
elif r.get("verdict") == "ERROR" branch "dead — a branch no production path can reach."
That is false and the branch must NOT be removed. verdict = "ERROR" has production writers
(core/analysis_core.py:47, :180; core/analyzer.py:185). An analyze-errored record evaluates
to the string "error" by either of two routes — core/analyzer.py:185 writes
"finding": "error" explicitly, and core/analysis_core.py:180 returns a dict with
verdict = "ERROR" and no finding key at all, so the or falls through to verdict. Either way
the value is not a key in counts, and the elif fires correctly. The branch is dead only for verify errors, which never set
verdict = "ERROR" — they write verification = {"incomplete": True} and put the message on
result["error"]. It counted 0 in this run because analyze had 0 errors.

The correct change is to ADD a verify-error branch alongside it, not to delete it.

One further scope note, so nobody applies a change this issue does not ask for: leave
confirmed_findings (core/verifier.py:351-356) alone. Filtering incomplete verifications out of
it would strip 182 of the 183 records in confirmed_findings, which is the source of
pipeline_output.findings (175 after caller/callee dedup) and of the disclosure documents —
reporter.py maps incomplete to "unverified", which core/verdict_taxonomy.py deliberately
places in DISCLOSURE_ELIGIBLE (PR #69 F4). Keeping unadjudicated findings disclosure-eligible is
the over-seed-safe behaviour. This issue is a counting defect only; fixing it must not change any
finding's disclosure eligibility.

What I am not claiming

• I am not claiming core/reporter.py:562-570 is fine. It is a separate defect (it folds
  protected into safe and never re-derives vulnerable after dedup) and is filed separately.
  I am claiming that fixing it alone does not fix this.
• I have not verified whether other language targets or other providers produce the same
  verdict/finding shapes on error; the evidence above is one Python run on a direct-Anthropic
  binding.
• The blast-radius statement above is scoped to consumers within this repo. I have not surveyed
  external consumers of results_verified.json.

[gadievron]

Two corrections to my own report, from a verification pass. The core finding reproduces exactly; one
sentence in Suggested fix was wrong, and the fix as written is incomplete.

1. Retraction: the elif is NOT unreachable

I wrote that elif r.get("verdict") == "ERROR" is "a branch no production path can reach". That is
false. verdict = "ERROR" has production writers:

core/analysis_core.py:47     result["verdict"] = "ERROR"
core/analysis_core.py:180    "verdict": "ERROR",
core/analyzer.py:185         "verdict": "ERROR",

An analyze-errored record has no finding key, so the expression evaluates to the string
"error", which is not a key in counts, and the elif fires correctly:

r = {"verdict": "ERROR"}
finding = str(r.get("finding") or r.get("verdict") or "error").lower()   # -> 'error'
finding in counts                                                        # -> False
(finding not in counts) and r.get("verdict") == "ERROR"                  # -> True, elif fires

So the branch works for analyze errors and is dead only for verify errors, which never set
verdict = "ERROR" — they write verification = {"incomplete": True} and put the message on
result["error"]. It counted 0 in this run because analyze had 0 errors
(results.json: SAFE 4844 / PROTECTED 408 / VULNERABLE 223, no ERROR).

The Summary and Evidence sections are unaffected — they are scoped to the verify path throughout.
Only that one sentence over-generalised, and the fix should not delete the elif; it should add a
verify-error branch alongside it.

2. The fix as written would leave the headline number wrong

Fixing metrics alone is not sufficient, because the same verdict-string filter builds
confirmed_findings twelve lines earlier — core/verifier.py:351-356:

        "confirmed_findings": [
            r for r in verified_only
            if str(r.get("finding") or r.get("verdict", "")).lower()
            in ("vulnerable", "bypassable")
        ],

In this run:

len(confirmed_findings) = 183
metrics.vulnerable      = 183
of those 183: incomplete = 182, errored = 48

If only metrics is corrected, results_verified.json ends up carrying metrics.vulnerable: 1
beside confirmed_findings: [183 records] in the same file — and since the disclosure producer reads
the findings list, 175 documents would still be written for unadjudicated units. The internal
contradiction moves rather than closes.

Both sites need the same treatment: classify on the verification state (r.get("error"), then
verification.get("incomplete")) before falling through to the verdict string.

Also worth noting

The finish tool schema at utilities/finding_verifier.py:158-161 does not include "error" in its
verdict enum, so the verify path cannot produce that value even in principle — which makes the
verify-side gap structural rather than incidental.

Everything else in the report reproduced: the four metric blocks, the 183 = 134 incomplete + 48 errored + 1 agreed decomposition, and core/reporter.py:304 being a pure transport
(metrics = experiment.get("metrics", {})).

[gadievron]

Retracting part of my own correction above — it would have caused a security false negative. Please
do not apply the confirmed_findings half.

In that comment I wrote that confirmed_findings (core/verifier.py:351-356) needs "the same
treatment" as the metrics recount — i.e. classify on verification.incomplete before the verdict
string. That is wrong, and the direction is dangerous.

Measured on the run:

confirmed_findings = 183, of which 182 carry verification.incomplete

confirmed_findings is the source of pipeline_output.findings (core/reporter.py:307), and
reporter.py:419-425 maps an incomplete verification to stage2_verdict: "unverified", which
core/verdict_taxonomy.py:74-81 places in DISCLOSURE_ELIGIBLE:

DISCLOSURE_ELIGIBLE = frozenset({
    "confirmed", "agreed", "unverified", "vulnerable", "bypassable", "error",
})

So filtering incomplete verifications out of confirmed_findings would remove 182 of 183 findings
from pipeline_output.json and from the disclosure documents — precisely the failure
core/reporter.py:403-412 records PR #69 F4 as having fixed:

agree=False alone is ambiguous: it can mean "Stage 2 disagreed" OR "Stage 2 could not complete".
Mapping the latter to "rejected" is wrong (verify never rejected) and silently drops it from
disclosures. Map incomplete → "unverified" so it renders distinctly and stays disclosure-eligible
(surfaced for manual review).

Keeping unadjudicated findings disclosure-eligible is the over-seed-safe behaviour, and my comment
mistakenly framed the 175 surviving disclosures as the problem. They are the safe state.

What still stands from the original issue: the metrics recount at verifier.py:359-373 cannot
see errored or incomplete verifications, so errors reports 0 while 48 hard-errored, and
needs_review never reaches the file. That is a counting defect and fixing it changes no
finding's disclosure eligibility.

Corrected scope of the fix: change the metrics buckets only. Leave confirmed_findings alone —
its verdict-string filter is deliberate, and PR #69 F4 is the reason.

I should have checked which direction my proposed change moved detection before proposing it. That is
on me.