Skip to content

fix(report): "deduplicated" is a true dedup delta — never negative (#423) - #465

Merged
gadievron merged 3 commits into
masterfrom
fix/issue423-dedup-reconcile
Sep 1, 2026
Merged

fix(report): "deduplicated" is a true dedup delta — never negative (#423)#465
gadievron merged 3 commits into
masterfrom
fix/issue423-dedup-reconcile

Conversation

@gadievron

Copy link
Copy Markdown
Collaborator

vulnerable is re-derived from the DEDUPED findings list (#289), but vulnerable_before_dedup came from the ANALYZE-stage metrics — a DIFFERENT population. A live 25-unit directed scan with --verify emitted "vulnerable": 2, "vulnerable_before_dedup": 0, "deduplicated": -2: detect classified 0 vulnerable, Stage-2 adjudication disagreed on 2 that stayed vulnerable per the verifier taxonomy, the two populations diverged, and the #289/#381 reconciliation contract (before >= after) broke — a negative "dedup" count that can never be explained as deduplication.

The fix (base commit): vulnerable_before_dedup counts the SAME list's PRE-dedup length (captured right before _dedup_caller_callee) — the delta is a true dedup delta. Holds on both the confirmed_findings path and the manual final-verdict filter path (both feed the one list).

The wave-r1 round (three axes, one verdict): the round-1 fix traded the visible negative for a SILENT over-summed partition — the verifier's two populations overlap BY DESIGN (verifier.py's #284 note keeps errored/incomplete rows whose Stage-1 finding is vulnerable in confirmed_findings; the recount buckets those same rows into errors/needs_review first), so counting vulnerable from the disclosure list while errors/needs_review came from metrics double-counted the overlap — on the live run's own shape the sum went 2+23+2=27 for a 25-unit scan (~182 units of over-count on the #289 run's 182 errored+incomplete retained rows). The negative -2 was a mislabeled closure term holding the partition together.

The final fix: the overlap rows are counted ONCE (in vulnerable, via the findings list — the #289 contract unchanged) and their metrics-bucket entries are subtracted — _k_overlap_error / _k_overlap_needs computed from the confirmed list with the recount's own predicates (r.get("error") / verification.incomplete). The dedup pair stays a true delta over ONE population; the F13 sum (vulnerable + deduplicated + safe + protected + inconclusive + errors + needs_review == total) reconciles on the divergence shape.

The RED fixture was also unproducible (wave r1 opus): its metrics summed to 23 against total 25 — _write_verified_results partitions every unit, so real metrics always sum to total; the faithful shape (the divergence rows in needs_review) makes the over-sum VISIBLE — the test asserts both contracts and the reconciliation on it.

Evidence: the faithful live-run fixture asserting before=2/after=2/delta=0 AND the sum == 25 AND needs_review reduced by the overlap (0, not 2); the #289/F13 reconciliation suites 12 passed together; the full suite green (1 known macOS artifact); ruff clean.

Fixes #423

)

`vulnerable` is re-derived from the DEDUPED findings list (#289), but
`vulnerable_before_dedup` came from the ANALYZE-stage metrics
(metrics.vulnerable + metrics.bypassable) — a DIFFERENT population. A live 25-unit
directed scan with --verify emitted `"vulnerable": 2, "vulnerable_before_dedup": 0,
"deduplicated": -2`: detect classified 0 vulnerable, Stage-2 adjudication disagreed
on 2 that stayed vulnerable per the verifier taxonomy, the two populations diverged,
and the #289/#381 reconciliation contract (before >= after) broke — a negative "dedup"
count that can never be explained as deduplication.

The fix (the issue's primary suggestion): `vulnerable_before_dedup` counts the SAME
population `vulnerable` counts — the confirmed findings list's PRE-dedup length,
captured right before `_dedup_caller_callee` — so the delta is a true dedup delta.
Stage-2 reclassifications no longer produce negative numbers; a genuine caller/callee
collapse (same CWE, callee reachable only via the caller) still reports the exact
number dropped. Both the confirmed_findings path and the manual final-verdict filter
path feed the same list, so the rule holds on both.

Evidence: 3 tests (RED 1 on pristine — the live-run shape emitting before=0/dedup=-2;
the true-dedup and manual-filter cases coincidentally agreed on pristine and now pin
the fixed population rule) — the Stage-2-only shape (before=2, after=2, delta=0), a
real collapse (before=2, after=1, delta=1), and the manual filter path. The
report/dedup/289/reconcile families 262 passed / 1 (the known macOS /private/var tmp
artifact, byte-identical on pristine); the full suite 3519 passed / 1 failed (same
artifact); ruff clean; hermeticity green.

Refs #423
…he dedup delta stays true (#423, wave r1)

Three axes, one verdict: the pre-round fix traded the visible negative
`deduplicated` for a SILENT over-summed partition — the verifier''s two
populations overlap BY DESIGN (verifier.py''s #284 note keeps errored/incomplete
rows whose Stage-1 finding is vulnerable in confirmed_findings; the recount
buckets those same rows into errors/needs_review first), so counting
`vulnerable` from the disclosure list while `errors`/`needs_review` came from
metrics double-counted the overlap — on the live run''s own shape the sum went
2+23+2=27 for a 25-unit scan, and on the #289 run''s 182 errored+incomplete
retained rows it is ~182 units of over-count. The negative `-2` was a
mislabeled closure term holding the partition together.

The fix: the overlap rows are counted ONCE (in `vulnerable`, via the findings
list — the #289 contract unchanged) and their metrics-bucket entries are
subtracted — `_k_overlap_error` / `_k_overlap_needs` computed from the
confirmed list with the recount''s own predicates (r.get("error") /
verification.incomplete). The dedup pair stays a true delta over ONE
population (the #423 contract unchanged), and the F13 sum
(vulnerable + deduplicated + safe + protected + inconclusive + errors +
needs_review == total) reconciles on the divergence shape.

The RED fixture was also unproducible (wave r1 opus): its metrics summed to
23 against total 25 — _write_verified_results partitions every unit, so real
metrics always sum to total; the faithful shape (the divergence rows in
needs_review) makes the old fix''s over-sum VISIBLE — the new test asserts
both contracts and the reconciliation on it.

Evidence: the faithful live-run fixture asserting before=2/after=2/delta=0
AND the sum == 25 AND needs_review reduced by the overlap (0, not 2); the
#289/F13 reconciliation suites 12 passed together; the full suite green
(3528+ passed / 1 known macOS artifact); ruff clean.
…nel round 1

Sonnet family-panel: the errors/needs_review overlap-subtraction (a
confirmed row that ALSO carries the error/incomplete marker — counted
once in vulnerable, subtracted from its metrics bucket) was completely
untested. The new test drives the real build_pipeline_output with the
exact live-run shape (a disclosure-listed vulnerable row whose
verification errored): vulnerable=1, errors=0 after the subtraction, the
partition reconciling to total exactly.
@gadievron
gadievron merged commit 1cfb5fc into master Sep 1, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

results.deduplicated can go NEGATIVE: vulnerable_before_dedup reads Stage-1 metrics while vulnerable reads post-verify findings

1 participant