fix(report): "deduplicated" is a true dedup delta — never negative (#423) - #465
Merged
Conversation
) `vulnerable` is re-derived from the DEDUPED findings list (#289), but `vulnerable_before_dedup` came from the ANALYZE-stage metrics (metrics.vulnerable + metrics.bypassable) — a DIFFERENT population. A live 25-unit directed scan with --verify emitted `"vulnerable": 2, "vulnerable_before_dedup": 0, "deduplicated": -2`: detect classified 0 vulnerable, Stage-2 adjudication disagreed on 2 that stayed vulnerable per the verifier taxonomy, the two populations diverged, and the #289/#381 reconciliation contract (before >= after) broke — a negative "dedup" count that can never be explained as deduplication. The fix (the issue's primary suggestion): `vulnerable_before_dedup` counts the SAME population `vulnerable` counts — the confirmed findings list's PRE-dedup length, captured right before `_dedup_caller_callee` — so the delta is a true dedup delta. Stage-2 reclassifications no longer produce negative numbers; a genuine caller/callee collapse (same CWE, callee reachable only via the caller) still reports the exact number dropped. Both the confirmed_findings path and the manual final-verdict filter path feed the same list, so the rule holds on both. Evidence: 3 tests (RED 1 on pristine — the live-run shape emitting before=0/dedup=-2; the true-dedup and manual-filter cases coincidentally agreed on pristine and now pin the fixed population rule) — the Stage-2-only shape (before=2, after=2, delta=0), a real collapse (before=2, after=1, delta=1), and the manual filter path. The report/dedup/289/reconcile families 262 passed / 1 (the known macOS /private/var tmp artifact, byte-identical on pristine); the full suite 3519 passed / 1 failed (same artifact); ruff clean; hermeticity green. Refs #423
…he dedup delta stays true (#423, wave r1) Three axes, one verdict: the pre-round fix traded the visible negative `deduplicated` for a SILENT over-summed partition — the verifier''s two populations overlap BY DESIGN (verifier.py''s #284 note keeps errored/incomplete rows whose Stage-1 finding is vulnerable in confirmed_findings; the recount buckets those same rows into errors/needs_review first), so counting `vulnerable` from the disclosure list while `errors`/`needs_review` came from metrics double-counted the overlap — on the live run''s own shape the sum went 2+23+2=27 for a 25-unit scan, and on the #289 run''s 182 errored+incomplete retained rows it is ~182 units of over-count. The negative `-2` was a mislabeled closure term holding the partition together. The fix: the overlap rows are counted ONCE (in `vulnerable`, via the findings list — the #289 contract unchanged) and their metrics-bucket entries are subtracted — `_k_overlap_error` / `_k_overlap_needs` computed from the confirmed list with the recount''s own predicates (r.get("error") / verification.incomplete). The dedup pair stays a true delta over ONE population (the #423 contract unchanged), and the F13 sum (vulnerable + deduplicated + safe + protected + inconclusive + errors + needs_review == total) reconciles on the divergence shape. The RED fixture was also unproducible (wave r1 opus): its metrics summed to 23 against total 25 — _write_verified_results partitions every unit, so real metrics always sum to total; the faithful shape (the divergence rows in needs_review) makes the old fix''s over-sum VISIBLE — the new test asserts both contracts and the reconciliation on it. Evidence: the faithful live-run fixture asserting before=2/after=2/delta=0 AND the sum == 25 AND needs_review reduced by the overlap (0, not 2); the #289/F13 reconciliation suites 12 passed together; the full suite green (3528+ passed / 1 known macOS artifact); ruff clean.
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
September 1, 2026 03:35
…nel round 1 Sonnet family-panel: the errors/needs_review overlap-subtraction (a confirmed row that ALSO carries the error/incomplete marker — counted once in vulnerable, subtracted from its metrics bucket) was completely untested. The new test drives the real build_pipeline_output with the exact live-run shape (a disclosure-listed vulnerable row whose verification errored): vulnerable=1, errors=0 after the subtraction, the partition reconciling to total exactly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vulnerableis re-derived from the DEDUPED findings list (#289), butvulnerable_before_dedupcame from the ANALYZE-stage metrics — a DIFFERENT population. A live 25-unit directed scan with --verify emitted"vulnerable": 2, "vulnerable_before_dedup": 0, "deduplicated": -2: detect classified 0 vulnerable, Stage-2 adjudication disagreed on 2 that stayed vulnerable per the verifier taxonomy, the two populations diverged, and the #289/#381 reconciliation contract (before >= after) broke — a negative "dedup" count that can never be explained as deduplication.The fix (base commit):
vulnerable_before_dedupcounts the SAME list's PRE-dedup length (captured right before_dedup_caller_callee) — the delta is a true dedup delta. Holds on both the confirmed_findings path and the manual final-verdict filter path (both feed the one list).The wave-r1 round (three axes, one verdict): the round-1 fix traded the visible negative for a SILENT over-summed partition — the verifier's two populations overlap BY DESIGN (verifier.py's #284 note keeps errored/incomplete rows whose Stage-1 finding is vulnerable in
confirmed_findings; the recount buckets those same rows intoerrors/needs_reviewfirst), so countingvulnerablefrom the disclosure list whileerrors/needs_reviewcame from metrics double-counted the overlap — on the live run's own shape the sum went 2+23+2=27 for a 25-unit scan (~182 units of over-count on the #289 run's 182 errored+incomplete retained rows). The negative-2was a mislabeled closure term holding the partition together.The final fix: the overlap rows are counted ONCE (in
vulnerable, via the findings list — the #289 contract unchanged) and their metrics-bucket entries are subtracted —_k_overlap_error/_k_overlap_needscomputed from the confirmed list with the recount's own predicates (r.get("error")/verification.incomplete). The dedup pair stays a true delta over ONE population; the F13 sum (vulnerable + deduplicated + safe + protected + inconclusive + errors + needs_review == total) reconciles on the divergence shape.The RED fixture was also unproducible (wave r1 opus): its metrics summed to 23 against total 25 —
_write_verified_resultspartitions every unit, so real metrics always sum to total; the faithful shape (the divergence rows inneeds_review) makes the over-sum VISIBLE — the test asserts both contracts and the reconciliation on it.Evidence: the faithful live-run fixture asserting before=2/after=2/delta=0 AND the sum == 25 AND
needs_reviewreduced by the overlap (0, not 2); the #289/F13 reconciliation suites 12 passed together; the full suite green (1 known macOS artifact); ruff clean.Fixes #423