feat(calibration): phase-2 backfill — retro reversal labels + raw-context re-fetch (GitHub truth) - #8207
Merged
Merged
Conversation
…w-context re-fetch (GitHub truth) Pass A runs #8166's evaluateSuccessorMatch retrospectively over the phase-1 backfilled close decisions: a bot-closed PR with a merged successor (shared linked issue, or same author reworking a majority of its files, merged inside the 30-day lookback) gets its override row's verdict flipped to reversed with distinct github_successor_scan provenance — the corpus's first organic-shaped negative labels. Pass B re-fetches PR diffs (public repos only) into the phase-1 fired rows' metadata.diff, the exact field the live #8130 capture records, bounded by the same cap. Same discipline as phase 1: pure core + thin IO wrapper, deterministic backfill: ids only (live rows unreachable by construction), idempotent patchers (already-patched rows return null), dry-run default, hard per-run GitHub request budget with a resumable state-file cursor. Advances #8170
…of a 10-page cap The first full production dry-run returned zero matches because page 10 of the sort=updated listing only reached back to July 5 while the closes under scan ended June 22 — a silently truncated listing is indistinguishable from 'no successors'. Depth is now bounded by the run's request budget (hard page ceiling 200 as a backstop), with the boundary condition unchanged. Advances #8170
…a hung remote execute fails loud Advances #8170
295/460 matches on the first full scan is too many to all be reversals: in a duplicate-competition culture a shared-issue match by a different author is usually the gate correctly closing a losing duplicate. The apply decision needs the same-author-rework vs shared-issue-only split. Advances #8170
…A — the definitive label class The operator's correction to the zero-reversals reading: bot/AI-closed PRs HAVE been reopened and merged, but latest-decision-wins erased the earlier close verdicts and the scan skipped merged PRs as 'not a standing close'. A close-verdict PR that itself shows merged_at is a definitive same-PR reversal — no heuristics — labeled under github_same_pr_merged provenance, counted separately from the successor classes in the report. Advances #8170
…imit resilience for phase 2 Production hardening from running the passes for real: - Scan-once/apply-from-plan: the dry-run emits its matches as a plan file (persisted even on mid-run abort), and --apply --plan-in replays it against any store with ZERO GitHub requests — the cloud D1 and selfhost Postgres applies share one scan instead of re-spending ~2.7k requests each. --pg rides #8171's driver seam for the store side. - Apply policy is class-gated: same_pr_merged + same_author apply by default; shared_issue_only (128 of the 301 production matches — routine duplicate competition, not reversal evidence) is counted and planned but never applied without an explicit --include-shared-issue-only. - GitHub IO survives reality: bounded retries on thrown fetches/5xx, Retry-After honored on 403/429 burst limits (90s default, 5m cap) with ~4 req/s pacing so a stall costs minutes instead of the whole scan, and the wrangler reads are time-bounded so a hung remote execute fails loud. Advances #8170
…rator's token pool is shared Advances #8170
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8207 +/- ##
=======================================
Coverage 92.09% 92.09%
=======================================
Files 773 773
Lines 78149 78149
Branches 23618 23618
=======================================
Hits 71968 71968
Misses 5062 5062
Partials 1119 1119
Flags with carried forward coverage won't be shown. Click here to find out more. |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The #8170 passes over the phase-1 corpus, built pure-core/thin-IO like phase 1 and hardened by running them against production for real.
Pass A — retro reversal labels, three classes
same_pr_merged(definitive): a close-verdict PR that GitHub shows MERGED — the operator reopened and merged it. The operator's own correction to the 'zero reversals' reading: latest-decision-wins had erased these. No heuristics involved.same_author(strong): calibration: detect one-shot-culture reversals — bot-closed PR superseded by a merged successor (reversal_superseded) #8166'sevaluateSuccessorMatchrun retrospectively — the one-shot close of work its author then landed (≥50% file overlap, ≤30d), imported from the live wire, never re-implemented.shared_issue_only(excluded by default): different-author shared-issue matches are routine duplicate competition, not bot-was-wrong evidence. Counted and planned, applied only under an explicit--include-shared-issue-only.Production dry-run (posted on the issue): 460 scanned → 6 definitive + 167 same-author applied-class labels; 128 shared-issue-only excluded.
Pass B — raw-context re-fetch
Public repos only; patches phase-1 fired rows with the PR diff the live #8130 capture records (
metadata.diff, same bound), distinct provenance, idempotent.Operational hardening (each learned from a real failure)
--apply --plan-inreplays against any store with zero GitHub requests — cloud D1 and the selfhost Postgres (--pg, calibration: self-host Postgres parity — verify every calibration surface on the pg shim + pg-capable corpus/track-record/backfill CLIs #8171's driver) share one scan.backfill:id prefix are ever touched — live capture rows are unreachable by construction.Validation
Full
npm run test:cigreen on this branch (TEST_CI_EXIT=0, zero failed files). Pure core 100% line+branch (11 tests). Wrapper is thin-IO (scripts exemption), exercised against production D1 and the live ORB Postgres.Closes #8170