Skip to content

fix(webhook): self-heal PR state drift from missed merge webhooks#178

Merged
LandynDev merged 1 commit into
testfrom
fix/pr-state-reconciliation
Jun 14, 2026
Merged

fix(webhook): self-heal PR state drift from missed merge webhooks#178
LandynDev merged 1 commit into
testfrom
fix/pr-state-reconciliation

Conversation

@LandynDev

Copy link
Copy Markdown
Collaborator

Problem

A merged PR was showing as OPEN in the mirror (and therefore on the dashboard, and therefore mis-scored by the validator). Confirmed on phase-rs/phase for miner 156195510: PRs #2745, #2751, #2752, #2753, #2756, #3095 are MERGED on GitHub (with merge commits) but sit at state=OPEN, merged_at=NULL in the mirror DB.

Root cause

PR state/merged_at is written by exactly one runtime path — the pull_request webhook handler. If the pull_request.closed (merged) delivery is missed/dropped, nothing ever corrects it:

  1. handlePrMetadata runs frequently (every synchronize/edited/closed) but only wrote closingIssueNumbers/body/lastEditedAt — never state. (Logs show #3095 metadata-fetched 9× on 06-12, the last 2s after its merge, yet it stayed OPEN.)
  2. No reconciliation existed — the only periodic job was webhook-delivery pruning.
  3. Backfill only ran on manual admin trigger / repo register.

So a single missed merge webhook = permanent false-OPEN. The validator reads PR state verbatim from the mirror, so the miner is genuinely under-credited (those PRs never enter merged_prs).

Changes

  • github-fetcher.tsfetchPrMetadata now also returns authoritative GraphQL state/mergedAt/closedAt/mergedBy. (GraphQL state is the source of truth: OPEN/CLOSED/MERGED.)
  • fetch.processor.tshandlePrMetadata re-asserts that state. MERGED is treated as terminal so an in-flight stale fetch can't revert it; logs a State drift corrected … warning whenever it flips a PR (observability — this drift was previously invisible).
  • pr-reconcile.service.ts (new) — hourly sweep enqueues a metadata fetch for every still-open PR in registered repos within the scoring window (PR_RECONCILE_WINDOW_DAYS, default 45). This is the actual self-heal for missed merge webhooks. Runs once at startup too. Interval/window env-tunable.
  • repo-backfill-schedule.service.ts (new) — daily full backfill per registered repo as a coarse safety net (also catches issue/label drift). Kill-switch: NIGHTLY_BACKFILL_ENABLED=false. Heavier than the sweep, hence daily + disableable.
  • pull-request.handler.ts — derive merged state from pr.merged alone (synthesize merged_at from closed_at if absent), so a closed event arriving before merged_at is populated isn't pinned to OPEN.
  • .env.example — documents the new tunables.

Layering

pr-reconcile (cheap, hourly, open-PRs-only) is the primary fix; nightly backfill is belt-and-suspenders. Both reuse existing queue jobs (meta-… jobId dedupes against webhook-triggered fetches; concurrency-5 worker provides backpressure).

Effect on the stuck PRs

Once deployed, the startup + hourly reconcile sweep re-fetches those 6 PRs, flips them to MERGED, and the next validator round (~2h) credits the miner. (A one-time manual admin/backfill for phase-rs/phase still recommended to correct them immediately rather than waiting for the first sweep.)

Verification

npm run build, npm run lint, prettier --check all pass locally. No existing tests reference the changed APIs.

PR state (OPEN -> MERGED/CLOSED) was written only by the pull_request
webhook handler. A single missed/dropped `pull_request.closed` delivery
left a merged PR stuck OPEN forever: the frequent metadata-fetch path
deliberately never touched state, there was no reconciliation, and
backfill only ran on manual trigger. Observed on phase-rs/phase
(#2745/2751/2752/2753/2756/3095 merged on GitHub, OPEN in the mirror),
which makes the validator under-credit the miner since it reads PR state
verbatim from the mirror.

- Fetcher: fetchPrMetadata now also returns authoritative GraphQL
  state/mergedAt/closedAt/mergedBy.
- Metadata handler: re-asserts that state (MERGED is terminal, so an
  in-flight stale fetch can't revert it) and logs corrected drift.
- PrReconcileService: hourly sweep enqueues a metadata fetch for every
  still-open PR in registered repos within the scoring window, so missed
  merge events self-heal. Window/interval env-tunable.
- RepoBackfillScheduleService: daily full backfill per registered repo as
  a coarse safety net; env kill-switch (NIGHTLY_BACKFILL_ENABLED=false).
- Webhook handler: derive merged state from `merged` alone (synthesize
  merged_at from closed_at) so a closed event with a not-yet-populated
  merged_at isn't pinned to OPEN.
@LandynDev
LandynDev merged commit b5c1c3f into test Jun 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant