fix(webhook): self-heal PR state drift from missed merge webhooks#178
Merged
Conversation
PR state (OPEN -> MERGED/CLOSED) was written only by the pull_request webhook handler. A single missed/dropped `pull_request.closed` delivery left a merged PR stuck OPEN forever: the frequent metadata-fetch path deliberately never touched state, there was no reconciliation, and backfill only ran on manual trigger. Observed on phase-rs/phase (#2745/2751/2752/2753/2756/3095 merged on GitHub, OPEN in the mirror), which makes the validator under-credit the miner since it reads PR state verbatim from the mirror. - Fetcher: fetchPrMetadata now also returns authoritative GraphQL state/mergedAt/closedAt/mergedBy. - Metadata handler: re-asserts that state (MERGED is terminal, so an in-flight stale fetch can't revert it) and logs corrected drift. - PrReconcileService: hourly sweep enqueues a metadata fetch for every still-open PR in registered repos within the scoring window, so missed merge events self-heal. Window/interval env-tunable. - RepoBackfillScheduleService: daily full backfill per registered repo as a coarse safety net; env kill-switch (NIGHTLY_BACKFILL_ENABLED=false). - Webhook handler: derive merged state from `merged` alone (synthesize merged_at from closed_at) so a closed event with a not-yet-populated merged_at isn't pinned to OPEN.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A merged PR was showing as OPEN in the mirror (and therefore on the dashboard, and therefore mis-scored by the validator). Confirmed on
phase-rs/phasefor miner156195510: PRs #2745, #2751, #2752, #2753, #2756, #3095 are MERGED on GitHub (with merge commits) but sit atstate=OPEN, merged_at=NULLin the mirror DB.Root cause
PR
state/merged_atis written by exactly one runtime path — thepull_requestwebhook handler. If thepull_request.closed(merged) delivery is missed/dropped, nothing ever corrects it:handlePrMetadataruns frequently (everysynchronize/edited/closed) but only wroteclosingIssueNumbers/body/lastEditedAt— neverstate. (Logs show #3095 metadata-fetched 9× on 06-12, the last 2s after its merge, yet it stayed OPEN.)So a single missed merge webhook = permanent false-OPEN. The validator reads PR state verbatim from the mirror, so the miner is genuinely under-credited (those PRs never enter
merged_prs).Changes
github-fetcher.ts—fetchPrMetadatanow also returns authoritative GraphQLstate/mergedAt/closedAt/mergedBy. (GraphQLstateis the source of truth:OPEN/CLOSED/MERGED.)fetch.processor.ts—handlePrMetadatare-asserts that state.MERGEDis treated as terminal so an in-flight stale fetch can't revert it; logs aState drift corrected …warning whenever it flips a PR (observability — this drift was previously invisible).pr-reconcile.service.ts(new) — hourly sweep enqueues a metadata fetch for every still-open PR in registered repos within the scoring window (PR_RECONCILE_WINDOW_DAYS, default 45). This is the actual self-heal for missed merge webhooks. Runs once at startup too. Interval/window env-tunable.repo-backfill-schedule.service.ts(new) — daily full backfill per registered repo as a coarse safety net (also catches issue/label drift). Kill-switch:NIGHTLY_BACKFILL_ENABLED=false. Heavier than the sweep, hence daily + disableable.pull-request.handler.ts— derive merged state frompr.mergedalone (synthesizemerged_atfromclosed_atif absent), so aclosedevent arriving beforemerged_atis populated isn't pinned to OPEN..env.example— documents the new tunables.Layering
pr-reconcile(cheap, hourly, open-PRs-only) is the primary fix; nightly backfill is belt-and-suspenders. Both reuse existing queue jobs (meta-…jobId dedupes against webhook-triggered fetches; concurrency-5 worker provides backpressure).Effect on the stuck PRs
Once deployed, the startup + hourly reconcile sweep re-fetches those 6 PRs, flips them to MERGED, and the next validator round (~2h) credits the miner. (A one-time manual
admin/backfillforphase-rs/phasestill recommended to correct them immediately rather than waiting for the first sweep.)Verification
npm run build,npm run lint,prettier --checkall pass locally. No existing tests reference the changed APIs.