Context
Found during the 2026-07-06 incident audit (parent: #1667). Two contributor PRs (#3782, #3793 on JSONbored/gittensory) had literally zero audit_events ever recorded, and no row in the local pull_requests table, until an operator manually forced a requestedBy: "schedule" sweep (whose open-PR discovery/backfill side effect happened to surface them) roughly two hours after they were opened. Their "PR opened" webhook silently vanished with no error, no audit event, no trace anywhere.
Confirmed: src/github/webhook.ts increments a receipt-time metric but never calls recordAuditEvent; the durable audit trail only starts deep inside src/queue/processors.ts, well after a job is claimed and reaches a decision point. The only mechanism that would organically re-discover a lost PR — the scheduled backfill-registered-repos job — skips any repo whose last sync was "fresh" (FRESH_SYNC_MS = 6 hours), so once a repo has any successful sync, the job won't re-diff its open-PR list against GitHub for up to 6 hours regardless of how many webhooks silently drop in the meantime. There is no dedicated, fast reconciliation or alert for "local open-PR list has diverged from GitHub's actual list" independent of that slow opportunistic sync.
Requirements
- Add a lightweight, frequent (5-15 min — much tighter than the 6h backfill freshness window) reconciliation check per actively-reviewed repo: fetch GitHub's current open PR numbers (a cheap list call, not a full detail sync) and diff against the local
pull_requests table.
- When a PR number appears in GitHub's list but has no local row, emit a structured Sentry-forwarded error and immediately enqueue a targeted backfill/upsert for just that PR — do not wait for the 6h freshness window.
- Add a gauge/counter for open-PR count divergence per repo, visible on a dashboard.
- Scope this to repos with a real GitHub App installation and acting/observe autonomy (mirroring
fanOutAgentRegateSweepJobs's own repo-selection logic), not the unrelated ecosystem-registry isRegistered flag.
- Separately: ensure
enqueueWebhookByEnv (src/github/webhook.ts) writes a webhook_events row before JSON.parse can drop an unparseable delivery with zero trace, and that the Orb relay's pull-mode drain loop (src/orb/broker-client.ts) logs + counts any malformed event it silently filters out of a batch, instead of discarding with no record. These are adjacent zero-trace loss points on the same ingestion path.
Deliverables
- New job type (e.g.
reconcile-open-prs) + processor, fanned out per repo on a short cron interval.
- Lightweight list-open-PR-numbers helper (distinct from the full detail backfill).
- New metric(s) + structured error log on divergence.
src/github/webhook.ts: initial recordWebhookEvent call before JSON.parse.src/orb/broker-client.ts: log + counter on malformed relay event filtering.- Unit tests for divergence detection, the invalid-JSON trace, and the malformed-relay-event trace.
Expected outcome
A PR whose opening webhook is silently lost is automatically detected and backfilled within minutes — not up to 6+ hours, and never requiring a human to notice review silence and manually query Postgres for hours, as happened with #3782/#3793 today.
Context
Found during the 2026-07-06 incident audit (parent: #1667). Two contributor PRs (#3782, #3793 on JSONbored/gittensory) had literally zero audit_events ever recorded, and no row in the local
pull_requeststable, until an operator manually forced arequestedBy: "schedule"sweep (whose open-PR discovery/backfill side effect happened to surface them) roughly two hours after they were opened. Their "PR opened" webhook silently vanished with no error, no audit event, no trace anywhere.Confirmed:
src/github/webhook.tsincrements a receipt-time metric but never callsrecordAuditEvent; the durable audit trail only starts deep insidesrc/queue/processors.ts, well after a job is claimed and reaches a decision point. The only mechanism that would organically re-discover a lost PR — the scheduledbackfill-registered-reposjob — skips any repo whose last sync was "fresh" (FRESH_SYNC_MS = 6 hours), so once a repo has any successful sync, the job won't re-diff its open-PR list against GitHub for up to 6 hours regardless of how many webhooks silently drop in the meantime. There is no dedicated, fast reconciliation or alert for "local open-PR list has diverged from GitHub's actual list" independent of that slow opportunistic sync.Requirements
pull_requeststable.fanOutAgentRegateSweepJobs's own repo-selection logic), not the unrelated ecosystem-registryisRegisteredflag.enqueueWebhookByEnv(src/github/webhook.ts) writes awebhook_eventsrow beforeJSON.parsecan drop an unparseable delivery with zero trace, and that the Orb relay's pull-mode drain loop (src/orb/broker-client.ts) logs + counts any malformed event it silently filters out of a batch, instead of discarding with no record. These are adjacent zero-trace loss points on the same ingestion path.Deliverables
reconcile-open-prs) + processor, fanned out per repo on a short cron interval.src/github/webhook.ts: initialrecordWebhookEventcall beforeJSON.parse.src/orb/broker-client.ts: log + counter on malformed relay event filtering.Expected outcome
A PR whose opening webhook is silently lost is automatically detected and backfilled within minutes — not up to 6+ hours, and never requiring a human to notice review silence and manually query Postgres for hours, as happened with #3782/#3793 today.