feat(github): re-plan sibling PR checks when an apply changes a target schema - #866
feat(github): re-plan sibling PR checks when an apply changes a target schema#866aparajon wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a durable “check refresh” guardrail so that when an apply successfully changes a target’s live schema, SchemaBot re-plans (or fail-closes) other open PRs whose stored plan checks were computed against the old schema—covering PR, CLI/gRPC, and rollback-driven applies.
Changes:
- Introduces
check_refresh_requestsoutbox + processor that leases, heartbeats, coalesces same-target requests, and fans out re-plans to sibling PRs (fail-closed on re-plan failure with a sanitized message). - Records refresh requests from the operator drive tail on terminal success, plus a backstop sweep for crash gaps.
- Extends stored plan-check upsert to support refresh attribution notes, adds target-wide check lookup/indexing, and adds metrics + integration tests.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/webhook/vschema_only_check_integration_test.go | Updates test calls for new upsertPlanCheckRecord signature. |
| pkg/webhook/handler.go | Adds check refresh processor lifecycle fields and defaults. |
| pkg/webhook/check_runs.go | Adds a fixed fail-closed blocking reason/message for refresh re-plan failures. |
| pkg/webhook/check_refresh.go | Implements the check refresh processor (sweep, lease/heartbeat, fan-out, fail-closed path). |
| pkg/webhook/check_refresh_integration_test.go | End-to-end integration tests for drive-tail recording, sweep backfill, fan-out, fail-closed behavior, in-flight guard, and coalescing. |
| pkg/webhook/check_records.go | Adds refresh attribution support and introduces errPlanCheckHeadStale for racing-head detection. |
| pkg/storage/types.go | Adds CheckRefreshRequest type + constants and max-attempts budget. |
| pkg/storage/storage.go | Extends storage interfaces with CheckRefreshRequestStore, CheckStore.GetByTarget, and MarkBlockedForFailedRefresh. |
| pkg/storage/mysqlstore/storage.go | Wires MySQL storage to expose CheckRefreshRequests(). |
| pkg/storage/mysqlstore/checks.go | Implements target-wide check lookup and fail-closed update with head-SHA + in-flight guards. |
| pkg/storage/mysqlstore/check_refresh_requests.go | Implements durable request recording/claiming/heartbeat/completion/failure/sweep queries. |
| pkg/storage/mysqlstore/check_refresh_requests_test.go | Integration tests for request leasing semantics, retry/terminal behavior, sweep selection, coalescing, and check-store helpers. |
| pkg/storage/errors.go | Adds durable check refresh request lease/not-found errors. |
| pkg/serve/serve.go | Starts/stops the check refresh processor alongside other server background work. |
| pkg/schema/mysql/checks.sql | Adds an index to support target-wide check lookups. |
| pkg/schema/mysql/check_refresh_requests.sql | Adds the new durable check_refresh_requests table. |
| pkg/metrics/metrics.go | Adds metrics for recording sources, record failures, PR fan-out outcomes, and request-level outcomes/terminations. |
| pkg/api/operator.go | Records refresh requests from operator terminal-success transitions (before control-request cleanup). |
| pkg/api/handlers_test.go | Updates mock storage to satisfy the extended storage interface. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
2ba0f41 to
dd294aa
Compare
705ec9a to
b054f62
Compare
b054f62 to
fd47dab
Compare
02dbe05 to
6ba60ea
Compare
…t schema An apply reaching terminal success mutates its target's live schema, staling the stored plan check state of every other open PR against that (environment, database type, database) target — acute for CLI applies with no PR surface, where a stale green check is a tier-0 safety gate failure. The webhook processor now drains the durable check refresh requests the drive tails record: each pass sweeps for completed applies missing a request, terminates requests wedged past the attempt cap, then claims pending requests under a lease and fans out. The fan-out re-plans each sibling PR's stored check at its current head with an attribution note naming the apply and caller, skipping the originating PR, in-flight apply rows (a started apply stays authoritative), closed PRs, targets the PR no longer manages, and rows a newer head already superseded. A failed re-plan flips the stored check to a blocking schema_changed_replan_failed conclusion — fail closed — and the request still completes because the block is durable. Same-target pending requests coalesce into one fan-out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… a request Without a wake-up, a request recorded right after a poll tick waits the full interval before sibling PR checks move. The handler now registers KickCheckRefresh as the service's recorded-notifier: the drive tail's call lands on a buffered channel the driver selects on alongside its ticker, so a co-located processor drains the request immediately. The durable request row stays the source of truth — a kick lost to a pod boundary or a stopped driver only costs poll latency, never the refresh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The processor consumes merge gate requests by their code-host neutral change identity: the sweep records change_key via ChangeKeyForPullRequest and the fan-out's originator skip is a named predicate that converts the stored check's PR number at the comparison boundary. Empty change keys (CLI/gRPC applies) exclude nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fd47dab to
6bc0242
Compare
6ba60ea to
56faf88
Compare
|
🤖 Review findings - created by Kiran's code review agent - for pull/866, 6bc0242. Verdict: 9 findings — 3 blocking (fan-out panics escape the recover, coalescing drops the originator re-plan, dead stale-head guard), 4 non-blocking, 2 suggestions. Blocking
Non-blocking
General suggestions
The one thing that could have broken, verifiedThe fan-out's concurrent stored-check writes racing other writers on the same row — in-flight applies, synchronize auto-plans, other merge-gate drivers. The in-flight-apply race is proven safe at the storage layer, not the snapshot layer: both UpsertPlanResult UPDATE branches carry Verified correct
This review was generated by Claude Code (claude-fable-5). |
morgo
left a comment
There was a problem hiding this comment.
🤖 Approving on Morgan's behalf (automated review).
Correction I owe this PR first: it is green, and an earlier automated sweep of mine had it recorded as failing. The three FAILURE entries are aggregate gate jobs from a run that was cancelled wholesale — their matrix dependencies were cancelled, not failed — and the live run is green on those same names, with mergeStateStatus: CLEAN. My sweep counted non-SUCCESS as failure, which is why this sat from 2026-08-08 without a verdict from me. That was my error, not a CI problem.
+1300/-14 reads worse than it is: 360 lines are tests, and the bulk is merge_gate.go arriving as a new 700-line file. I'd already audited that file's machinery at the stack tip, so rather than re-reading it I diffed this revision against the audited one: the function set here is a strict subset, and the shared bodies are textually identical apart from doc comments, one import, a metric constant, and the later sweeps. So the audit transfers, and it covers the parts that matter — exactly two writes of MergeGateCompleted (lease-guarded MarkCompleted after a successful fan-out, and pending-guarded CompletePendingCoalesced on same-kind siblings captured before the fan-out began), TerminateStuckProcessing terminalizing as Failed rather than completed, a heartbeat failure leaving the row for reclaim instead of completing work it can't prove it owns, panic recovery routed through the attempt cap so a poison request can't crash-loop every replica, and a failed coalesce degrading to a redundant-but-safe re-plan.
The lifecycle wiring is the part I'd expect to go wrong, and it doesn't. startMergeGateProcessor/stopMergeGateProcessor are nil-guarded, registered on both the single-app and multi-app runtimes (a missed path there would silently disable the gate for one deployment shape), and stopped in Close. The processor also starts before StartOperator. That ordering isn't load-bearing in this PR, but it becomes so once #941 keys its apply-start gate on hasMergeGateConsumer() — an operator started first could drive an apply that sees no consumer and skips the gate entirely. Worth a comment pinning the order deliberately, since right now it reads as incidental.
I verified the sanitization claim rather than taking it. mergeGateNote's comment says RequestedBy is caller-influenced so the note is "sanitized for markdown-table rendering," and the only call is clampDriftSummary — a name that sounds like pure truncation. It isn't: it collapses \n and \r, rewrites | to /, and truncates on a rune boundary. So a crafted username can't break the aggregate's Change column. The finding is the name. The function's contract is "sanitize and clamp" while it advertises "clamp," and it's now called from three places that depend on the sanitizing half. Someone simplifying it to a truncation, or writing a new summary path that reaches for a plain clamp instead, reintroduces the injection with nothing failing. Renaming it to say what it guarantees is cheap.
One thing I expected to be a behavior change and isn't, worth recording so the next reviewer doesn't re-flag it: the plan-head staleness check in upsertPlanCheckRecord already existed. This PR only wraps the existing error with %w and the new errPlanCheckHeadStale sentinel so the fan-out can treat a racing synchronize as a benign skip. Ordinary plan writes behave exactly as before. That %w is load-bearing though — switching it to %v during a later cleanup would turn a benign race into a hard fan-out failure that retries the request for nothing.
Nit: when drift blocks, changeSummary = drift.summary overwrites the appended refresh attribution, so a drift-blocked PR loses the "why was I re-planned" line. The comment says this is deliberate and the reasoning is sound; it just means the two explanations are mutually exclusive rather than combined.
Scope, so this isn't read as broader than it is: I verified merge_gate.go by structural diff against the audited tip, plus serve.go, check_records.go, check_runs.go and clampDriftSummary directly. I did not read the 360 test lines or the metrics additions closely.
Not blocking.
Closes the merge gate guardrail. When an apply changes a target's schema, every other open PR holding stored plan check state against that (environment, database type, database) is holding a verdict computed against a schema that no longer exists. This PR drains the durable requests #868 records and re-plans those PRs — or blocks them. Builds on #868 (drive-tail recording) and #867 (storage).
The case this is acute for is a CLI-driven apply: nothing on GitHub even hints the target moved, so a sibling PR's green check simply stays green.
The drain loop. Each pass sweeps for completed applies with no request row (the outbox backstop), terminates requests wedged past the attempt cap, then claims pending requests under a rotating lease and fans out. The drive tail's notifier wakes the loop so a request drains immediately after its apply completes rather than waiting out the poll interval; the wake-up is an in-memory hint over the durable row, so losing it costs latency and nothing else. Same-target pending requests coalesce into one fan-out — a re-plan against the live schema already covers every schema change recorded before it started.
Fail closed. A failed re-plan durably flips the stored check to a blocking
schema_changed_replan_failedconclusion with a fixed message — the raw re-plan error stays in server logs and is never rendered on the PR. The request still completes: the block is a durable outcome, not something to retry.What a reviewer sees. A PR that nobody touched can acquire a new check result, so every re-planned check carries an attribution note saying which apply moved the schema under it.
Sibling PR re-planned after another PR's apply landed
ordersSibling PR where the re-plan itself failed (fail closed)
ordersThe check's own detail carries the operator instruction:
Invariants
main, sincedocs/invariants.mdpostdates this branch; itsEnforced:line names the request outbox, the drive-tail recording, and this fan-out.Observability: per-PR and per-request outcome counters, a stuck-termination counter, and triage-complete logs on every skip and failure path.
The chain: #867 (storage) → #868 (drive-tail recording) → #866 (settle re-plan processor) → #939 (request kinds + hold storage) → #940 (preflight hold fan-out) → #941 (apply-start gate) → #942 (plan-time holds). Merges bottom-up; each PR retargets to
mainas its base merges.🤖 Drafted by Armand's AI agent (Claude Fable 5)