Skip to content

ci: teach CI the merge_group event and add the merge queue (ADR 0050) - #1867

Merged
jhgaylor merged 3 commits into
mainfrom
ci/merge-queue
Sep 11, 2026
Merged

ci: teach CI the merge_group event and add the merge queue (ADR 0050)#1867
jhgaylor merged 3 commits into
mainfrom
ci/merge-queue

Conversation

@jhgaylor

@jhgaylor jhgaylor commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

What

Teaches CI the merge_group event and adds a GitHub merge queue, per
ADR 0050. The ruleset is external state, so merging this does not turn the
queue on — see Activating below.

Why

Main took 775 merges in 30 days (~26/day, peak 55). already-tested skips
main's re-run only when the pushed tree equals a tree some PR run actually
tested, and over 100 recent pushes it hit 53 times. The other ~47% are the
honest report that half of what lands on main is a tree no CI run has ever
seen
.

That is not a correctness crisis — main failed 4 times in 300 push runs (1.3%).
The cost is a full re-run on main per miss, a delayed deploy, and
build-in-PR-and-retag being impossible while the merged tree isn't the tested
tree.

How

gate.py's PROBES table is the authority on what each event owes:

Event Probes Plan
pull_request changes classify the diff
merge_group changes + already-tested classify the group, and skip it when its tree is one a PR run already tested
push already-tested reuse the queue run (or a PR run) that tested this tree

A merge group publishes the tested-tree artifact and main's push finds it
through the queue's gh-readonly-queue/<base>/pr-<number>-<sha> branch name —
the only link back from a squashed commit to the run that tested it. So a
queued merge costs one full run, not two.

Sizing

One group at a time, batching up to five PRs into it. The org is on the free
plan's 20 concurrent jobs and one full CI run is 19 of them, so
speculating two groups deep cannot run two groups; it queues the second behind
the first while starving every open PR.

One rule comes off, one deliberately stays

  • Up-to-date comes off — it was an approximation of "tested against what it
    merges into", which the queue now does directly.
  • Review stays at 1. The queue answers does this tree build; review
    answers did anyone read it. On a repo this agent-heavy the second is worth
    keeping, so --merge-queue does not touch it.

The accepted cost: GitHub will not enqueue a PR whose merge requirements are
unmet, so an unreviewed PR never enters the queue — it doesn't fail, and
--auto looks like it did nothing. CLAUDE.md and the CI README both say to
check reviewDecision first when a queued PR seems stuck.

Activating (after this merges)

python3 scripts/ci/require-checks.py --merge-queue            # preview
python3 scripts/ci/require-checks.py --merge-queue --apply

--apply refuses while main's workflows lack a merge_group: trigger. That
ordering is the one failure with no visible cause: the queue waits out
check_response_timeout_minutes for a check that never starts and ejects the
PR, with nothing red anywhere. It also flips allow_auto_merge, which is
currently false — without it --auto errors and nothing can reach the queue.

Verification

  • 23 CI policy tests pass (was 18). Each new guard was confirmed to fail
    when reverted
    : removing the merge_group trigger fails the new
    trigger/PROBES parity test; dropping the queue-run lookup fails main's
    reuse test; letting a queue run prove itself from a sibling queue run fails
    its own test.
  • actionlint and shellcheck clean on all touched files.
  • okf validate decisions valid, 0 errors; index regenerated.
  • The ruleset payload was dry-run against the live ruleset.

Note for whoever merges

This touches .github/workflows/, which needs the workflow OAuth scope —
see the two-accounts note. Merge normally (--admin, since the queue is not
on yet); the activation command above is the step that switches it on.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RJBSE57BbSEDF5s5fvWEga

Main took 775 merges in 30 days and `already-tested` misses about half of
them, because the branch was one or two commits behind at merge. Those misses
are the honest report that half of what lands on main is a tree no CI run has
ever seen. A merge queue builds the tree the merge will produce and merges
only if it passes.

CI learns a third event. `gate.py`'s PROBES table is the authority: a merge
group runs both probes, classifying the group like a PR and skipping it when
its tree is one a PR run already tested. The group publishes `tested-tree` and
main's push finds it through the queue's `gh-readonly-queue/.../pr-<n>-<sha>`
branch name, so a queued merge costs one full run rather than two.

`require-checks.py --merge-queue` adds the ruleset rule. It refuses to apply
while main's workflows lack a `merge_group:` trigger, because that ordering is
the one failure with no visible cause: the queue waits for a check that never
starts and ejects the PR with nothing red to explain it. A new gate test keeps
the workflow's triggers and PROBES from drifting apart.

The queue builds one group at a time and batches up to five PRs into it: the
org is on the free plan's 20 concurrent jobs and one full CI run is 19 of
them. The review requirement drops to zero in the same flag, because a
requirement no solo-authored PR can satisfy made every merge `--admin`, and an
admin merge skips the queue.

The ruleset is external state, so this commit does not turn the queue on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RJBSE57BbSEDF5s5fvWEga
@jhgaylor jhgaylor changed the title ci: teach CI the merge_group event and add the merge queue (ADR 0049) ci: teach CI the merge_group event and add the merge queue (ADR 0050) Sep 11, 2026
@jhgaylor

Copy link
Copy Markdown
Collaborator Author

Rebased onto main and renumbered ADR 0049 → 0050: #1839 landed its own 0049 (the acp runtime runs a command you name) while this was in review, which is the ADR-number collision this repo keeps hitting.

Worth noting what happened here, since it is the PR's own argument: this branch was opened, and by the time it was pushed main had moved 11 commits and the PR was DIRTY with zero checks run. That is the failure mode the queue removes — not a broken build, just a PR that silently cannot be tested until someone notices and rebases it.

`--merge-queue` no longer drops `required_approving_review_count` to zero.
The two gates answer different questions: the queue answers whether a tree
builds, review answers whether anyone read it, and on a repository where most
PRs are written by agents the second is the one worth keeping. Enabling a
machine that checks the first is not a reason to stop asking the second.

The cost is accepted rather than engineered around. GitHub will not enqueue a
PR whose merge requirements are unmet, so an unreviewed PR does not fail — it
never enters the queue, and `--auto` looks like it did nothing. CLAUDE.md and
the CI README both say to check `reviewDecision` first when a queued PR seems
stuck.

ADR 0050 records this as reconsidered: dropping review to zero moves from the
decision into the rejected alternatives, with why.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RJBSE57BbSEDF5s5fvWEga
@jhgaylor

Copy link
Copy Markdown
Collaborator Author

Reversed the review decision. --merge-queue no longer drops required_approving_review_count to 0 — it stays at 1, and the flag does not touch it.

The reasoning in ADR 0050 now says why: the queue answers does this tree build, review answers did anyone read it, and on a repo where most PRs are written by agents the second is the one worth keeping. Dropping review to zero moved from the decision into the rejected alternatives.

The cost is accepted rather than engineered around: GitHub won't enqueue a PR whose merge requirements are unmet, so an unreviewed PR never enters the queue — it doesn't fail, --auto just appears to do nothing. Both CLAUDE.md and the CI README now say to check reviewDecision first when a queued PR looks stuck.

23 policy tests still green; the new one asserts the queue flag leaves the pull_request rule byte-identical.

The index row carries each ADR's frontmatter description, so editing one
without rerunning scripts/decisions-index.sh leaves the two disagreeing and
the decisions workflow red. `okf validate` alone does not catch it; the index
drift check does, which is why this went green locally and failed in CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RJBSE57BbSEDF5s5fvWEga
@jhgaylor
jhgaylor merged commit ea71d0e into main Sep 11, 2026
24 checks passed
@jhgaylor
jhgaylor deleted the ci/merge-queue branch September 11, 2026 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant