CI cost survey: every PR lane measured, and the queue is a runner-label problem

[Astro-Han]

Following #4461, I measured every lane that runs on a pull request, not just test. This is a survey with numbers attached, not a proposal — each item below states what it would cost us in coverage, and whether it buys queue time, runner slots, or minutes inside one slot. Those are three different budgets and only one of them turned out to be the binding constraint.

Headline: the binding constraint is not how much work we run. It is which runner label we ask for.

Method

• gh api repos/apache/maka/actions/... over every run in a complete ~19h snapshot (1000 runs, 1756 jobs, 2026-08-31T19:37Z → 2026-09-01T14:32Z), plus per-workflow histories reaching back to 2026-08-13.
• Queue time is job.started_at - job.created_at, restricted to jobs that concluded success/failure (cancelled jobs report unusable timestamps).
• Path-filter and planner hit rates are replays of planTests() and of each workflow's paths: globs over the last 300 --first-parent commits.
• Nothing was dispatched. No repo-wide build or test suite was run locally.

Where a claim is inferred from reading code rather than measured, it says so.

---

The slot budget, measured

19h snapshot, summing completed_at - started_at across all 1756 jobs:

runner class	slot-minutes	share	median queue (UTC 07–14)
ubuntu-latest	3306	45%	15.8–33.4 min
windows-*	2895	40%	0.1–0.3 min
ubuntu-24.04* (pinned)	796	11%	0.3 min
macos-*	331	5%	0.2 min

40% of everything we spend is on Windows runners that we never wait for. Cutting Windows work saves the ASF bill; it does not save a single minute of anyone's wall clock.

---

1. ubuntu-latest is the entire queue problem. Nothing else queues. — queue time

Median queue by UTC hour, first-layer jobs only (no needs:), same repo, same snapshot:

UTC hour	ubuntu-latest	ubuntu-24.04*	windows-*	macos-*
00–06	0.0–0.1	0.0–0.1	0.0–0.1	0.1
07	4.3	0.4	0.1	0.1
08	18.6	0.8	0.2	0.6
09	19.7	0.3	0.1	0.1
10	8.3	0.1	0.1	0.2
11	13.1	0.7	0.6	0.6
12	22.0	1.5	0.1	0.8
13	22.6	0.2	0.1	0.2
14	18.3	0.2	0.1	0.2

Controls, in order of strength:

• Same run, same instant, three OSes. gitoxide-helper-admission fans out to ubuntu-latest / macos-latest / windows-latest from one run, so all three jobs share a created_at. Median queue over 8 runs: 15.0 / 0.1 / 0.1 min. In 4 of 8 runs ubuntu-latest was more than 5 minutes slower than both siblings. macos-latest and windows-latest are -latest aliases too, so this is not about the alias.
• Paired ±15-minute windows, cross-workflow. 426 pairings of an ubuntu-latest first-layer job against ubuntu-24.04* first-layer jobs created within 15 minutes: median 11.4 vs 0.2 min; ubuntu-latest slower by >1 min in 285/426, faster in 22.
• Single-job control, congested window only. dependency-audit is one ubuntu-latest job with no needs:. UTC 07–14, run.created_at → job.started_at: 15.8 min. cli-package-validation's first-layer Build direct-peer addon on pinned labels, same window: 0.3 min.
• Not a concurrency pending artifact. Median run.run_started_at - run.created_at is 0.0 for every workflow measured, cli-package-validation included. The clock we are reading is runner allocation, not queueing behind our own concurrency groups.
• Not a self-hosted split. runner_group_name is GitHub Actions for both classes.

What this means. Every controllable confounder is excluded, and the residual explanation that fits is that ASF-wide demand saturates the shared ubuntu-latest pool during European/American working hours while pinned images sit idle. The observed 18-queued-1-running snapshot and the 37-minute wait on run 33511334930 both land inside 07–14 UTC.

Not verified: that changing the label actually lands us in the idle pool. That is a causal claim and the data above is correlational, however tightly controlled. Cheapest way to settle it: pin one lane — dependency-audit, one job, p99 0.9 min, 22% of PRs — from ubuntu-latest to ubuntu-24.04 and read its queue distribution for a week. Zero coverage risk, one line.

Gives up: automatic OS image upgrades. A pinned label has to be bumped by hand when GitHub moves latest, and going stale is a real (if slow) maintenance cost. Worth noting cli-package-validation already pins every one of its runners, so the repo has both conventions today.

Budget: queue time only. It removes zero work and zero slots. If it holds, it is worth more than every other item on this list combined — CI is one job now, so it is ~15 min of wall clock per PR push during working hours.

---

2. native/, patches/, .asf.yaml, deny.toml fall through the planner into --full — slots + minutes

full fires on 57/300 commits (19.0%). Attribution:

commits	trigger
22	unknownCode fallback — no FULL_SUITE_FILES entry involved
16	package.json
14	.github/workflows/ci.yml
11	package-lock.json
11	scripts/ci-test-plan.mjs
1	scripts/run-workspace-tests-parallel.mjs

(rows overlap; a commit can hit several of the named files)

The single largest source of full-suite runs is not package-lock.json. It is the classifier loop in planTests() reaching code = true; unknownCode = true and recursing with forceFull: true. Broken out by bucket:

commits	paths that reach the fallback
14	native/** — runtime-host-peer/src/engine.rs (12), bindings.rs (11), gitoxide-helper/**
4	patches/**
3	.asf.yaml
1	deny.toml

7.3% of commits run the entire suite because the planner has no opinion about them. patches/ is the sharpest case: isCliPackagePath() and isAsfSourcePath() both handle it explicitly, but the main loop never gets there, so the --full return short-circuits both. .asf.yaml is a branch-protection config file with no code reachability at all. And native/** already has three dedicated lanes (runtime-host-peer-admission, gitoxide-helper-admission, the Rust build inside cli-package-validation) — a Rust change currently pays for those and for appIcons, asfSource, storybook, e2e, and all 10 workspaces.

Gives up: for whatever gets classified, the blanket guarantee that an unrecognized path is maximally validated. That fail-safe should stay for genuinely unknown paths — the fix is to classify these four, not to weaken the fallback.

Flagging this as the one item that is very small and very valuable. Someone should decide whether it warrants its own PR. Related to #4473 (same file, same authority), so it may belong there.

---

3. cli-package-validation has a concurrency group but no cancel-in-progress — slots

concurrency:
  group: cli-package-validation-${{ github.workflow }}-${{ github.ref }}

Measured over the 19h window: it holds more runner slots than any other workflow — 736 jobs against CI's 523 — from only 60 PR runs (it is second by slot-minutes: 1672 to CI's 3126, because its jobs are short and numerous), because it fans out to 13.6 jobs per run (4-way peer-native matrix → build → 4-way smoke matrix → state-root-qualification → eval).

• 14 of 60 runs (23%) were still executing when a newer push landed on the same branch.
• 275 of 1672 slot-minutes (16%) were spent on work that was already obsolete.
• Only 5 runs were cancelled, versus 76 for CI, which does set cancel-in-progress.

Without cancel-in-progress, a superseded run is not cancelled; it runs to completion while the newer one queues behind it in the same group.

Gives up: nothing on pull requests. workflow_call from the release path and workflow_dispatch land on different refs and would need to keep the current behaviour, exactly the way windows-sandbox-w0.yml already scopes its group (${{ github.head_ref || github.run_id }}).

Budget: pure slots (mostly pinned-Linux/Windows/macOS, so no queue relief). ~275 slot-minutes per 19h at current PR volume.

---

4. gitoxide-helper-admission is triggered by a manifest that cannot affect it — slots

Path-filter replay over 300 commits: 16 hits, and the top driver is packages/runtime/package.json at 12 of 16 (75%).

Inspecting what actually changes in workspace manifests over those 300 commits: "version" (41 occurrences), "test:dist" (36), "test" (35), "pretest" (27), "clean" (27), "typecheck" (18), "build" (16). Version bumps and npm script edits. Each one currently spends three runners (ubuntu-latest + macos-latest + windows-latest, ~1.5 min each) building and testing a Rust helper whose inputs did not move — and the ubuntu-latest one of those three is the job that produced the 15.0-minute median queue in §1.

The same shape, less severe, in runtime-host-owner-platform.yml: 29 hits, of which package.json drives 12 and package-lock.json 7 — 19 of 29 from root manifests rather than from the owner-platform sources the lane exists to check.

Gives up: coverage of the case where a dependency range in packages/runtime/package.json changes what the helper links against. Narrowing to packages/runtime/package.json and an actual dependency-section diff, or dropping the manifest and keeping native/gitoxide-helper/** + Cargo.lock, would need someone who knows whether that link is real. Unverified either way — I did not trace the build.

---

5. Release Windows check: the download failures were already fixed in #4461, and my sample was skewed — corrected

This item was wrong when first filed. Keeping it visible rather than deleting it.

The lane is real and expensive: 20.5 min median, p95 25.3, 1711 slot-minutes / 19h, 129 PR runs (36% of PR branches, 4.2 runs per branch). But the diagnosis was not.

What I originally reported: 14 of 20 sampled failures were Download and verify the pinned Windows upgrade baseline, and ci.yml retries its equivalent download while this lane does not.

Both halves are wrong:

• It is already fixed. ci: allocate a runner only when its inputs changed #4461 added a Restore the pinned Windows upgrade baseline cache step keyed on hashFiles('scripts/windows-upgrade-baseline.json'), immediately before the download. Its comment records the same observation from a wider window than mine: "14 of 36 across 300 runs, always on branches that could not have caused it."
• My sample was skewed. All 60 sampled runs came from the 19h snapshot, i.e. 2026-09-01, which was the day the baseline fetch broke at scale. Failures by day, pull_request only:

day	failures	dominant failing step (sampled)
08-24	5
08-25	9
08-26	12	Package the Windows installer and ZIP (9/12)
08-27	3
08-28	7
08-29	9	Package the Windows installer and ZIP (7/9)
08-30	4
08-31	10
09-01	36	Download and verify the pinned Windows upgrade baseline (11/12)

Across nine days the failures are mixed, and outside 09-01 the dominant step is a different one entirely.

What survives: Package the Windows installer and ZIP is the other repeat offender — 9 failures on 08-26 and 7 on 08-29, on a 20-minute lane. I could not determine the cause; job logs from those days have aged out of retention. Unverified, flagged for whoever can catch it live. If it is an electron-builder artifact download, it has the same shape as the baseline fetch and the same fix applies.

Same lane as #4470.

6. The CLI tarball is packed and smoke-tested twice on the same commit — slots + minutes

• ci.yml runs 8 steps behind cli_package == 'true', ending in release:cli:pack (45s) + release:cli:smoke (81s). Planner hit rate: 70.0% of commits.
• cli-package-validation.yml builds the addon on 4 platforms, packs an immutable tarball, and runs the same scripts/smoke-release-cli-package.mjs across 4 runners × 2 Node versions. Path-filter hit rate: 21.3%.

Replayed together over 300 commits: cli-package-validation fires on 64, and all 64 also run CI's cli_package steps. Zero the other way. Its trigger set is a strict subset. On those 21.3% of commits we pack and install-smoke a CLI tarball on Linux x64 twice.

Gives up: if CI's steps were gated on "and cli-package-validation is not also running", the required test check would stop being the CLI packaging gate on 21.3% of commits, and cli-package-validation is not in .asf.yaml's required_status_checks.contexts. That trade is bad as stated — the merge gate must not get weaker. Any fix here has to move the contexts, not just the steps, which is a bigger conversation than a CI-cost issue should start.

Recording it as measured duplication, not as a recommendation.

6b. Two of the three State Root transitions cannot change on a pull request

cli-package-validation's state-root-qualification job (155 slot-minutes / 19h, runs unconditionally on all 64 triggers) qualifies three transitions. Its own comment says it:

Two of these transitions are between tarballs that were published and frozen, so nothing in a pull request can change their outcome except the qualifier itself

Measured: the qualifier and its fixture (scripts/qualify-released-cli-state-root.mjs, released-cli-state-root-fixture.mjs, and the test) change on 3 of 300 commits (1.0%). So ~⅔ of that job's work has a PR-controllable input on 1% of commits and runs on 21.3%.

ci.yml independently qualifies durable state against the published baseline (33s) on state_root_compat, which fires on 28.7% of commits — overlapping cli-package-validation on 14.3%.

Gives up: the published→published transitions would stop being re-proven on every unrelated CLI change. They would still run on the 1.0% that can move them, on main, and on the release path. The risk is drift in something outside the repo (a republished tarball) going unnoticed longer — the nightly lane is the natural place to keep catching that.

---

7. The planner's high-hit surfaces are high for a structural reason

Replay over 300 first-parent commits (reproduces the numbers in #4461's discussion):

surface	hit rate
code	94.7%
releaseContract	71.7%
cliPackage	70.0%
runtimeSandbox	69.0%
runtimeHost	62.7%
e2e	57.7%
storybook	52.0%
astryxSurface	48.0%
stateRootCompat	28.7%
appIcons	21.3%
asfSource	19.7%
full	19.0%
storageStress	5.7%

These three are not independently wide — they are one predicate wearing three hats:

• isCliPackagePath() matches any path under 7 of the 10 workspaces (cli, core, eval, mcp, runtime, runtime-host, storage). It is effectively "did you touch the backend", which is why it reads 70%.
• releaseContract = cliPackage || isReleaseContractPath(...) is a strict superset by construction: 71.7% vs 70.0% means the release-contract file list adds 1.7 points over "backend changed".
• runtimeSandbox = workspaces.includes('packages/cli'), and packages/cli is the top of the reverse-dependency closure, so it is reachable from core, storage, mcp, runtime, runtime-host, and eval. 69% is the closure's shape, not a predicate that can be tightened in place.

No recommendation attached. Narrowing any of these means deciding which backend workspaces genuinely cannot affect CLI packaging, and I have no evidence for that — it is a design question, not a measurement one. Recording the structure so nobody spends time "tuning" a predicate whose width comes from somewhere else.

---

8. The dependency closure sends 36% of commits to all 10 workspaces — minutes

Workspace-count histogram over 300 commits (avg 5.36 of 10 selected):

workspaces selected	commits
0	32
1	55
2	23
3	2
4	41
5	2
6	15
7	21
10	109

109/300 (36.3%) select everything. Only 57 of those are full. The other 52 (17.3%) get there purely through reverseDependencyClosure, and every one of them touches packages/core:

packages/core   → all 10          packages/storage → 7
packages/runtime → 6              packages/runtime-host → 4
packages/mcp    → 3               packages/ui → 2

57 commits touch packages/core at all; the 52 non-full ones select 10.00 workspaces on average — no exceptions. Top files: package.json (11), events.ts (10), session.ts (6), foreign-session.ts (5), agent-run.ts (5).

The closure is correct — core really is imported everywhere. The question is whether it needs to be manifest-level. A change to packages/core/src/model-catalog.ts pulls packages/computer-use into the run because computer-use depends on @maka/core, not because it imports that module.

Gives up: a module-level import graph would be a real authority to build and keep honest, and it can be wrong in a way a manifest graph cannot (dynamic imports, re-exports, type-only edges). This is the most expensive item on the list to implement and the one I am least confident about. Same family as #4478, which does exactly this reasoning for e2e specs — if that lands and the mechanism generalizes, this is the follow-on.

---

9. dependency-audit runs on PRs that change no dependency — slots

paths: includes apps/*/package.json and packages/*/package.json. Replay: 50/300 commits hit, and 37 of those come from workspace manifests versus 23 from root package.json/package-lock.json. Per §4, workspace manifest diffs are dominated by "version" and npm scripts edits.

The job is genuinely cheap — 0.7 min median, p99 0.9 min — but it is ubuntu-latest, so during 07–14 UTC it occupies a congested-pool slot for a measured median of 15.8 minutes of queue to do 42 seconds of work. It also runs nightly at 03:17 UTC, in the idle window, where it has been green 24/25.

Gives up: the PR-time signal that a dependency change introduces a disallowed license or an unshipped transitive. Given the nightly lane, the loss is detection latency (hours), not detection. Alternatively keep the trigger and just pin the label per §1, which costs nothing at all.

---

10. timeout-minutes: 45 is uniform and nothing comes close to it — slots, preventive

Every job in every workflow uses 45. Measured durations:

workflow	p50	p95	p99	max	headroom
Dependency audit	0.7	0.8	0.9	0.9	50×
Runtime Host owner platforms	1.1	1.7	1.9	1.9	24×
Runtime Host peer admission	1.5	2.2	2.2	2.2	20×
Gitoxide helper admission	1.5	2.7	3.0	3.0	15×
Windows recovery	2.8	3.7	4.4	4.5	10×
CLI package validation	1.7	6.1	7.6	11.5	4×
Release Windows check	20.5	25.3	29.9	29.9	1.5×
CI	0.3	25.5	26.7	30.9	1.5×

CI and Release Windows check are sized correctly. A hung dependency-audit holds a congested-pool slot for 45 minutes instead of 3.

I observed no hung job in this window, so this is preventive, and it is last on the list for that reason. It also has a real downside: a too-tight timeout turns a slow runner into a red required check.

---

Places I looked and found nothing to cut

• windows-recovery — 57% of PR branches, which looked high, but 170 runs cost only 452 slot-minutes (2.8 min median) on the uncongested Windows pool, and its seven steps (NTFS alternate data streams, root-initialization replacement race, Local IPC trust boundary, SQLite crash recovery, Runtime continuation recovery, owner-death recovery) are things no Linux lane can prove. 176 green / 6 red. Cheap, in the right pool, and the only evidence for what it checks. Leave it.
• concurrency groups — correct everywhere except §3. CI, release-windows-check, windows-recovery, runtime-host-owner-platform, runtime-host-peer-admission, gitoxide-helper-admission, dependency-audit all set cancel-in-progress, and windows-sandbox-w0 correctly gives scheduled and manual runs their own group so they cannot discard each other.
• Cron schedules — six of them, all offset off the hour, all in the 03:00–07:30 or 18:00 UTC idle window, none overlapping. windows-baseline 22/22 green, windows-sandbox-w0 12/12, dependency-audit 24/25. Green cron is not automatically waste, and these are cheap and out of the way. No change.
• windows-baseline on PRs — it did run on 675 pull requests between 2026-08-16 and 2026-08-20, and was removed on 08-20. Already fixed; noting it so nobody re-discovers it in the run history.
• take.yml — 578 runs on issue_comment, but it uses ubuntu-slim, medians 3s queue / 5s duration, and 50 of 53 sampled jobs skip immediately. Negligible, and it is already the only lane using the cheap label.
• Node-side build caching — confirming what was measured before ci: allocate a runner only when its inputs changed #4461: npm ci 17s, Build 19s, Typecheck 9s on a 1528s job. 2.9%. Nothing there.
• CI matrix dimensions — CI is one job with no matrix after ci: allocate a runner only when its inputs changed #4461. cli-package-validation's two 4-way matrices map to four genuinely different build targets and four different install environments; no dimension is redundant.

Measurement caveats

• The CI job-level data is pre-ci: allocate a runner only when its inputs changed #4461. ci: allocate a runner only when its inputs changed #4461 merged at 2026-09-01T14:08:38Z; the snapshot ends at 14:32Z. Almost all 523 CI jobs in it are the old plan → heavy → test shape, which queued three times per PR (measured example, run 33490755623: 81.5 min wall clock, 55.1 min of it queue, plan running 22s and test running 4s). That is exactly what ci: allocate a runner only when its inputs changed #4461 fixed, and the numbers in §1 are the reason the fix mattered more than the job count suggested. Step timings in this issue come from run 33511334930, which is the post-ci: allocate a runner only when its inputs changed #4461 single-job shape: 37 min queue, 25.5 min run.
• Two hit-rate methodologies, and they disagree for one lane. Squash-replay over merged commits gives windows-recovery 21.0%; the live snapshot gives 57% of PR branches. Squash-replay measures the final diff of merged work; the snapshot measures currently-open branches, where a long-lived PR that once touched a wide path keeps re-triggering that lane on every subsequent push. Both are in this issue and each item says which one it uses. cli-package-validation agrees across both (21.3% vs 21%), which is a decent check that the replay itself is sound.
• The 19h snapshot is one weekday. The hourly table is consistent across it, but a single day cannot rule out that 2026-09-01 was unusually busy ASF-wide.

Cross-references

• Select Desktop e2e specs by what a change can reach, instead of all 103 #4478 (Desktop e2e spec selection) — §8 is the same reasoning applied to workspace selection. If Select Desktop e2e specs by what a change can reach, instead of all 103 #4478's mechanism generalizes, §8 is its follow-on, not a separate effort.
• Derive the planner's test-file inputs from the gate scripts that run them #4473 (planner derives test inputs from gate scripts) — §2 is the same file and the same authority. The four unclassified path buckets may belong in that PR.
• Cache the Rust builds in the packaged Windows lane #4470 (Kache for the packaged Windows lane) — §5 is the same lane. The baseline-fetch half of it turned out to be already fixed in ci: allocate a runner only when its inputs changed #4461; what remains there is the unexplained Package the Windows installer and ZIP failures.
• Strengthen the CI contracts added in #4461, and delete the authority one of them guards #4475 — untouched here by design.

Nothing in this issue would remove test from .asf.yaml's required_status_checks.contexts, rename it, or put it behind a paths: filter.

[Astro-Han]

take

[Astro-Han]

Acted on four of these. Two PRs, plus a correction and one new finding.

#4482 — three lanes selecting on inputs that cannot change their verdict (§2, §3, §4)

	measured effect
§2 planner classifies native/, patches/, .asf.yaml, deny.toml	full 19.0% → 11.7%, appIcons 21.3% → 14.0%, asfSource 19.7% → 13.7%, stateRootCompat 28.7% → 21.7%, mean workspaces 5.36 → 4.86 of 10, commits selecting all ten 109 → 87
§3 cancel-in-progress on cli-package-validation	recovers the 275 of 1672 slot-minutes spent on superseded runs
§4 drop packages/runtime/package.json from gitoxide-helper-admission	lane selects 4 of 300 commits instead of 16

On §4 the filter turned out to be worse than "too wide" — it is stale. That entry has been there since #3561, where it stood for a source-import dependency, and packages/runtime now contains no Gitoxide reference of any kind. Every source the lane actually reads was already named directly.

#4483 — pin one lane's runner label to settle §1's causal question

dependency-audit from ubuntu-latest to ubuntu-24.04: one job, p99 42 seconds, no OS-version surface, nothing lost if the answer is no. gitoxide-helper-admission deliberately keeps its ubuntu-latest matrix leg, so a same-instant three-OS control keeps running for the whole observation period. The PR says how to read the result after a week and what to do in either direction.

§5 was wrong and has been corrected in place above. The baseline-download retry I recommended was already added in #4461, in the same commit this issue was written against, and my failure sample was drawn entirely from 2026-09-01 — the one day that fetch broke at scale. Across nine days the failures are mixed, and outside that day the dominant failing step is Package the Windows installer and ZIP (9 failures on 08-26, 7 on 08-29), cause unknown because those job logs have aged out of retention.

New, and worth a maintainer's judgement rather than a patch: test is the only entry in .asf.yaml's required_status_checks.contexts, so gitoxide-helper-admission, cli-package-validation, release-windows-check, windows-recovery and runtime-host-peer-admission all run on pull requests but none of them can block a merge. That is the existing design and it is defensible — a required context must report on every PR, which a path-filtered lane cannot do. But it changes how §2 reads: narrowing the planner shifts Rust coverage onto lanes that were already non-blocking, which does not make the merge gate weaker than it is today, but does make that reliance load-bearing where the full-suite fallback used to mask it. Flagged in #4482 rather than decided there.

Deliberately not touched:

• §6 (CLI packaging duplication). Removing CI's copy needs cli-package-validation to become a required context first, and a path-filtered lane cannot be one. That is repository governance, not a CI-cost change, and getting it wrong freezes every merge.
• §8 (dependency closure). Select Desktop e2e specs by what a change can reach, instead of all 103 #4478 is building exactly this reasoning for e2e specs. Doing it for workspace selection now would duplicate that mechanism and collide with it.
• §7, §9, §10 stand as recorded; §7 in particular is structure rather than a tunable, and is there to stop someone spending a day tuning a predicate whose width comes from the closure.