ci: name the Ubuntu image instead of asking for the queue - #4483
Conversation
08efd0f to
bdd9010
Compare
Astro-Han
changed the title
|
This PR's own checks turned into the controlled experiment the description said the run history could not provide. Same push, same Its first push (15:11:42Z), when only
#4482's push at 15:10:33Z, eight jobs created in the same second:
That is the same-instant After the full pin (15:26:15Z): CI / test on And the direct comparison: #4482's CI / test, created 15:21:39Z on This does not explain the mechanism. The likeliest reading is still that ASF-wide demand saturates whatever capacity is behind the |
M4n5ter
left a comment
M4n5ter
left a comment
There was a problem hiding this comment.
English
The runner pin itself looks sound: all nine substitutions preserve the jobs, matrices, permissions, release authority, and required context, and the live ubuntu-24.04 audit/helper jobs have succeeded. I found one Important issue in the new drift guard, so I am commenting rather than approving.
The policy does not cover the runner declaration forms it claims to govern. The matcher at ci-workflow-policy.test.mjs:391-405 recognizes only a single-token runs-on:, a runner: field, or any standalone - value line. The repository already has a different valid form in runtime-host-owner-platform.yml:
matrix:
os: [windows-latest, macos-latest]
runs-on: ${{ matrix.os }}Adding ubuntu-latest to that inline matrix passes the new test while allocating the forbidden runner. - os: ubuntu-latest, an inline runs-on: [self-hosted, ubuntu-latest], and workflows using the supported .yaml suffix also evade it. Conversely, an unrelated scalar list item - ubuntu-latest is rejected even when it cannot feed runs-on. This makes the test both incomplete and a false authority.
The smallest reliable contract here is probably the simpler one: after stripping comments, reject the literal ubuntu-latest anywhere in every .yml/.yaml workflow. There is no legitimate live workflow use for that literal under this policy, and this removes the brittle declaration-shape regex. If exact runner-only scoping is required, then derive jobs.*.runs-on and every referenced matrix value, with fixtures for inline lists, block lists, and include objects.
Validation: 108 planner/workflow/Windows-harness tests pass; git diff --check passes. A synthetic replay confirmed that the current matcher rejects the existing block-list shape but accepts both inline-matrix and include.os forms.
中文
Runner pin 本身是正确的：9 处替换都没有改变 job、matrix、权限、release authority 或 required context，实际运行的 ubuntu-24.04 audit/helper jobs 也已成功。但新增 drift guard 有 1 个 Important 问题，因此这里发普通 review，而不是 approve。
该 policy 没有覆盖它声称要约束的 runner declaration 形态。 ci-workflow-policy.test.mjs:391-405 的 matcher 只识别单 token 的 runs-on:、runner: 字段，或任意独立的 - value 行。但仓库已经在 runtime-host-owner-platform.yml 使用另一种合法形式：
matrix:
os: [windows-latest, macos-latest]
runs-on: ${{ matrix.os }}把 ubuntu-latest 加进这个 inline matrix 后，新测试仍会通过，但真实 job 会使用被禁止的 runner。- os: ubuntu-latest、inline runs-on: [self-hosted, ubuntu-latest]，以及合法的 .yaml workflow 后缀也会漏掉。反过来，任意无关的 - ubuntu-latest 标量列表，即使不可能进入 runs-on，也会被误判。因此这条测试既不完整，又会形成虚假的 authority。
这里最小且可靠的契约可能反而更简单：去掉注释后，在所有 .yml/.yaml workflow 的有效内容中直接禁止字面量 ubuntu-latest。在这项 policy 下，该字面量不存在合法的 workflow 用途，也就可以删除脆弱的 declaration-shape regex。如果确实要求只约束 runner，则需要从 jobs.*.runs-on 推导所有被引用的 matrix value，并覆盖 inline list、block list 与 include object 的 fixture。
验证：108 个 planner/workflow/Windows-harness 测试全部通过，git diff --check 通过；定向合成验证确认当前 matcher 能抓 block-list，却会放过 inline matrix 与 include.os。
Asking for `ubuntu-latest` and asking for `ubuntu-24.04` gets the same machine. Both labels reported `Image: ubuntu-24.04`, `Version: 20260823.283.1`, provisioner `20260819.586`, runner `2.336.0` in their `Set up job` logs on the same day. So this changes which queue a job waits in, and nothing else. The queues are not the same. Across 1183 runner-backed jobs in eleven ASF repositories, `ubuntu-latest` has a median wait of 0.05 minutes and a p90 of 19.07; `ubuntu-24.04` has a median of 0.03 and a p90 of 0.53, while carrying 3.6x the jobs (455 against 126). The alias is not slower on average — it is unpredictable, and a required context is paid at the tail. `apache/flink` sees a p90 of 105.8 minutes on the alias. The sharpest control comes from this repository. One push to #4482 created nine first-layer jobs within the same second, none of them declaring `needs`. The three on `ubuntu-latest` waited 7.5, 7.68 and 7.7 minutes. The six on `ubuntu-24.04`, `ubuntu-24.04-arm`, `windows-2025`, `macos-15`, `windows-latest` and `macos-latest` waited between 3 and 24 seconds. It is not the `-latest` alias as such, since two of the fast six are aliases. It is not our own `concurrency` groups, since every workflow's median `run_started_at - created_at` is 0.0. It is not a self-hosted split, since `runner_group_name` reads `GitHub Actions` on all nine. `ci.yml` also has a reason of its own. Its bubblewrap step disables `apparmor_restrict_unprivileged_userns` specifically because Ubuntu 24.04 gates user namespaces that way. The required context already assumes this image; the alias only left that assumption free to drift without a commit. `windows-latest` and `macos-latest` stay as they are. Across 84 same-instant groups containing both `windows-latest` and `windows-2025`, the median paired difference in wait is 0.00 minutes, so pinning them would buy nothing measurable. Nine substitutions across eight workflows; thirteen jobs in the release and packaging lanes were already pinned. Two costs, both accepted deliberately. The automatic image upgrade becomes a manual commit — `ci-workflow-policy.test.mjs` holds the rule and says how to take an exemption. And `ubuntu-latest` carries 79.1% of this repository's Ubuntu job-minutes (3141 of 3970), so this raises our demand on the pinned label 6.2x, onto a pool whose wait was measured while it carried one sixth of that. The pool absorbs 3.6x more jobs than the alias today at a p90 of 0.53 minutes, and the worst pinned figure anywhere in the sample is `apache/iceberg` at 0.43 median against 18.78 on its own alias. If the p90 on `ubuntu-24.04` passes 2 minutes after this lands, that is the signal to revisit, with both sides finally measurable. Refs #4480
bdd9010 to
568f211
Compare
me2seeks
left a comment
me2seeks
left a comment
There was a problem hiding this comment.
Re-reviewed exact head 568f211. The runner substitutions preserve the jobs, matrices, permissions, and required test context. The drift guard now deliberately rejects ubuntu-latest anywhere in every .yml/.yaml workflow after comment-only lines are removed, so inline matrices, include objects, and runner-label sequences cannot bypass it.
Validation: node --test --test-concurrency=1 scripts/ci-test-plan.test.mjs scripts/ci-workflow-policy.test.mjs scripts/verify-windows-harness.test.mjs passed 108/108 locally. act parsed the CI job and its pull_request/push/workflow_dispatch triggers; full container execution is unavailable here because Nerdctl does not expose the Docker API required by act.
me2seeks
left a comment
me2seeks
left a comment
There was a problem hiding this comment.
Re-reviewed exact head 568f211. The runner substitutions preserve the jobs, matrices, permissions, and required test context. The drift guard now deliberately rejects ubuntu-latest anywhere in every .yml/.yaml workflow after comment-only lines are removed, so inline matrices, include objects, and runner-label sequences cannot bypass it.
Validation: node --test --test-concurrency=1 scripts/ci-test-plan.test.mjs scripts/ci-workflow-policy.test.mjs scripts/verify-windows-harness.test.mjs passed 108/108 locally. act parsed the CI job and its pull_request/push/workflow_dispatch triggers; full container execution is unavailable here because Nerdctl does not expose the Docker API required by act.
#4460 added this lane and #4483 banned `ubuntu-latest`, 35 minutes apart. Both were green when they ran; main is red now that they sit together, which blocks every merge because `test` is the only required context. One-line fix, and exactly the drift the rule exists to catch: `ubuntu-latest` is what every tutorial writes, so it comes back one new workflow at a time. The rule caught it on the first one. Refs #4480
3 participants
Nine substitutions across eight workflows move
ubuntu-latesttoubuntu-24.04. Thirteen jobs in the release and packaging lanes were already pinned; these were the ones still on the alias.The two labels are the same machine
Read out of
Set up joblogs on the same day, one job that asked for each label:Image:Version:ubuntu-latestubuntu-24.0420260823.283.120260819.5862.336.0ubuntu-24.04ubuntu-24.0420260823.283.120260819.5862.336.0Same image, same build. This changes which queue a job waits in, and nothing else.
The queues are not the same
Restricted to 07:00–15:00 UTC, when the contention happens, across 22 ASF repositories (
runner_group_name == "GitHub Actions", jobs that actually acquired a runner):ubuntu-latestubuntu-24.04The gain is in the median, not the tail. Thirty-two times better in the middle; 4.7x at p90; and at the extreme the two are indistinguishable — the pinned pool has been seen at 36 minutes. This buys "almost always seconds instead of minutes", not "never waits again". Over the full day and all labels, 46% of
ubuntu-latestjobs wait more than a minute against 36% ofubuntu-24.04ones, which is the same story: the alias is worse everywhere, and it is dramatically worse in the body of the distribution rather than at the edge.apache/flinkreaches a p90 of 105.8 minutes on the alias. Withinapache/iceberg, the same-repository comparison is 0.43 median onubuntu-24.04(n=58) against 18.78 onubuntu-latest(n=3, small).This repository's own figures, before the pin:
ubuntu-latest6.62 median / 11.12 p90 (n=11);ubuntu-24.040.29 / 1.23 (n=8). Small samples, same direction.Note the volume:
ubuntu-24.04carries 1053 jobs in this sample against the alias's 2034. It is a heavily used pool that happens to be faster, not an empty one that looks fast for lack of users.The sharpest control is one push in this repository
One push to #4482 created nine first-layer jobs within the same second, none of them declaring
needs:(verified against the YAML — the dependent jobs in that run appear correctly six minutes later):ubuntu-latestubuntu-latestubuntu-latestwindows-latestmacos-latestubuntu-24.04-armmacos-15windows-2025ubuntu-24.04Same push, same
created_atto the second, label the only variable.Confounders ruled out. Not the
-latestalias as such — two of the six fast jobs are aliases. Not our ownconcurrencygroups — medianrun_started_at - created_atis 0.0 across 1036 runs. Not a self-hosted split —runner_group_namereadsGitHub Actionson all nine. Not burst rank or repo load —spearman(queue, rank within same-instant group)is 0.23, and jobs that were the only job at their instant wait the same as those in bursts (17.68 vs 17.47 min median, n=47/96).ci.ymlhas a reason of its ownIts bubblewrap step disables
apparmor_restrict_unprivileged_usernsspecifically because Ubuntu 24.04 gates user namespaces that way — the comment atci.yml:158says so. The required context already assumes this image. The alias only left that assumption free to drift on GitHub's schedule, with no commit to review. Pinning is a net reduction in risk for the one check that blocks every merge.What this does not settle, stated plainly
apache/kafkahas a 0.30 min median onubuntu-latestin the same hoursapache/sparksits at 21–37. Something is allocated per-repository that the API does not expose. "The alias pool is saturated org-wide" cannot be the whole story.ubuntu-latest(median 7.2 min waited before cancellation). They are excluded from every median above, which biases the alias figures low. The effect is larger than measured, not smaller.created_at → first job startedis 23.0 min, not 33.4; and the gitoxide three-OS median of 15.0/0.1/0.1 was one 8-run slice, not the 46-run median of 1.58.Scope
ci.ymltest— the required contextdependency-audit.ymlauditgitoxide-helper-admission.ymlubuntu-latestmatrix legruntime-host-peer-admission.ymlqualitypr-effort-label.ymllabelissue-pr-lifecycle.ymllifecyclerelease.ymlrelease-identityandpublishcopilot-auto-review.ymlrequest-review(currently disabled; pinned so re-enabling does not reintroduce it)That leg in
gitoxide-helper-admission.ymlwas kept on the alias in #4480 as a live control. Removing it is deliberate: the measurement is complete and reproducible from history and from the ten other repositories above, while keeping one lane on the alias would cost every Gitoxide pull request its tail wait indefinitely.windows-latestandmacos-lateststay as they are, for two different reasons.Windows has no queue to escape and no faster label to escape to.
windows-latestwaits 0.05 min median across 98 jobs and has never been seen above 1.20;windows-2025is slower at 0.10 median (n=26), and same-instant pairs differ by +0.02 min. In this repository the two sit at 0.12 and 0.14. Pinning would be a real image change bought with nothing.macOS does queue, just rarely: 3 of 86 jobs waited over a minute, the worst being 9.73 (apache/pulsar) and 5.98 (apache/lucene). That is a fifteenth of
ubuntu-latest's rate of 46%. It stays on the alias because no named macOS label is measurably better and one is clearly worse —macos-14exceeds a minute on 35% of its 37 jobs, andmacos-15has only 8 observations. Unlike Ubuntu, where both labels resolve to one image,macos-latestis macOS 26 andmacos-15is macOS 15: pinning here changes the operating system to chase a tail that the pinned labels do not demonstrably avoid.The rule
ci-workflow-policy.test.mjsnow rejects the literalubuntu-latestanywhere in any.ymlor.yamlworkflow, after stripping comments.The first version of this test matched runner declaration shapes. That was both incomplete and a false authority:
runs-onreaches a runner through matrix values, inline sequences andincludeobjects, and the repository already usesos: [windows-latest, macos-latest]inruntime-host-owner-platform.yml. Adding the alias there would have passed. Verified: the current rule is red for each ofos: [.., ubuntu-latest],- os: ubuntu-latest,runs-on: [self-hosted, ubuntu-latest]and a trailing-spaceruns-on: ubuntu-latest, all four of which the shape matcher let through — and green as committed across all 21 workflows.Under this rule the literal has no legitimate use anywhere, which makes its mere presence the honest contract. There is no exemption syntax: taking one back means editing this test, which is a commit someone reviews. No lane needs one today.
ubuntu-latestis what every tutorial writes, so without a rule this drifts back one new workflow at a time — which is how these nine got here.What is given up
The automatic image upgrade. When GitHub moves
latest, these labels move by hand. Dependabot does not updateruns-on:, so this obligation currently has no owner beyond the test comment. Filed as a known gap rather than solved here.Concentration risk.
ubuntu-latestcarries 79.1% of this repository's Ubuntu job-minutes (3141 of 3970 in a 19h window). This raises our demand on the pinned label 6.2x, onto a pool whose wait was measured while it carried one sixth of that load. The evidence that it absorbs the shift is indirect: across the ASF it already carries more jobs than the alias, and during the contended hours it holds a 0.37 min median against the alias's 12.00 while both reach the high thirties at the extreme — the pinned pool is better in the body of the distribution, not immune.Rollback criterion: if this repository's median wait on
ubuntu-24.04during 07:00–15:00 UTC passes 2 minutes after this lands, revisit. The median is the criterion because the median is what this buys — both pools already reach the high thirties at the extreme, so a tail figure would trip immediately and mean nothing. With both sides finally measurable, a split becomes an arithmetic question rather than a guess.Refs #4480