Skip to content

Commit 3a7c64b

Browse files
committed
ci(codspeed): stop manufacturing spurious benchmark changes
Two CI causes of reports flagging benchmarks that the PR cannot affect. Serialised develop baselines. All develop pushes shared one concurrency group and never cancel, so they ran strictly one at a time. A burst of merges leaves later commits without a finished baseline, and CodSpeed then falls back to an older comparison base with its own footnote: "No successful run was found on develop (X), so Y was used instead. There might be some changes unrelated to this pull request in this report." That footnote appears on four of twelve open-PR reports, and it lines up with the benchmarks that flip between two fixed values in opposite directions on unrelated PRs. Give each develop push its own group so baselines still run on every commit, as the comment intends, without queueing behind each other. Develop runs can now overlap, which raises peak demand on a preemptible runner pool. CUDA walltime suite triggered by any workflow edit. The paths filter matched .github/workflows/**, so every dependency bump ran the walltime CUDA benchmarks. Eleven CI-only PRs reported cuda/bitpacked_u8/unpack/3bw[100M] improving 18-19% from an identical base, each carrying CodSpeed's own warning that walltime on standard hosted runners produces inconsistent data. Only this workflow defines those jobs, so match just this file. Non-pull_request events still always run the CUDA suite, so develop keeps full coverage. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ypCZPxvLuJK9FgWKF8U4P
1 parent 9d4ac37 commit 3a7c64b

1 file changed

Lines changed: 7 additions & 4 deletions

File tree

‎.github/workflows/codspeed.yml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,13 @@ name: Codspeed Benchmarking
22

33
# Concurrency control:
44
# - PRs: new commits on a feature branch will cancel in-progress (outdated) runs.
5-
# - Push to develop: runs queue sequentially, never cancelled. This allows us to have benchmarks
6-
# run on every commit for our benchmarks website.
5+
# - Push to develop: every commit gets its own group, so baseline runs never cancel and never
6+
# queue behind each other. Serialising them meant a burst of merges left later commits without
7+
# a finished baseline, so CodSpeed fell back to an older comparison base and reported changes
8+
# unrelated to the PR being tested.
79
# - `workflow_dispatch`: groups by branch and queues if run on develop.
810
concurrency:
9-
group: ${{ github.workflow }}-${{ github.ref }}
11+
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name == 'push' && github.sha || '' }}
1012
cancel-in-progress: ${{ github.ref != 'refs/heads/develop' }}
1113
on:
1214
push:
@@ -40,7 +42,8 @@ jobs:
4042
filters: |
4143
cuda:
4244
- "vortex-cuda/**"
43-
- ".github/workflows/**"
45+
# Only this workflow defines the CUDA benchmark jobs.
46+
- ".github/workflows/codspeed.yml"
4447
4548
bench-codspeed:
4649
strategy:

0 commit comments

Comments
 (0)