Skip to content

Leave the monthly CHS current run to the results repository - #481

Merged
clarkbw merged 4 commits into
mainfrom
chs-current-results-repository
Oct 24, 2026
Merged

clarkbw merged 4 commits into
mainfrom
chs-current-results-repository

Conversation

@clarkbw

@clarkbw clarkbw commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

The benchmarks workflow no longer runs the full CHS current benchmark on main. openwatersio/chs-current-validation now does that every month, and published its first release, fit-2026-10-22-8f85912-r1, today. Branch pushes keep the quick check. The docs and benchmark comments now describe the results repository in the present tense.

Part of openwatersio/slackwater-ios#752 (Rollout step 1: chs-current-fit out of the main-branch and scheduled runs). It is the last task of the results-repository plan, openwatersio/slackwater-ios#813.

Why

This workflow's monthly run scored only the newest grid point and kept its rows as an artifact for 90 days. It was a stopgap until the results repository existed. That repository now:

  • scores every grid point since the newest one it committed;
  • keeps the rows in its history;
  • pools a year per gate;
  • publishes each gate's tier and bounds.

A second full run here would only duplicate those IWLS requests.

What changes

  • .github/workflows/benchmarks.yml: the CHS current step runs only when FAST is set, which means branch refs. It keeps NO_FLOORS. The schedule and workflow_dispatch triggers stay for the other benchmarks, and so does the upload step, which carries their outputs.
  • docs/validation/chs-currents.md, README.md, CONTRIBUTING.md: the sentences about this workflow's monthly run and its 90-day artifact are gone, and every sentence about the results repository is in the present tense, linking its releases. CONTRIBUTING says to run the benchmark locally to score every gate and held-out week, and that the results repository scores the recipe stamps its targets.json names. The README labels its October 1 figures as this repository's scorer version 1 run.
  • In the cross-shard check's description (chs-currents.md, chs-current-fit.ts), "window" is now "recipe window". The results repository's check leaves out the 365-day window, which is reported only.
  • Comments in benchmarks/chs-current-{grid,score,fit}.ts say the results repository pools, checks and finds, where they used to say it will.

Verification

  • npm run lint, npm run build and git diff --check pass.
  • The benchmarks vitest project passes: 91 tests in 3 files.
  • benchmarks.yml parses as YAML.
  • Every present-tense claim about the results repository was checked against its code, line by line:
    • its monthly cron, and that it runs on main only;
    • the grid points and windows it scores;
    • the 13 pooled grid points;
    • the cross-shard check over recipe windows;
    • the release assets and tag.
  • Nothing ran the workflow change in Actions. Codex checked the condition's cases against FAST's definition.

Review

During implementation the change had its own review against its brief. That review checked each claim about the results repository against that repository's code, and found one overstatement, "several" grid points a month, fixed in 80a1eff. One review-branch round followed.

Round 1: Codex, Claude Opus and Claude Sonnet. Copilot was skipped because its CLI isn't installed. Codex read the results repository and verified the new claims against it, and found no issues. Opus and Sonnet couldn't read outside this worktree, so they couldn't verify those claims. Opus and Sonnet found only low issues, so no second round ran.

Fixed in cf67900:

  • The README's October 1 figures, which now follow the sentence about the results repository's releases, read as a release result (Opus). They are labelled as this repository's scorer version 1 run.
  • CONTRIBUTING didn't say where the full run happens now that main skips it (Opus, Sonnet). It says to run the benchmark locally, and that the results repository scores the stamps its targets.json names.
  • "Every run in the benchmarks workflow sets NO_FLOORS" (Sonnet). It now names the quick run, the only one the workflow makes.

Not fixed:

  • workflow_dispatch on main can no longer produce a full CHS current run (Opus). This is the point of the change, as the plan specifies. A full run is npm run chs-current-fit locally.
  • A recipe change that reverses an axis at a gate outside the quick check's three shows up only in the results repository, after merge (Sonnet). There it is recorded as that gate's reversed_axis disqualifier in the release, not as a failed run. That is the design of the rows-only mode (Add a rows-only mode to the CHS current benchmark for the results repository #471).

cf67900 had no model review after it. Lint and git diff --check ran on it.

🤖 Generated with Claude Code

clarkbw and others added 4 commits October 24, 2026 12:06
openwatersio/chs-current-validation now runs the full CHS current fit benchmark every month and publishes each gate's tier and bounds in its releases, so this repository's benchmarks workflow no longer runs it on main. Branch pushes still run the quick check as a regression check, with NO_FLOORS and the reversed-axis check unchanged. The validation record, README and CONTRIBUTING described the results repository as upcoming and this workflow as keeping the rows for 90 days. They now describe what the results repository does and link its releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
openwatersio/chs-current-validation is live, so the benchmark's code comments that said it "will pool" rows, "will check" across shards or "will find" a gate's worst slacks now say what it does. The comment about its cross-shard check says "recipe window", because that check leaves out the report-only 365-day window. Comments only, no code change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The grid steps 35, 35, 35 and 7 days, so a calendar month holds at most two grid points; "several" overstated it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main no longer runs the full benchmark, so CONTRIBUTING says to run it locally and that the results repository scores only the stamps its targets.json names. The README's October 1 figures are labelled as this repository's scorer version 1 run, so they don't read as a results-repository release, and the NO_FLOORS note now names the quick run, the only one the workflow makes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@clarkbw
clarkbw merged commit 05864b4 into main Oct 24, 2026
22 checks passed
@clarkbw
clarkbw deleted the chs-current-results-repository branch October 24, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant