Skip to content

v1.91.119.0 test(seed): refresh the free-suite duration seed (#3224) - #3238

Merged
capy-ai[bot] merged 4 commits into
mainfrom
capy/seed-refresh-2026-10-17
Oct 17, 2026
Merged

capy-ai[bot] merged 4 commits into
mainfrom
capy/seed-refresh-2026-10-17

Conversation

@garrytan

Copy link
Copy Markdown
Owner

In plain words

Since the 2026-10-15 recording (#3212), 54 new free test files have landed. CI therefore guesses their run time as the 99th-percentile duration, and the weekly health check fails CEO-10 again (limit 5). This PR re-records every free file's duration on a CI-mirrored Ubicloud VM, so the shard packer has real numbers for all of them. Only data files change.

What changed

  • scripts/free-test-durations.json: recorded with UBI_OWNER=gsta33 bun run test:ubicloud --record-durations (standard-16) on origin/main cadbd0af (v1.91.116.0). The recorder exited 0 after 316 s, and no file failed or timed out. Durations went from 1219 to 1273: 54 added, none removed. The serial sum went from 3202 s to 3515 s.
  • scripts/free-test-seed-allowlist.json: removes test/setup-codex-scope.test.ts, which now records 58.9 s, under the 60 s ceiling. The shrink-only ratchet requires dropping entries that no longer exceed it. The two other files over 60 s from v1.91.105.0 fix(tests): the free suite stops needing its retry pass; test:health tells the truth (#3154) #3212 remain allowlisted with their split todos: setup-install-registry at 86.5 s and host-renders-prune at 63.5 s.

free-tests.yml is untouched; #3236 owns its curation. A standing recorder is planned as a separate lane (W4).

Validation

  • unseededCheck (the CEO-10 check in scripts/test-health-report.ts) on this tree: pass 0 unseeded file(s) (limit 5). On main it reports fail with 54 unseeded files.
  • GSTACK_FREE_SEED_BASE=$(git merge-base HEAD origin/main) bun test test/free-seed-ratchet.test.ts test/test-health-report.test.ts: 26 pass, 0 fail.
  • The diff is data only: no code, tests or prompts change, so no paid eval selection is expected.

Behavior changes

None. The shard packer reads the new durations.

Refs #3224. Expect CEO-10 to go red again as new test files accumulate until the standing recorder lands.

Open in Capy

…rding (#3224)

54 free test files had landed since the 2026-10-15 recording, so CEO-10
failed again (limit 5). Recorded with `bun run test:ubicloud --record-durations`
(standard-16, recorder exit 0, no failed files) on origin/main cadbd0a:
1219 -> 1273 durations, none removed. test/setup-codex-scope.test.ts is back
under the 60 s ceiling (58.9 s), so its allowlist entry is removed as the
shrink-only ratchet requires.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

@github-actions

Copy link
Copy Markdown
Contributor

E2E Evals: ✅ PASS

[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 0/0 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 0m00s · cost $0.00

0 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $0 rule/judge cost | reconcile exit: 0

Profile: pr / mode pr; selected behaviors: 0, judges: 0. Deferred to scheduled/release coverage: 0 behaviors and 0 changed prompt files. Deferred checks did not run and receive no PR-pass credit.

Rule and judge shards
Shard Automated result Manual/unscored Executed Reused Status Cost
Fail-closed reconciliation
[test:paid] report: 1/1 slices, 0 planned shards, tier=gate, attempt 1
[test:paid] coverage: profile=pr mode=pr; selected E2E=0, judges=0
[test:paid] deferred: 0 broad behaviors, 0 changed prompts without quick live coverage; these are not PR passes
[test:paid] collector final outcomes: 0 executed, 0 reused; 0 passed, 0 failed, 0 manual accepted (unscored; no score-cache credit) (0 attempt records from 0 collectors; every record counts)
[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 0/0 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 0m00s · cost $0.00
[test:paid] report: every planned shard accounted and passed

Sliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry.

garrytan and others added 3 commits October 17, 2026 04:56
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@github-actions github-actions Bot changed the title test(seed): refresh the free-suite duration seed (#3224) v1.91.119.0 test(seed): refresh the free-suite duration seed (#3224) Oct 17, 2026
@capy-ai
capy-ai Bot merged commit f47321e into main Oct 17, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant