[garrytan]

In plain words

The free test suite has a safety net: when a test file fails, it reruns that file once and, if it passes, calls the run green and writes the file into a "flake ledger". Last week 6 of 46 main runs needed that net. Five test files caused it. Three of them failed for one reason: since git 2.29, every git commit quietly starts a background cleanup job that creates a lock file inside .git and deletes it a moment later. A test that counts, watches or deletes .git right after a commit sometimes catches that file mid-life. Those tests now switch the cleanup job off for their throwaway repos, and the test runner switches it off for every test as a backstop. The other two were a test that counted to 300 instead of waiting for a signal, and a test that called the live GitHub API. With those fixed, the next flake in the ledger is a real signal.

The weekly health report (#3154) also gets truthful: the duration seed is refreshed so new test files stop packing at the 99th-percentile guess, the Windows timing line stops averaging the wrong runs, and the "how to fix" line stops pointing at a CI recording that does not exist. gstack-timeline-read now refuses filters it cannot apply.

CEO-10 will go red again until a standing seed recorder lands (main adds about 15 free test files a day against a limit of 5). The value of this refresh is packing accuracy for the 164 files that were packed at p99; the recorder is a P1 follow-up below.

Per-flake evidence

Reproducer: the repo's CI-mirrored diagnostic, flaky retry off, isolated HOME/state/ledger per rerun. Counts are free-measurement.json (completed / passed / capHit). Before = 47d639a (origin/main, unmoved), after = 81f7a5d (all repairs, before the stamp and seed commits).

Shard file lists come from each ledger run's free-plan artifact:

gh run download <run> -R garrytan/gstack -n free-plan -D /tmp/free-plan
files=$(jq -r --argjson s <shard> '.shards[] | select(.shard == $s) | .files | join(",")' /tmp/free-plan/free-plan.json)
bun run test:ubicloud --diagnostic --files "$files" --reruns 30 --wall-cap 3600

Reproducer	Before	After
Ubicloud, the five ledger files together (--files <5 files> --reruns 200 / 600)	200 / 197 / false (1a x3)	600 / 600 / false
Local runner env under git-commit load (bun run scripts/ship-measure.ts free --backend local --files test/gstack-docs-candidate.test.ts,test/qa-functional-observer.test.ts,test/plan-design-sdk-fixture.test.ts --reruns 200, 4 parallel, 4 git commit loops)	200 / 37 / false (1b x159, 1e x42)	200 / 200 / false
Ubicloud ledger shard 16 (1b), 30 reruns	30 / 30 / false	30 / 30 / false
Ubicloud ledger shard 6 (1e), 30 reruns	30 / 30 / false	30 / 30 / false
Ubicloud ledger shard 9 (1d), 30 reruns	not run (deterministic repro below)	30 / 30 / false
Ubicloud ledger shard 3 (1a), 30 reruns	30 / 0 / false: 1a x1, pair-agent-tunnel-eval x30	30 / 0 / false: 1a x0, pair-agent-tunnel-eval x30
Ubicloud ledger shard 4 (1c), 30 reruns	30 / 0 / false: 1c x0, path-validation x30, handoff x12, diagram-render-drift x1	30 / 0 / false: 1c x0, path-validation x30, handoff x10, diagram-render-drift x3

Acceptance (n >= 3/p clean): 1a p = 3/200, n >= 200, observed 600 clean. 1b p = 159/200, n >= 4, observed 200 clean. 1e p = 42/200, n >= 15, observed 200 clean plus 30 shard reruns.

Shards 3 and 4 cannot report passed = n in diagnostic mode, before or after. ship-measure free gives each rerun a private HOME under /tmp and sets GSTACK_STATE_ROOT. browse/test/path-validation.test.ts builds its "outside" directory under os.userInfo().homedir, which Bun reads from HOME, so it lands under /tmp (allowed) and the fixture refuses. browse/test/pair-agent-tunnel-eval.test.ts expects its denial log under its fake HOME, but the daemon follows the inherited state root. Both pass in the normal suite and are filed P2. handoff and diagram-render-drift fail intermittently at 16-way rerun parallelism in shard 4 before and after; neither is in any main ledger, and neither is investigated here.

1a browse/test/terminal-agent-publication-lock.test.ts (ledger 38153371796/3, 38161906980/3)

Diagnosis: two fixed-count polls (300 x 10 ms for a readiness file, then 300 x 10 ms of /proc for state Z), plus a partial-write race. The Ubicloud failures show the race directly: ENOENT: open '/proc/0/stat', i.e. an empty readiness file read as pid 0. Repair: the Python parent waits with os.waitid(P_PID, WEXITED|WNOWAIT), which leaves the child in state Z, then prints its PID. The test reads one newline-framed line under the test's own unchanged 10 s timeout and makes one /proc read that must show Z. Early exit, a malformed line or a timeout reports the parent's exit status and stderr tail (each checked by hand with a planted hang, early exit and 0 PID).

1b test/gstack-docs-candidate.test.ts (ledger 37742176952/16)

Diagnosis: replicating repo() + snapshot 150 times showed 4 listing differences, each one -objects/maintenance.lock, and 0/150 with the keys. Repair: repo() sets maintenance.auto=false and gc.auto=0. The count assertion is unchanged; its message now names the changed entries.

1c test/qa-functional-observer.test.ts (ledger 37800857251/4): root cause shown in a harness, not reproduced in the file

Diagnosis: fixture + commit + observeQAWrites 200 times: 0 idle. Under 4 git-commit load loops, 6/200 setups threw ENOENT ... lstat '<fixture>/.git/objects/maintenance.lock'; with the keys in the inherited env, 0/400. The test file itself never failed (0/200 local loaded, 0/230 on Ubicloud). Repair: the runner backstop only. createQAFunctionalFixture and the observer select 22 paid cases, so they are not edited (priced follow-up below). Item 2 (observer tolerance) was not needed.

1d test/gstack-next-version.test.ts (ledger 37804770972/9)

Diagnosis, deterministic: with a delayed fake gh, 8 PRs x 0.5 s took 4108 ms (serial; the plan measured 4160 ms), and 27 PRs x 1.2 s took 32637 ms, past the test's 30 s spawn timeout. Empty stdout, then JSON Parse error: Unexpected EOF. Repair: the smoke test keeps the real repo and CLI and puts a named fake gh first on PATH. The fake logs every argv and exits 2 with FAKE_GH_UNHANDLED on anything unexpected. The test checks the exit code with stderr, the exact call set, no marker, no glab call, host: github, offline: false, no fallback, no warnings, the canned claim, and 1.6.3.0 + a claim on 1.6.4.0 giving 1.6.5.0. The CLI defect is filed P1, not fixed.

1e test/plan-design-sdk-fixture.test.ts (ledger 37807608021/6)

Diagnosis: a diagnostic copy whose rmSync wrapper logs throws and post-return survivors, under commit load: 18/150 file runs failed. In every failure Bun's rmSync(dir, { recursive: true, force: true }) returned without throwing and left the scratch repo behind (.git/objects emptied, refs and config gone, plan.md left). The suite's catch {} had nothing to swallow. With the keys in the fixture's replacement git env (which never sees the runner's), 0/100. Repair: the fixture env carries the two keys. The collector assertion is unchanged; its message names surviving directories and removal outcomes.

Shared mechanism, deterministic

GIT_TRACE=1 git commit on git 2.55.0 prints 3 git maintenance run --auto --quiet --detach lines with defaults and 0 in the runner child env (test/test-free-shards-git-maintenance.test.ts pins this, with the default env as control).

Intentional pin changes

• test/test-health-report.test.ts free-wall value: windows-free-tests 0.0 min (1 runs) becomes windows-free-tests (successful PR runs) 0.0 min (1 runs) (item 6).
• browse/test/terminal-agent-publication-lock.test.ts: the readiness-file existsSync check and both polling loops are replaced by the framed PID line (a positive integer) and one /proc read. expect(state).toBe('Z') and every lock-recovery assertion are unchanged.
• No other assertion changed. Three failure messages became self-explaining (docs-candidate count, plan-design collector, next-version exit code).

Behavior changes

• gstack-timeline-read --since / --branch with no value, an empty value or a following --flag exit 2 before any read: --since must be a relative time like "7 days ago" (got no value); accepted: "<N> minute|hour|day|week|month[s] ago", "<N>h|d|w", or an ISO date YYYY-MM-DD or YYYY-MM-DDTHH:MM:SSZ, --branch must be a branch name (got flag '--limit'), (got an empty value).
• An unparseable --since (yesterday, 2 weeks, 7days, 2026-13-01, anything with extra words) exits 2 with (got '<value>') instead of returning unfiltered history. Newly accepted: 7d / 24h / 2w and ISO dates (UTC). The relative form stays case-insensitive. Parsing happens once in bash (no date -d, no bash-4 features); the Bun regex is gone.
• Unknown arguments, previously ignored, exit 2 with the usage line. --help / -h print it and exit 0.
• test:health free-wall: windows-free-tests (successful PR runs) <p50> (<n> runs) or no successful PR runs in window, from pull_request runs only (dispatch duration recordings excluded). The free-tests half is byte-identical.
• test:health CEO-10 fix text names UBICLOUD_API_KEY, says --record-durations copies the seed back, tells contributors without the key to leave it to a maintainer, and drops "or a CI recording" (also from docs/TESTING_INTERNALS.md).
• The free runner's shard children get maintenance.auto=false and gc.auto=0 appended to GIT_CONFIG_*. A malformed inherited GIT_CONFIG_COUNT is passed through so git still rejects it.

Duration seed (#3154)

Recorded with UBI_OWNER=fw2build bun run test:ubicloud --record-durations (standard-16) on this branch, which is origin/main 47d639a (unmoved at every fetch, last checked before the final run) plus this wave. The recorder exited 0 after 292 s and wrote 1219 durations: 1055 before, 164 new, none removed; serial sum 2439 s to 3202 s. bun run test:health --enforce on the recorded tree: [PASS] Free test files missing from the duration seed (CEO-10): 0 unseeded file(s) (limit 5). (--enforce still exits 1 on W7a, the flaky-pass rate, because its 7-day window holds two pre-fix runs of the 1a file.)

Two earlier recordings did not count and are not used: the first failed on an unrelated fixed 50 ms sleep in test/plan-seed-submission.test.ts (filed P2, not edited), the second failed free-seed-ratchet because five files were over 60 s. No file was excluded.

free-seed-ratchet with GSTACK_FREE_SEED_BASE=$(git merge-base HEAD origin/main): 10 pass, 0 fail. Three files are over the 60 s ceiling and are allowlisted as annotated exceptions with a split todo: test/setup-install-registry.test.ts 82 s (58 s on 2026-10-04), test/host-renders-prune.test.ts 64 s (unseeded before), test/setup-codex-scope.test.ts 60.4 s (48 s). No stale entries were found.

Outliers against the old seed (at least 10 s on either side and a ratio of 1.5x or more, or 0.67x or less): 9.

File	Old (s)	New (s)	Ratio
test/timeline.test.ts	1.8	11.9	6.56x
test/context-identity.test.ts	6.8	25.6	3.74x
test/gstack-skill-start.test.ts	6.6	21.7	3.26x
test/brain-sync.test.ts	26.5	57.2	2.16x
test/evidence.test.ts	7.5	13.6	1.81x
test/review-start-evidence.test.ts	14.2	22.7	1.60x
test/hook-scripts.test.ts	16.6	26.5	1.60x
test/setup-claude-code-migration.test.ts	10.5	16.5	1.58x
browse/test/navigation-guard.test.ts	14.6	6.6	0.45x

The largest new files: test/host-renders-prune.test.ts 63.5s, test/model-role-invocation.test.ts 34.7s, test/cso-file-ceiling.test.ts 31.6s, test/host-renders.test.ts 23.9s, test/gstack-doctor.test.ts 23.3s, test/setup-runtime-root-guard.test.ts 16.6s, test/redact-prepush-emails.test.ts 15.5s, test/codex-probe-cache.test.ts 12.5s. These are recordings of one VM run, so single-file ratios carry run-to-run noise; none is over the ceiling except the three allowlisted files.

Paid selection

bun run eval:select --base origin/main on the final tree:

Profile: pr (pr)
Changed files: 20
Behavioral probes: none
Quality judges: none
Deferred to broad coverage: aside-browse-basic, aside-browse-flow, carve-section-loading-browse, ship-measure-seeded-flake, timeline-event-flow

The full touchfile profile (--profile full) lists, all deferred and none run on this PR: aside-browse-basic, aside-browse-flow, carve-section-loading-browse (from browse/test/terminal-agent-publication-lock.test.ts), ship-measure-seeded-flake (from scripts/test-free-shards.ts), timeline-event-flow (from bin/gstack-timeline-read) and ship-skipped-queued-finding (from the package.json version stamp). LLM judges: 0 of 24. Nothing paid was run.

Follow-ups (TODOS.md, appended section; owner Garry)

• P1 gstack-next-version serial claimed-slot fetch (4160 ms for 8 PRs at 0.5 s; async spawn with bounded overlap; its own priced PR, selects 108 paid cases and 24 judges). Revisit 2026-10-29. The plan's GitHub issue for this repro: filed as TODOS P1; no issue opened in this wave.
• P1 standing Linux seed recorder (scheduled recording or the free-plan job reading the latest recording once v1.91.107.0 fix: the GBRA-40 field list (one Bun for install and check, rec gate approval, pregate never silent) #3190 lands); agree the design with the owner of free-tests.yml before 2026-10-29.
• P1 PR full-fallback 45.9% / $73.52 recorded spend per merged PR / push-to-verdict p50 11.4 min: touchfile mappings for bin/, lib/, hosts/, make-pdf/. Candidate headline for the next wave. Revisit 2026-10-29.
• P1 flaky-pass target checks on 2026-10-22 and 2026-10-29 (under 5% of main runs).
• P2 createQAFunctionalFixture sets maintenance off itself (22 paid cases, priced).
• P2 diagnostic-mode HOME/state-root artifacts in path-validation and pair-agent-tunnel-eval.
• P2 test/plan-seed-submission.test.ts fixed 50 ms sleep (failed once during the first seed recording).
• P2 split the three files the new seed puts over 60 s (allowlisted with that todo); brain-sync and setup-codex-scope-namespaces (57 s each) are next.
• The audit follow-up is recorded in docs/test-audit-2026-10.md ("Post-merge follow-up (2026-10-15)"); the census reliability item stays due 2026-10-26 with Weekly periodic evals: red lane needs triage #3153/v1.91.108.0 fix(evals): census repair — the late-input and simplification fixtures stop inviting their own failures (#3153) #3200.

Coordinator note: TODOS.md is also edited by GSTA-32's wave (it drops two lines in its Oct 22 close follow-up). GSTA-32 (jam_01M503N530E8HD5SHJ9A8FQQ24) agreed to treat this PR's appended section and one-bullet deletion as append-only, like CHANGELOG.md; whichever merges second resolves the merge.

After merge: dispatch test-health.yml on main; it should close #3154 itself (not comment). It will reopen once CEO-10 exceeds 5 unseeded files again, until the recorder lands.

Validation

Revision: final head after the 1.91.105.0 restamp; tests and pregate ran on 4fd931eae (tree 05797cd06334), which differs only in the stamp files, base 47d639a (origin/main, unmoved at last fetch).

• bun run test once on the frozen tree, on Ubicloud (bun run test:ubicloud, standard-16, VM ubirun-fw2build-1792109069-e2855ad7, destroyed): 17/17 shards pass; 1219 files, 28228 tests, 28137 pass, 91 skip, 0 fail. The flaky retry pass did not fire (no ledger written).
• bun run typecheck: 0 errors. bun run typecheck:test: 865 known diagnostics, none new.
• GSTACK_FREE_SEED_BASE=$(git merge-base HEAD origin/main) bun test test/spawnsync-timeout-tripwire.test.ts test/module-size-ratchet.test.ts test/free-seed-ratchet.test.ts: 19 pass, 0 fail.
• Focused tests for every touched code file (13 files): 385 pass, 0 fail.
• bun run test:health --enforce: CEO-10 PASS (0 unseeded); exits 1 on W7a only (the 1a file's two pre-fix flaky runs in the 7-day window).
• Pregate: PREGATE: incomplete tree=05797cd06334 head=4fd931eaedc4 checks=6 (16-vCPU Ubicloud VM, --allow-repo-commands; run before the restamp, which changed only VERSION, package.json, the agents digest and the CHANGELOG header). Accepted exception: the only lane failure, test/diagram-render-drift.test.ts "deep: fresh build reproduces committed dist", reproduces identically on base 47d639a, where a Bun 1.4.2 rebuild of lib/diagram-render moves the committed sha256 ec759ca4 → 87f4f236; the test rebuilds dist in place, so the tree changed mid-check (incomplete). This PR does not touch lib/diagram-render. Secrets: 5 MEDIUM non-credential warnings (pii.ip_public on the version string and a duration line, pii.phone.e164 on run ids/durations), no credentials. Lanes otherwise: 696 files, 17416 tests, 16991 pass, 424 skip. Required CI is the gate; the macOS and Windows remote lanes clear there.
• bun run eval:select (pr profile): no behavioral probes, no quality judges; full profile 6 E2E (all deferred, listed above) and 0 of 24 judges. Nothing paid was run.

[trunk-io]

😎 This pull request was merged.

[github-actions]

E2E Evals: ✅ PASS

[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 0/0 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 0m00s · cost $0.00

0 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $0 rule/judge cost | reconcile exit: 0

Profile: pr / mode pr; selected behaviors: 0, judges: 0. Deferred to scheduled/release coverage: 5 behaviors and 0 changed prompt files. Deferred checks did not run and receive no PR-pass credit.

Rule and judge shards

Shard	Automated result	Manual/unscored	Executed	Reused	Status	Cost

Fail-closed reconciliation

[test:paid] report: 1/1 slices, 0 planned shards, tier=gate, attempt 1
[test:paid] coverage: profile=pr mode=pr; selected E2E=0, judges=0
[test:paid] deferred: 5 broad behaviors, 0 changed prompts without quick live coverage; these are not PR passes
[test:paid] collector final outcomes: 0 executed, 0 reused; 0 passed, 0 failed, 0 manual accepted (unscored; no score-cache credit) (0 attempt records from 0 collectors; every record counts)
[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 0/0 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 0m00s · cost $0.00
[test:paid] report: every planned shard accounted and passed

---

Sliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry.