[garrytan]

Lane TM3 of GSTA-49. It moves every paid case still pinned to an older frontier model onto the current generation. Garry's direction, 2026-10-18: "We don't have to test the older frontier models."

Re-pinned cases

Pins to claude-opus-4-7 now follow the capture model, resolveEvalModel('capture') (Opus 5.5 since #3275). These cases test skills, not a particular model, so they track the capture model:

• skill-e2e-plan: plan-ceo-review, -selective, -expansion-energy, plan-eng-review;
• skill-e2e-plan-prosons: four cases;
• skill-e2e-design: design-consultation-core and -existing;
• skill-e2e-office-hours-phase4;
• skill-e2e-qa-bugs: qa-b6, b7, b8.

The tests that mirror these pins are updated.

Fixed per-model references keep a literal id, moved to the new generation:

What	Old	New
overlay-nudges Opus arms	claude-opus-4-7	claude-opus-5-5
overlay-nudges Sonnet arm	claude-sonnet-4-6	claude-sonnet-5-5
cookie workflow judge	claude-sonnet-4-6	claude-sonnet-5-5
codex-sol-scope case	gpt-5.6-sol	gpt-6.1-sol

The cookie judge's manual-review JSON is unchanged. It records Garry's approval of Fable refusals.

Fresh baselines. Each re-pinned case ran once on its new model as a fresh measured reference. Thresholds, assertions, trial counts and case kinds are unchanged.

Repairs from that run (GStack Minder, 2026-10-18)

• plan-review-prosons-hardstop-neg. The provider refused the capture-writing turn (stop_reason: refusal, category reasoning_extraction). The refusal was triggered by the capture instruction's "verbatim … Do NOT paraphrase. This is a format-capture test" wording. In replays of the exact request, the old wording was refused 7 times in 16 and the reworded instruction 0 times in 24. The shared capture helper now asks for "the text of the single AskUserQuestion you would have made". The plan content, assertions and trials are unchanged. All four prosons cases then passed 3/3.

• Cookie workflow judge on sonnet-5-5. The judge scored the /setup-browser-cookies workflow below threshold for real text defects:

  • the SETUP block conflicted with the browse probe;
  • "stop importing" appeared on READY;
  • the target came before any source was chosen.

  The template is restructured: probe, then decisions first, then prepare, import and report. Both hosts are regenerated. The judge passes 4/4/4 with the thresholds unchanged.

Results on the new models

18 of 20 re-pinned cases passed on the first run. The two reds were repaired as above and pass. Recorded paid spend for the lane is about $13.5.

Gate-integrity edits

Edit	Effect
Model pins moved to the current generation	Each case now measures its new model; fresh baselines as above.
Prosons capture-instruction wording	Avoids a provider refusal; same content and assertions.
/setup-browser-cookies template rewrite	Product text fix. The cookie-judge input tests' phrase and order assertions follow the new text.

Checks

• Remote free stage on the merged tree: 17,942 tests, 0 failed.
• Focused tests: 1,851 pass, covering parity-suite, the SKILL.md size guards, fence conformance, and the cookie and overlay tests.
• Typecheck is clean; typecheck:test reports no new diagnostics.

[trunk-io]

😎 This pull request was merged.

[github-actions]

E2E Evals: ✅ PASS

[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 5/5 · behavior 0/0 · judge 24/24 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 6m03s · cost $3.06 known + 5 of 29 trial(s) cost unknown

29 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $3.06 rule/judge cost | reconcile exit: 0

Profile: pr / mode pr; selected behaviors: 5, judges: 24. Deferred to scheduled/release coverage: 36 behaviors and 0 changed prompt files. Deferred checks did not run and receive no PR-pass credit.

Rule and judge shards

Shard	Automated result	Manual/unscored	Executed	Reused	Status	Cost
e2e-design/skill-e2e-design--design-review-detector-shim-dom	1/1	0	1	0	✅	$0.22
e2e-design/skill-e2e-design--design-review-plugin-handoff	1/1	0	1	0	✅	$0.15
e2e-design/skill-e2e-design--design-review-detector-shim	1/1	0	1	0	✅	$0.13
e2e-design/skill-e2e-design--plan-design-review-no-ui-scope	1/1	0	1	0	✅	$0.08
e2e/skill-e2e-plan--plan-review-report	1/1	0	1	0	✅	$1.04
llm-judge/skill-llm-eval	24/24	0	24	0	✅	$1.44

Fail-closed reconciliation

[test:paid] report: 1/1 slices, 6 planned shards, tier=gate, attempt 1
[test:paid] coverage: profile=pr mode=pr; selected E2E=5, judges=24
[test:paid] deferred: 36 broad behaviors, 0 changed prompts without quick live coverage; these are not PR passes
  slice 1  passed            336s  test/skill-llm-eval.test.ts
  slice 1  passed            245s  test/skill-e2e-plan.test.ts#plan-review-report
  slice 1  passed             56s  test/skill-e2e-design.test.ts#design-review-detector-shim-dom
  slice 1  passed             33s  test/skill-e2e-design.test.ts#design-review-plugin-handoff
  slice 1  passed             23s  test/skill-e2e-design.test.ts#design-review-detector-shim
  slice 1  passed             13s  test/skill-e2e-design.test.ts#plan-design-review-no-ui-scope
[test:paid] collector final outcomes: 29 executed, 0 reused; 29 passed, 0 failed, 0 manual accepted (unscored; no score-cache credit) (29 attempt records from 6 collectors; every record counts)
[test:paid] deselected testcases, zero credit (44):
[test:paid]     44  sibling case of a case shard (its own shard runs it)
[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 5/5 · behavior 0/0 · judge 24/24 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 6m03s · cost $3.06 known + 5 of 29 trial(s) cost unknown
[test:paid] exercised-path map (shadow, non-blocking): 3 MAP_GAP line(s) over 5 executed case(s) with recorded commands
  MAP_GAP plan-review-report bin/gstack-plan-reality: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP plan-review-report browse/dist/browse: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
  MAP_GAP plan-review-report design/dist/design: non-blocking for this PR; the next census refresh widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)
[test:paid] report: every planned shard accounted and passed

---

Sliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry.

[capy-ai]

flaky: test/gstack-doctor.test.ts (the CSO row reads the build-result record, windows-free-shard (1)), failed in 39304879908 attempt 1, passed on rerun 39304879908 attempt 2, logged for the daily flaky-fix pass