Skip to content

v1.91.167.0 fix(evals): plan-design focus and tpa pitch detectors accept the rewordings Opus 5.5 uses (GSTA-57) - #3320

Merged
capy-ai[bot] merged 4 commits into
mainfrom
flaky/2026-10-20-plan-design-tpa
Oct 20, 2026
Merged

capy-ai[bot] merged 4 commits into
mainfrom
flaky/2026-10-20-plan-design-tpa

Conversation

@garrytan

Copy link
Copy Markdown
Owner

Daily flaky-fix pass (GSTA-57), group 1 plus one group 5 detector.

plan-design-with-ui-scope went red three times on the flaky ledger with ceiling_reached; no acknowledged Design focus and subsequent review question (runs 39315041557, 39344306426, 39363705692, attempt 1 each, slice 4). In all three the agent asked Step 0D and answered an issue-labelled finding. Two of them already pass on main's T1b detector. Run 39363705692, the one after T1b, worded the 0A rating "I rated this plan 4/10 for design completeness" and the option "All 7 areas (recommended)", and the detector only accepted "on design completeness" and "dimensions|passes". The repair defines the input class instead of adding another spelling:

  • The rating is any sentence that scores design completeness out of 10. That covers the template's own 0A example, "This plan is a 3/10 on design completeness", which the old regex rejected.
  • The all-seven option is any option label that leads with "All 7" or "All seven" followed by a word.
  • The focus-title, option-count and option-order requirements are unchanged.

tpa-absent-darwin (run 39315041557, slice 14): "If you want to use Aside: download it (macOS 15+) at aside.com, open it, sign in, and re-run" is the contract pitch with a pronoun. The canonical branch now takes Aside|it before the macOS 15+ floor and keeps its clause-start, negation and quoted-mention guards. Short forms without the floor still count only at a sentence start.

Evidence

  • Free replay of every captured slice-4 observation (3 failed and 4 passed attempts, including 39355689639) through the repaired predicates: 7/7 pass. Before the repair, 39363705692 a1 fails.
  • New fixture design-ui-focus-capture-39363705692.json (public native calls only), with a positive replay and a no-focus negative. New negative controls: a score in a different sentence from "design completeness", a score out of 7, a non-design completeness score, all-7 without a focus title, a non-leading, issue-labelled or bare "All 7" option. All of them still fail.
  • TPA: the 39315041557 transcript is now a pitch capture and counts exactly once. New negatives: negated, line-negated, reported and quoted "download it (macOS 15+)" still don't count. All earlier captures and controls are unchanged.
  • test/plan-design-with-ui-fixture.test.ts 28/28, test/third-party-actions.test.ts 58/58, free-fixtures and touchfiles tests, typecheck, and the typecheck:test ratchet (none new) all pass. Both new regressions fail against main's detectors.
  • Focused paid runs, Opus 5.5: plan-design-with-ui-scope 5/5 (129, 106, 147, 153 and 127 s); tpa-absent-darwin 5/5 (15–19 s, $0.41 total).

Gate integrity: test-only detector edits. No threshold, trial count, case kind, quarantine or budget changed.

Open in Capy

…rdings Opus 5.5 uses (GSTA-57)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

…-design-tpa

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

# Conflicts:
#	CHANGELOG.md
@github-actions github-actions Bot changed the title fix(evals): plan-design focus and tpa pitch detectors accept the rewordings Opus 5.5 uses (GSTA-57) v1.91.167.0 fix(evals): plan-design focus and tpa pitch detectors accept the rewordings Opus 5.5 uses (GSTA-57) Oct 20, 2026
garrytan and others added 2 commits October 20, 2026 17:52
…-design-tpa

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

# Conflicts:
#	CHANGELOG.md
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@garrytan
garrytan force-pushed the flaky/2026-10-20-plan-design-tpa branch from 66f7da3 to a3c2773 Compare October 20, 2026 17:53
@capy-ai
capy-ai Bot merged commit 6a1b3be into main Oct 20, 2026
62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant