You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 635535d
Browse filesBrowse the repository at this point in the historyBrowse files
v1.91.150.0 test(evals): re-pin older paid models to the current generation; prosons and cookie-judge repairs (GSTA-49 TM3) (#3293)
* feat(evals): Opus 5.5 is the paid gate's model under test; judge stays Fable (GSTA-49 TM1)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: re-baseline old-model paid pins onto the current generation (TM3)
Repair the prosons capture prompt that tripped the reasoning_extraction
classifier, and restructure /setup-browser-cookies so the Aside probe,
decisions and target order are unambiguous.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: CHANGELOG entry for the TM3 re-baseline under [Unreleased]
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* chore: stamp v1.91.150.0
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Copy file name to clipboardExpand all lines: BROWSER.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -584,7 +584,7 @@ SITE=app.example.com
584
584
DOMAIN=example.com
585
585
$B goto "https://$SITE"
586
586
$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera
587
-
$B reload #confirm the intended account
587
+
$B reload #view the page; only --verify-auth proves sign-in
588
588
```
589
589
590
590
Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display).
Copy file name to clipboardExpand all lines: CHANGELOG.md
+13Lines changed: 13 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,18 @@
1
1
# Changelog
2
2
3
+
## [1.91.150.0] - 2026-10-19
4
+
5
+
**The paid evals that still pinned older models now run on the current generation, and two cases that went red on the move are repaired at their cause.** Garry's direction (2026-10-18) is to stop testing older frontier models, so these files are re-baselined as a fresh measured reference rather than held to a threshold change.
6
+
7
+
### Changed
8
+
- **Old-model pins:** the `claude-opus-4-7` pins in `skill-e2e-plan`, `skill-e2e-plan-prosons`, `skill-e2e-design`, `skill-e2e-office-hours-phase4` and `skill-e2e-qa-bugs` now follow `resolveEvalModel('capture')` (Opus 5.5). The overlay benchmark fixtures pin `claude-opus-5-5` and `claude-sonnet-5-5`, the cookie workflow judge pins `claude-sonnet-5-5`, and `codex-sol-scope-termination` runs on `gpt-6.1-sol` while still rendering the `gpt-5.6-sol` overlay.
9
+
- **/setup-browser-cookies:** the workflow is now five ordered steps. It checks for Aside first and ends there on `READY`, since no import is needed. It then asks every decision (source and profile, domains and target, verification, storage reset, all cookies) before any `$B` command, prepares gstack's browser (verification variables before the daemon starts, the fallback, the `Mode: cdp` stop), imports, and reports on three lines: Imported, Storage reset, Sign-in. The `{{BROWSE_SETUP}}` block, which contradicted the Aside probe, is gone from this skill.
10
+
- **Gate integrity:** no threshold, trial count, case kind or assertion changed. Free-test edits follow the new cookie template: two phrase and order assertions, the approval fixture's reverse edits, and the positional-literals checksum list, which no longer includes this skill.
11
+
12
+
### Fixed
13
+
- **plan-review-prosons-hardstop-neg:** on Opus 5.5 one trial in three stopped with a provider refusal (`stop_reason: refusal`, category `reasoning_extraction`) on the turn that writes the captured question. Replays of the refused request were stopped 7 of 16 times with the capture prompt's "verbatim" and "Do NOT paraphrase. This is a format-capture test." wording and 0 of 24 without it. The shared prosons capture prompt now asks for "the text of the single AskUserQuestion you would have made", with the same required parts. The decision content and every assertion are unchanged; all four prosons cases pass 3/3.
14
+
- **Cookie workflow judge:** on `claude-sonnet-5-5` it scored clarity 3, completeness 4, actionability 3 against the old template. The rewritten template scores 4/4/4 with unchanged thresholds.
15
+
3
16
## [1.91.149.0] - 2026-10-19
4
17
5
18
**Paid eval cases can now run on the Codex host, and the Monday census runs two arms: Claude Opus 5.5 and Codex gpt-6.1-sol.** `EVALS_HOST=codex` runs the 52 adapter-only gate cases through a full Codex runtime per case. On its first pilot gpt-6.1-sol passed 48 of 52 with no retries, at $4.63 for the 52 cases, against $28.93 for the same cases on Fable (0.16×). Claude-only cases are skipped on Codex with a recorded reason and never count as failed. GSTA-49 TM2.
Copy file name to clipboardExpand all lines: docs/OVERLAY_BENCHMARK_CONTRACT.md
+5-3Lines changed: 5 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,8 +14,10 @@ Selection IDs for the six retained paid cases stay unchanged.
14
14
## Blocking checks
15
15
16
16
Both arms use the real Claude Code preset; ON appends the resolved overlay and
17
-
OFF does not. Each retained fixture runs ten samples per arm, separately for
18
-
`claude-opus-4-7` and `claude-sonnet-4-6`. Each model keeps its own results.
17
+
OFF does not. Each retained fixture runs ten samples per arm on its pinned
18
+
model: `claude-opus-5-5`, plus `claude-sonnet-5-5` for the dedicated-tools `-sonnet` case.
19
+
Each model keeps its own results. The `opus-4-7-*` cases keep their overlay
20
+
(`model-overlays/opus-4-7.md`) but run on Opus 5.5 since GSTA-49.
19
21
20
22
Every planned sample in both arms must execute successfully, produce exactly
21
23
one successful native terminal result without an error flag, preserve workspace
@@ -38,7 +40,7 @@ are not reinterpreted.
38
40
|`opus-4-7-effort-match-trivial`| Read-only workspace; native final JSON contains the exact version string | Correct answer in every ON sample | Native `usage.output_tokens_details.thinking_tokens`|
39
41
|`opus-4-7-literal-interpretation`| Only `src/auth.ts`, `src/billing.ts`, and `src/notifications.ts` may change; public tests stay frozen; independent behavior checks run outside the writable fixture | All three target behaviors pass in every ON sample | Correct target behaviors, 0..3 |
40
42
41
-
Each family also has a `-sonnet`case. Output validation checks the native final
43
+
The dedicated-tools family also has `-sonnet`and `-opus-5-5` cases. Output validation checks the native final
42
44
answer, so a correct word mentioned earlier cannot conceal a wrong final answer.
43
45
The read-only prompts explicitly request the JSON shape. These exact-answer
44
46
prompts are new in v2: compare ON/OFF within the same contract and fixture, not
Copy file name to clipboardExpand all lines: setup-browser-cookies/SKILL.md
+34-58Lines changed: 34 additions & 58 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -152,83 +152,59 @@ Skills that run plan reviews (`/plan-*-review`, `/codex review`) include the EXI
152
152
153
153
# Setup Browser Cookies
154
154
155
-
## 1. Choose the browser
155
+
## 1. Check for Aside
156
156
157
-
Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md`**BROWSER SETUP** section and run its probe first. On `READY`, stop importing: use Aside's sessions or ask the user to sign in there. Otherwise follow the probe's fallback handling, then continue below.
157
+
Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md`**BROWSER SETUP** section and run its Asideprobe.
158
158
159
-
## SETUP (run this check BEFORE any browse command)
159
+
-**Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill.
160
+
-**Any other result:** continue with gstack's own headless browser (`$B`).
if [ -x"$B" ] &&"${B%/browse/*}/bin/gstack-browser-ensure";then
167
-
echo"READY: $B"
168
-
else
169
-
echo"NEEDS_SETUP"
170
-
fi
171
-
```
162
+
## 2. Get every decision from the user first
172
163
173
-
`gstack-browser-ensure` installs Chromium on first use when the install skipped it (lazy browser); it prints `BROWSER_OK` or `BROWSER_UNAVAILABLE <reason>` first.
174
-
175
-
If `NEEDS_SETUP`:
176
-
1. Tell the user: "gstack browse needs a one-time build (~10 seconds). OK to proceed?" Then STOP and wait.
actual_sha=$(shasum -a 256 <"$tmpfile"| awk '{print $(1)}')
191
-
fi
192
-
if [ "$actual_sha"!="$BUN_INSTALL_SHA" ];then
193
-
echo"ERROR: bun install script checksum mismatch">&2
194
-
echo" expected: $BUN_INSTALL_SHA">&2
195
-
echo" got: $actual_sha">&2
196
-
rm "$tmpfile";exit 1
197
-
fi
198
-
bash "$tmpfile""bun-v$BUN_VERSION"
199
-
rm "$tmpfile"
200
-
fi
201
-
```
164
+
Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent.
202
165
203
-
```bash
204
-
$B status
205
-
```
206
-
If status says `Mode: cdp`, stop: the real browser already has sessions.
166
+
1.**Source browser and profile.**`--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty.
167
+
2.**Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL.
168
+
3.**Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately.
169
+
4.**Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back.
170
+
5.**Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage.
207
171
208
-
## 2. Confirm options before import
172
+
Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains.
209
173
210
-
Open the known target and keep its tab unchanged. Both options default off and require explicit request:
174
+
## 3. Prepare gstack's browser
211
175
212
-
-**`--verify-auth` / picker checkbox:** reloads the target. Have the user privately configure daemon `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY`**before startup**. Never invent values or assume CLI env reconfigures an existing daemon. Missing config rejects before mutation. Require a successful same-origin response and exactly one visible element whose whitespace-normalized text exactly matches the expected identity.
213
-
-**`--clear-storage`:** for suspected stale storage, obtain explicit approval. Chromium only: clears captured-origin localStorage (shared across same-origin context tabs) and target-tab sessionStorage. Other origins, other tabs' sessionStorage, IndexedDB, and service workers stay intact. It uses an isolated world/native deadline; other engines reject reset, not imports/auth checks. Never auto-approve or claim rollback after partial failure.
176
+
1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values.
177
+
2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`.
178
+
3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill.
179
+
4. Open the target URL from answer 2 and leave that tab alone until the import finishes:
214
180
215
-
## 3. Select source and scope
181
+
```bash
182
+
$B goto "<url>"
183
+
```
184
+
185
+
## 4. Import
186
+
187
+
Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires):
216
188
217
189
```bash
218
190
$B cookie-import-browser
219
191
```
220
192
221
-
Ask the user to choose browser, account/profile, and domains, then say when done. Never guess accounts or treat default Comet as consent. Unreadable profiles are unknown, not empty. Rerun for an expired five-minute one-use link.
193
+
Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2.
194
+
195
+
If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup.
222
196
223
-
Direct import: pass the chosen browser and `--domain` after navigating to a matching target. `--profile` takes a directory, not a display name; omit only for an unambiguous relevant profile, otherwise use the picker. `--all` requires consent for all non-expired profile cookies; it cannot accompany `--domain` or `--clear-storage`.
197
+
## 5. Report honestly
224
198
225
-
Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for examples, profile labels, supported sources and platform setup.
199
+
Report from the import receipt, not from raw `$B cookies`, as three lines:
226
200
227
-
## 4. Report honestly
201
+
-`Imported: <count> (<imported|partial|empty|failed>, as the receipt says) from <browser> for <domains>`
202
+
-`Storage reset: <done|failed|not requested>`
203
+
-`Sign-in: <verified|not verified|not checked>`
228
204
229
-
Report receipt/picker counts, partial/zero/error and reset outcomes, not raw `$B cookies`. Imports affect the context, not one tab. **Not checked** means no requested check; **not verified** means it failed; **verified**requires positive target evidence. Zero imports, counts, or HTTP 200 never prove login.
205
+
**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified**needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab.
230
206
231
-
Never request/publish cookie values, passwords, identity/profile text, session details, or raw errors in public logs.
207
+
Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs.
0 commit comments