Repository navigation
v1.91.149.0 feat(evals): Codex host for gate cases, sol weekly and Fable/Astra monthly census arms (GSTA-49 TM2) - #3292
Merged
Merged
Conversation
…s Fable (GSTA-49 TM1) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…monthly census (GSTA-49 TM2) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md # VERSION # agents-digest/gstack-AGENTS.md # package.json # scripts/harness-version.json
…the harness manifest after the main merge Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
github-actions
Bot
changed the title
|
😎 This pull request was merged. |
…request run fired) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Contributor
E2E Evals: ❌ FAIL121 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $32.47 rule/judge cost | reconcile exit: 1 Profile: pr / mode Derived dependents (scripts/pr-dependencies.ts):
Rule and judge shards
Fail-closed reconciliationSliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry. Failures and split verdicts |
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md # scripts/harness-version.json
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
github-actions
Bot
changed the title
Contributor
|
flaky: plan-ceo-finding-floor, failed in run 39293281037 (attempt 1, slice 6), passed on rerun 39293281037 (attempt 2, slice 6), logged for the daily flaky-fix pass |
Contributor
|
Merging under Garry's flaky-test rule (2026-10-19: "you can just run the failed test again and note that it is flaky"). plan-ceo-finding-floor failed in attempt 1 of run 39293281037 and passed on the slice rerun (attempt 2). Every slice passed; slices-report stays bound to attempt 1 by design. Logged in the flaky ledger. — GStack Minder |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lane TM2 of GSTA-49. It lets gstack's paid E2E cases run on the Codex host, and puts the model tiers Garry asked for on the census schedules.
Garry's direction, 2026-10-18: test the best code model from each lab, gpt-6.1-sol and claude-opus-5-5. Drop the older frontier models, and run the most expensive ones (claude-fable-5-1 and gpt-6-astra) only monthly.
Codex host
Host switch.
EVALS_HOSTselects the host:claude(the default, unchanged) orcodex. Each case carries host metadata intest/helpers/e2e-hosts.ts:On Codex, Claude-only cases are SKIPPED with a recorded reason. They never count as failed.
Runner (
test/helpers/codex-host-runner.ts). Each case gets the full Codex runtime. The runner maps file-change events, applies sandbox settings, and passes a reasoning-effort setting (model default unless set). Codex events are rewritten into the Claude stream-json shape the existing checks read, and the runner supports native lifecycle hooks.Real cost.
codex-eval.tsno longer hard-codescost_known: false. Cost now comes fromturn.completedusage: input, cached input, cache-write input and output, priced fromlib/pricing.ts. Plan, report and shard-reuse records carry the host; the fields are absent when the host isclaude.Model tiers (
evals-periodic.yml,scripts/census-arms.ts)0 6 1 * *: census jobs only, with a Fable arm and an Astra arm. It uses a-monthlyconcurrency key and acensus_armsdispatch input.evals.ymlare untouched. Weekly periodic and marathon runs move to Opus 5.5 through TM1's capture default; nothing pins them otherwise.Sol pilot
All 52 adapter cases ran once on gpt-6.1-sol, with no retries.
Gate-integrity edits
Five workflow tests now assert per arm instead of on one census manifest. No assertion was dropped.
include: gate_matrix, and run-id uniqueness is checked per arm × slice.executor.ifisperiodic_lane == 'true', so a census-only run executes no periodic slice.census-arms.ts./tmp/gate-census-plan/$arm/manifest.json.No threshold, trial count or case kind changed. The harness manifest re-records the new runner as non-behavioral at version 6.
Checks
bun run typecheckis clean, andtypecheck:testreports no new diagnostics.