eval_flagship.yml

# Copyright(C) 2025-2026 Advanced Micro Devices, Inc. All rights reserved.
# SPDX-License-Identifier: MIT

# The flagship agent's eval gate: every scenario category, scored against the
# agent users actually meet.
#
# Unit tests cover code paths, not model behavior, so this is the only thing that
# catches a behavioral regression. It runs on release and RC tags (see `on:`);
# CLAUDE.md says how an LLM-affecting PR relies on the next RC's run.
#
# Runner-only by construction. `gaia eval agent` drives the real GAIA UI backend
# against a real Lemonade Server on AMD hardware; a GitHub-hosted runner has
# neither. Hence the self-hosted pool.
#
# WHY THIS RUNS IN PARALLEL LANES
# ------------------------------------------------------------------------------
# 82 scenarios on one serial Lemonade slot is ~5.8 h expected and 9.2 h at the
# observed ceiling, holding a shared single-slot pool for the whole time. That is
# not a PR gate. The categories are split into independent LANES (eval/ci_lanes.json)
# that run at the same time on the EPHEMERAL `devlab-dispatch` pool, where each job
# gets its own machine and therefore its own Lemonade.
#
# This does not weaken CLAUDE.md's "run evals serially" rule. That rule exists
# because two `gaia eval agent` processes sharing ONE Lemonade race-evict each
# other's model out of the single slot — it is per-BACKEND, not global. Separate
# machines have separate backends and cannot collide. Within a lane the categories
# still run serially in one foreach, which is the part the rule actually governs.
#
# Do NOT give the lanes a shared concurrency group. `lemonade-eval` serialises by
# group NAME regardless of which machine a job lands on, so a single shared group
# would silently put the lanes back in a queue and undo all of this. Each lane
# holds its own group; see the job-level `concurrency:` blocks below.
#
# WHAT GETS UPLOADED, AND WHY THAT CHANGED
# ------------------------------------------------------------------------------
# `devlab-dispatch` DESTROYS the machine when the job ends. Anything not uploaded
# is gone permanently — so the old "transcripts and logs stay on the runner,
# reproduce locally" posture would mean every failure this eval finds is
# unreadable. Each lane uploads its scorecards, per-scenario traces (transcript,
# judge reasoning, root cause), and the backend + Lemonade logs, on `always()`,
# because a crashed lane is exactly when they matter. Everything bound for the
# artifact is secret-scanned first: the artifact is PUBLIC and, unlike the job
# log, not masked by GitHub.
#
# THIS POOL IS WINDOWS. Every inline `run:` step below is PowerShell, and Python
# setup goes through ./.github/actions/setup-venv (uv, Windows-aware) — the
# lab-runner convention shared by test_email_agent_eval.yml and
# email_scorecard_refresh.yml. Do NOT reintroduce
# bash here: this workflow's first-ever execution died instantly on
# `source .venv/bin/activate` because GitHub ran it through powershell.EXE, and
# every other bash step in the job hit the same parser. Windows also breaks the
# text-processing assumptions bash invites — the eval prints an `Output:` line
# containing a backslash path, which bash test/`[ -f ]` cannot consume.
#
# WHY THIS REPORTS BY DEFAULT INSTEAD OF HARD-FAILING (`enforce`, default false)
# ---------------------------------------------------------------------------
# `gaia eval agent --compare BASELINE CURRENT` already exits 2 on any status /
# score / time regression and 0 otherwise (src/gaia/cli.py). Wiring that exit
# code straight to the step result would make this job PERMANENTLY RED, because
# some regressions here are EXPECTED and accepted: code-generation and
# data-analysis scenarios are dropping from a 35B MoE to a ~4B dense model. A
# gate that is always red gates nothing — people stop reading it.
#
# So the default posture is REPORT: every category is compared, the full diff is
# printed, the scorecards are uploaded as an artifact, and a regression surfaces
# as a `::warning::` annotation. Make it blocking by dispatching with
# `enforce: true`; a tag run always reports.
#
# This mirrors how intentional regressions are handled elsewhere: the
# `--allow-regression` flag in src/gaia/eval/scorecard_gate.py prints a
# `::warning::` and returns 0 rather than pretending the regression did not
# happen, and test_email_agent_eval.yml ships its gates in report mode until the
# bars are trustworthy. Same idea, inverted default: report unless asked to
# enforce.
#
# REPORT MODE COVERS "WORSE" AND "DIDN'T MEASURE" ALIKE — VISIBLY. `--compare`
# scores only scenarios present on BOTH sides: scenarios that vanish from the
# current run land in an `only_in_baseline` bucket that is NOT part of its exit-2
# verdict (runner.py::compare_scorecards). Verified: dropping 6 of 7 rag_quality
# scenarios still exits 0. So a broken harness would otherwise sail through as "no
# regression". The integrity gate below is what refuses that: it fails if a
# baseline scenario is missing or any scenario produced no measurement
# (infra_error / errored / timeout / blocked / budget_exceeded / skipped).
#
# That gate used to ignore `enforce` and fail unconditionally, on the reasoning
# that report mode may soften "worse" but never "did not measure". The reasoning
# is sound and the outcome was not: the embedder cannot load on this runner
# (#3016), which is an environment defect no PR can fix, so the gate was red on
# every PR touching the path filter — including seven whose diffs never went near
# the eval — while this same job printed "report mode ... build stays green". A
# check that is always red gates nothing, and one that advertises non-blocking
# while blocking is worse than either honest posture.
#
# So `enforce` now decides the EXIT CODE for an incomplete run as well as a worse
# one. It never decides VISIBILITY: in report mode every finding is still a
# `::warning::` annotation AND a block in the job summary, so an incomplete run
# cannot be mistaken for a complete one. `enforce: true` restores blocking for
# both.
#
# ONE EXCEPTION, AND IT IS ABOUT BLAME, NOT CONVENIENCE: when the RAG embedder
# will not load on the runner at all (#3016, EMBEDDER_OK=false), rag_quality and
# context_retention warn instead of failing. Every other non-measurement says
# something about the PR under test; #3016 says only that this SoC cannot serve
# the embedder for ANY commit, so failing on it made the gate permanently red
# and unread. Nothing else is softened — tool_selection's own non-measurements
# still fail. Remove the exception when #3016 closes. See the gate step below.
#
# TIME REGRESSIONS. `--compare` folds a >2x per-scenario wall-clock delta into
# the same exit 2 as a quality regression. The baseline was captured on this
# pool, so a time regression is comparable, but it comes from one run: re-run a
# lone time-only flip before treating it as a defect.
#
# SEQUENCING. `--compare` only DIFFS two scorecards — it does NOT run an eval.
# So each category is evaluated first, the ABSOLUTE run directory is parsed from
# the eval's own `Output: <run-dir>` line, and `<run-dir>/scorecard.json` is then
# compared against the committed Gemma baseline. Never glob by mtime — a fresh
# clone stamps every file with the checkout time.
#
# PREREQUISITES ON THE RUNNER. Lemonade is NOT assumed to be already running:
# install-lemonade pins the runner to the expected LEMONADE_VERSION and
# ensure-lemonade-running.ps1 starts it as a persistent Scheduled Task and warms
# the model — exactly what test_email_agent_eval.yml and
# email_scorecard_refresh.yml do. (An earlier revision of this file claimed those
# two workflows assume a persistent server and start nothing; they do not, and
# that claim is what left this job with no way to bring Lemonade up.) The GAIA UI
# backend is separate and NOT persistent, so this workflow starts it itself — in
# the SAME shell session as the evals, because a backgrounded child does not
# survive a GitHub Actions step boundary.
#
# Lemonade must serve TWO models, not one. ensure-lemonade-running.ps1 warms
# only the chat model, but rag_quality and context_retention answer out of RAG
# chunks, so the embedder is just as load-bearing — and when it cannot start,
# those categories collapse to ~2/10 non-answers that read as a model
# regression. The preflight therefore re-registers, re-pulls and then makes a
# real /embeddings call through tests/ci_lemonade_check.py — repairing the one
# cause CI can repair (a stale registration missing the embeddings label) and
# failing in seconds on the ones it cannot, instead of spending three and a half
# hours measuring nothing.
#
# The third prerequisite is the Claude Code CLI, which is easy to miss because
# ANTHROPIC_API_KEY looks like the whole story: `gaia eval agent` DRIVES each
# scenario by shelling out to `claude -p` with an MCP config, and its preflight
# refuses to start without it on PATH. Installed below via npm, as
# docs/guides/eval-ci.mdx prescribes.
#
# WHAT TRIGGERS IT, AND HOW MUCH RUNS. Tiers, from eval/ci_tiers.json via
# util/eval_tiers.py (#5493 section 6.6):
#
#   RC tag (v*-rc*)            smoke: 30 stable scenarios, 13 protected, 3 machines
#   RC tag with [full eval]    full: every lane minus the weekly backlog's
#     in its annotation          leaves_full scenarios. Nominate the RC you
#                                will promote; [skip eval] still skips
#   final tag (vX.Y.Z)         verify: nothing re-runs (the release train
#                                promotes the measured RC, #5492); verify-final
#                                checks a full sweep measured this commit or an RC
#   weekly (Sun 06:17 UTC)     backlog: always-fail and high-flip scenarios on 3
#                                machines, with a rolling trend artifact. Skipped
#                                while any other run of this workflow holds the pool
#   dispatch                   `tier` (default smoke), or a targeted selection
#                                through `categories` / `scenarios`
#
# THE MODEL UNDER TEST on this pool is Qwen3.6-35B-A3B, what `gaia init`
# installs on a 64 GB+ Strix Halo, pinned in every category's GAIA home and
# checked per scenario (--agent-model). It compares only against the Qwen
# baseline (tests/fixtures/eval_baselines/gaia-flagship-qwen); until one is
# committed every category reports "no baseline yet". Gemma-4-E4B is scored on
# the Strix Point pool. A dispatch can still pick `model: Gemma-4-E4B-it-GGUF`
# here, which compares against the Gemma-on-Halo baseline in gaia-flagship/ -
# the way a targeted re-run joins an earlier Gemma tag run.
#
# FORKS AND DRAFTS. The job-level `if` guards below still skip fork and draft
# PRs, in case a `pull_request` trigger is ever added back: a fork PR could not
# pass (ANTHROPIC_API_KEY is not exposed to it) and must not put unreviewed code
# on the lab pool. If the key is ever missing, the preflight fails with
# instructions instead of running a judge-less eval.

name: Agent Eval — flagship

on:
  # Release and release-candidate tags (v0.24.1, v0.25.0-rc14), one weekly
  # backlog run, and manual dispatch. Not per-PR and not nightly: the Strix Halo
  # pool is too scarce for either - per-PR runs queued for hours and had to be
  # cancelled. An LLM-affecting PR names the categories to watch; the next RC's
  # run covers it. A tag run checks out the tagged commit, compares it with the
  # committed baseline in report mode, and files an issue when it does not pass
  # (release-escalation below). `v*.*.*` already matches the -rc tags; the
  # second pattern says so out loud. Agent-package tags (agent-pkg-*) match
  # neither.
  push:
    tags:
      - 'v*.*.*'
      - 'v*.*.*-rc*'
  # The weekly backlog lane (maintainer-approved). Sunday 06:17 UTC is the
  # quietest hour on the pool; the minute is off the hour so it does not land
  # with every other cron. Owner and README: eval/ci_tiers.json, docs/reference/eval.mdx.
  schedule:
    - cron: '17 6 * * 0'
  workflow_dispatch:
    inputs:
      enforce:
        description: 'Fail the build on a regression (false = report as ::warning:: only)'
        required: false
        type: boolean
        default: false
      eval_model:
        description: 'Driver + judge model for the eval harness. Blank = the model the flagship baseline was captured under (tests/fixtures/eval_baselines/gaia-flagship/meta.json)'
        required: false
        default: ''
      deep_diagnostics:
        description: 'Run the full embedder-failure ladder (purges the model from disk, unloads all models). Off by default - it costs ~7 extra min and disturbs other jobs on the box.'
        required: false
        type: boolean
        default: false
      task_suite:
        description: 'Space-separated suites from eval/tasks/tasks.json (a tag run uses everyday therock; adversarial is the traps; full is everything)'
        required: false
        default: 'core'
      tier:
        description: 'smoke (30 scenarios, 3 machines), full (the promotion sweep, ~13 machines), or backlog (the weekly lane). Ignored when categories or scenarios is set.'
        required: false
        type: choice
        options: [smoke, full, backlog]
        default: smoke
      # Qwen is what a 64 GB+ Strix Halo runs. Gemma compares against the
      # Gemma-on-Halo baseline: use it to re-run part of a Gemma tag run.
      model:
        description: 'Local model under test. Each compares only against its own baseline (eval/ci_tiers.json).'
        required: false
        type: choice
        options: [Qwen3.6-35B-A3B-GGUF, Gemma-4-E4B-it-GGUF]
        default: Qwen3.6-35B-A3B-GGUF
      # Targeted runs. Blank both to run the `tier`. Validated by
      # util/eval_lane_matrix.py before any lane takes a lab machine.
      categories:
        description: 'Comma-separated categories to run (blank = the tier). Add "outcomes" to also run the outcomes job, which a targeted run otherwise skips.'
        required: false
        default: ''
      scenarios:
        description: 'Comma-separated scenario ids to run (blank = every scenario in the selected categories). Only lanes holding them get a machine.'
        required: false
        default: ''

# NO WORKFLOW-LEVEL CONCURRENCY, deliberately. A group declared here is held by
# the whole run, so every lane would wait on the one the first lane took and the
# split would buy nothing — the same shape as the phantom-job bug this repo has
# already paid for once (a caller holding the group its callee needed; see
# test_email_agent_eval.yml). Each lane declares its own group at job level,
# where `matrix.lane` is in scope.
#
# `lemonade-eval` is gone from this file on purpose and must not come back. It
# serialises by NAME across machines, which is right for the persistent pool the
# email eval still runs on and wrong for the ephemeral one these lanes use.

permissions:
  contents: read

env:
  # No BASELINE_DIR here: it depends on the model under test, so the lanes job
  # resolves it from eval/ci_tiers.json (`baselines`) and each job sets it.
  # A baseline holds one scorecard per category, copied unedited from a real run
  # on this pool; its meta.json names the model, the run and the eval_model every
  # job runs under unless a dispatch overrides it. Never edit a number in one.
  #
  # THE LANE MAP IS THE SOURCE OF TRUTH: eval/ci_lanes.json. The `lanes` job below
  # reads it, checks it, and hands each lane its own category list as
  # EVAL_CATEGORIES at job level — which is what the four steps downstream already
  # consume, so they keep working unchanged and still read one list, not four.
  #
  # NEVER ADD web_system OR vision TO A LANE: they read the runner's clipboard,
  # windows and screen into a scorecard that uploads as a PUBLIC artifact. They
  # are in the map's `excluded` block, the `lanes` job re-checks it at runtime,
  # and tests/unit/eval/test_ci_lanes.py fails the build if either appears.
  EVAL_LANES_FILE: eval/ci_lanes.json
  # What still means something when the RAG embedder will not load (#3016).
  # Not the whole list above: for the categories left out, only a real run says
  # whether they stay interpretable without retrieval, and guessing would put
  # unmeasurable categories in a scorecard. `memory` needs no such run - 0 of
  # its 25 scenarios carry a non-empty `setup.index_documents`, so the embedder
  # is not what would stop them scoring, and leaving it out meant a degraded
  # run measured 2 scenarios out of 82.
  # Scenarios inside a listed category may still fail on their own - 3 of
  # tool_selection's 5 do index documents. That is the posture on purpose: run
  # the category, let the scenario fail, do not call the whole category
  # unmeasurable.
  # BEFORE WIDENING FURTHER, do the timeout arithmetic - but do it per LANE,
  # not across this list. A degraded run only ever cuts a lane down to its own
  # categories (the intersection in the run step), so the budget that can
  # overflow is the lane's, and it is always smaller than the full run the job
  # ceiling above was sized for. Widening is therefore free on time and costs
  # only interpretability; what it must not do is add a category no run has
  # shown to be measurable without retrieval.
  # Everything in EVAL_CATEGORIES but not here is reported NOT MEASURED.
  EVAL_CATEGORIES_NO_EMBEDDER: memory tool_selection
  BACKEND_URL: http://127.0.0.1:4200
  # This runner's home is C:\Windows\System32\config\systemprofile and the
  # checkout is not under it, so without this every scenario carrying a
  # `setup.index_documents` block 403s and scores as a failure (#3585).
  # Declares the checkout's corpus, nothing wider.
  GAIA_DOCUMENT_ROOTS: ${{ github.workspace }}\eval\corpus
  # Only a dispatch can enforce; a tag run compares with the baseline and reports.
  ENFORCE: ${{ github.event.inputs.enforce || 'false' }}
  # EVAL_MODEL is not set here: each job reads it from $BASELINE_DIR/meta.json
  # (eval_model), because `--model` is not a judge-only knob. runner.py passes it
  # to `claude -p`, which DRIVES every scenario as well as scoring it, so the
  # driver is part of the agent under test and must be the one the baseline was
  # recorded under. It changes only with a re-captured baseline. #2859 moved
  # every other Claude workflow to Opus 5. A dispatch input overrides it per run.
  EVAL_MODEL_OVERRIDE: ${{ github.event.inputs.eval_model }}
  # Deep ladder is opt-in. Its rungs purge the embedder from disk and unload every
  # resident model, which is fine while investigating and wrong to repeat on every
  # PR: the cause is known (#3016) and the box is shared, so the next job landing
  # on it pays for a cold reload it did not ask for.
  DEEP_DIAG: ${{ github.event.inputs.deep_diagnostics || 'false' }}
  # Loopback services — bypass any runner proxy. This covers the PYTHON side
  # (httpx/requests in the eval, the agents and the backend); PowerShell's
  # Invoke-RestMethod ignores NO_PROXY and uses the system proxy instead.
  NO_PROXY: 'localhost,127.0.0.1'
  # Bare host (no /api/v1): LemonadeClient normalizes the suffix itself. Mirrors
  # test_email_agent_eval.yml so the preflight probe and the agents agree.
  LEMONADE_BASE_URL: 'http://127.0.0.1:13305'
  # The eval prints box-drawing characters and em dashes; the runner's default
  # console codepage is cp1252 and Python dies on the first one without this.
  PYTHONIOENCODING: 'utf-8'
  # Connector resolution blocks on the OS keyring in a headless runner, which
  # hangs agent construction. Same guard as test_email_agent_eval.yml.
  PYTHON_KEYRING_BACKEND: keyring.backends.null.Keyring
  # The default for every lane; the memory and gaia_* lanes override all three at
  # job level (util/eval_lane_matrix.py).
  # Long-term memory surviving between categories makes a later category's score
  # depend on an earlier one's run order (convention: test_chat_agent.yml,
  # test_email_agent_eval.yml). The memory lane is the one place that state IS
  # the subject, and it can opt back in safely because every category runs on
  # its own fresh GAIA home (see the run step). Do not rely on the machine being
  # wiped: the runner's ~/.gaia survived between runs and leaked memory (rc16).
  #
  # Do NOT flip these here. Workflow scope would re-enable memory for every lane
  # and bring back the cross-category bleed this is guarding.
  GAIA_MEMORY_DISABLED: '1'
  # Both gate the memory surface the scenarios verify through, and both are read
  # from the environment only: nothing in src/gaia/ sets them, despite two code
  # comments that say the eval runner does. ADMIN gates memory_clear / the admin
  # REST endpoints the runner's own memory preflight probes; MCP_ALWAYS gates
  # whether the Agent UI MCP server registers the memory READ tools at all.
  # Without the second, a scenario's "VERIFY VIA MCP: memory_recall(...)" has no
  # tool to call and the judge is left scoring the reply text alone.
  GAIA_MEMORY_ADMIN: '0'
  GAIA_MEMORY_MCP_ALWAYS: '0'
  # Lets a scenario's setup.decline_commands script a user who says no
  # (gaia_resilience); inert for every scenario that does not set it.
  GAIA_EVAL_SCRIPTED_USER: '1'

jobs:
  # An RC tag annotated with `[skip eval]` skips every lane below; a final
  # release, a lightweight tag and a dispatch always run (util/eval_tag_gate.py).
  # Then the tier (util/eval_tiers.py): RC smoke, [full eval] full sweep, final
  # verify, weekly backlog, or a dispatch's choice. Hosted and seconds long, so
  # the decision costs no lab-pool time.
  eval-gate:
    name: Eval gate (release tag)
    runs-on: ubuntu-latest
    permissions:
      contents: read
      # The weekly run lists this workflow's queued and running runs, to stay
      # off the pool while a release eval is on it.
      actions: read
    outputs:
      run: ${{ steps.gate.outputs.run }}
      tier: ${{ steps.tier.outputs.tier }}
    steps:
      - name: Checkout
        uses: actions/checkout@v7
      - name: Read the tag annotation
        id: gate
        run: python util/eval_tag_gate.py
      - name: Pick the tier
        id: tier
        if: steps.gate.outputs.run == 'true'
        env:
          GH_TOKEN: ${{ github.token }}
          EVAL_DISPATCH_TIER: ${{ github.event_name == 'workflow_dispatch' && inputs.tier || '' }}
          EVAL_SELECT_CATEGORIES: ${{ github.event_name == 'workflow_dispatch' && inputs.categories || '' }}
          EVAL_SELECT_SCENARIOS: ${{ github.event_name == 'workflow_dispatch' && inputs.scenarios || '' }}
        run: python util/eval_tiers.py decide >> "$GITHUB_OUTPUT"

  # Build the lane matrix, and refuse to build a bad one. Runs on a hosted runner
  # in seconds, BEFORE anything reaches the scarce AMD pool, so a broken lane map
  # costs nothing to discover.
  #
  # It checks the two things a matrix cannot check itself. A category that belongs
  # to no lane is never passed to `--category`, so it simply stops being measured
  # while every lane still reports green — an absence that looks exactly like a
  # pass. And `vision` / `web_system` in a lane would publish the runner's screen
  # and clipboard in a world-readable artifact. Both fail here rather than being
  # left to a comment nobody re-reads.
  lanes:
    name: Resolve the lane matrix
    needs: eval-gate
    if: >-
      !cancelled() && needs.eval-gate.result == 'success' &&
      needs.eval-gate.outputs.run == 'true'
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.build.outputs.matrix }}
      has_lanes: ${{ steps.build.outputs.has_lanes }}
      targeted: ${{ steps.build.outputs.targeted }}
      outcomes: ${{ steps.build.outputs.outcomes }}
      selection: ${{ steps.build.outputs.selection }}
      tier: ${{ steps.build.outputs.tier }}
      agent_model: ${{ steps.build.outputs.agent_model }}
      baseline_dir: ${{ steps.build.outputs.baseline_dir }}
    steps:
      - name: Checkout
        uses: actions/checkout@v7

      - name: Validate the lane map and tiers, and emit the matrix
        id: build
        # The rules (which lane gets memory, which gets the gaia_* fixtures,
        # what may never be in a lane, what a dispatch may select) live in
        # util/eval_lane_matrix.py; which scenarios each tier runs, on which
        # machine, against which model's baseline, in util/eval_tiers.py and
        # eval/ci_tiers.json. It re-runs the tier coverage check first, so a bad
        # tier file fails here in seconds. Tests: tests/unit/eval/test_eval_lane_matrix.py,
        # tests/unit/eval/test_eval_tiers.py. Inputs arrive through env, never
        # inline in the script.
        env:
          EVAL_TIER: ${{ needs.eval-gate.outputs.tier }}
          EVAL_AGENT_MODEL: ${{ github.event_name == 'workflow_dispatch' && inputs.model || '' }}
          EVAL_SELECT_CATEGORIES: ${{ github.event_name == 'workflow_dispatch' && inputs.categories || '' }}
          EVAL_SELECT_SCENARIOS: ${{ github.event_name == 'workflow_dispatch' && inputs.scenarios || '' }}
        run: python util/eval_tiers.py matrix >> "$GITHUB_OUTPUT"

      - name: Label a partial run in the summary
        if: steps.build.outputs.targeted == 'true'
        env:
          SELECTION: ${{ steps.build.outputs.selection }}
          TIER: ${{ steps.build.outputs.tier }}
          CATEGORIES: ${{ inputs.categories }}
          SCENARIOS: ${{ inputs.scenarios }}
        run: |
          {
            echo "## ${SELECTION}"
            echo ""
            echo "This is NOT a full sweep. Only the selection below ran; every other scenario has no result in this run."
            echo ""
            if [ "${TIER}" = "targeted" ]; then
              echo "- categories input: \`${CATEGORIES:-<blank>}\`"
              echo "- scenarios input: \`${SCENARIOS:-<blank>}\`"
            else
              echo "- tier \`${TIER}\`: the scenarios and the reason each is in it are in \`eval/ci_tiers.json\`"
            fi
          } >> "$GITHUB_STEP_SUMMARY"

  # The flagship GaiaAgent on outcome-scored tasks (`gaia eval tasks`, docs:
  # docs/reference/eval.mdx), served by GAIA's own embedded Lemonade. Three steps
  # with separate environments: the agent runs with no credentials (it runs
  # shell commands), Claude judges quality (and decides the question tasks) in
  # one call with no tools, and the gate compares passes, tested changes,
  # quality, honesty, tokens, steps and runtime with main's committed results in
  # eval/tasks/expectations/<model>.<suite>.json. The gate always enforces: a
  # missed expectation fails this job. A model with no committed expectations
  # is reported, not failed, and a task the backend could not serve is "not
  # measured", never a regression. Memory is off here, as it is outside the
  # memory and gaia_* lanes.
  #
  # Which suites: a release or RC tag runs `everyday` (the 14 tasks the
  # published model rankings cite) and `therock` (four real fixes in a
  # 144K-line repo). A dispatch runs the suites it names (default `core`, the
  # traps, ~7 tasks). Each suite is judged and gated on its own against
  # eval/tasks/expectations/<model>.<suite>.json once they are committed (from
  # `gaia eval tasks propose` over repeated TASK_MODEL runs on this pool); until
  # then the gate reports only. A failing tag run files an issue naming the
  # suite (release-escalation below).
  #
  # A lane in everything but the matrix. It stays its own job because its
  # steps are a different shape - embedded Lemonade, `gaia eval tasks` - not
  # because it is ordered against the others; it runs alongside them.
  #
  # Only the full sweep runs it, so the RC smoke tier stays on three machines.
  # A targeted dispatch skips it unless its `categories` input names
  # `outcomes`; the weekly backlog and a final tag never run it. The lanes job
  # decides (`outcomes` output) for every event.
  flagship-tasks:
    name: Scenario eval - outcomes
    needs: [eval-gate, lanes]
    outputs:
      failed_suites: ${{ steps.gate.outputs.failed }}
    if: >-
      !cancelled() && needs.eval-gate.result == 'success' &&
      needs.eval-gate.outputs.run == 'true' && needs.lanes.result == 'success' &&
      needs.lanes.outputs.outcomes == 'true' &&
      (github.event_name != 'pull_request' ||
      (github.event.pull_request.head.repo.full_name == github.repository &&
      (github.event.pull_request.draft == false ||
      contains(github.event.pull_request.labels.*.name, 'ready_for_ci'))))
    concurrency:
      group: ${{ github.workflow }}-outcomes-${{ github.ref }}
      cancel-in-progress: false
    runs-on: [self-hosted, Windows, devlab-dispatch, strix-halo]
    defaults:
      run:
        # Windows PowerShell 5.1 and pure-ASCII run bodies, for the reasons
        # spelled out on the agent-eval job below.
        shell: powershell
    # At least the sum of the step limits below plus setup, or a job-level
    # timeout cancels and skips the judge, gate and upload steps.
    timeout-minutes: ${{ github.event_name != 'pull_request' && 420 || 200 }}
    env:
      # The venv's interpreter and CLI by full path. A bare `python` resolves
      # through PATH, which a runner's own setup can put ahead of the venv.
      VENV_PY: ${{ github.workspace }}\.venv\Scripts\python.exe
      VENV_GAIA: ${{ github.workspace }}\.venv\Scripts\gaia.exe
      TASK_SUITES: ${{ github.event.inputs.task_suite || (github.event_name == 'push' && 'everyday therock') || 'core' }}
      # The local model the expectations were measured on. Changing it means
      # measuring and committing new expectations for it first.
      TASK_MODEL: Qwen3.6-35B-A3B-GGUF
      TASK_JUDGE_MODEL: claude-opus-5-5
      TASKS_OUT: eval-out\tasks
    steps:
      - name: Show which Claude account this job uses
        run: echo "::notice title=Claude account::${{ vars.CLAUDE_ACCOUNT != 'secondary' && 'This job uses the primary Claude account.' || secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY != '' && 'This job uses the secondary Claude account.' || 'CLAUDE_ACCOUNT is secondary but CLAUDE_CODE_OAUTH_TOKEN_SECONDARY is not set, so this job uses the primary Claude account.' }}"
      - name: Checkout
        uses: actions/checkout@v7

      - name: Setup Python Environment
        id: venv
        uses: ./.github/actions/setup-venv
        with:
          python-version: '3.12'
          # The flagship ships as its own hub package and builds on ChatAgent.
          install-package: '-e .[dev,eval,ui,api] -e hub/agents/chat/python -e hub/agents/gaia/python'

      - name: Check the venv interpreter
        env:
          SETUP_VENV_PY: ${{ steps.venv.outputs.python-path }}
        run: |
          if ($env:SETUP_VENV_PY -ne $env:VENV_PY) {
            Write-Host "::error::setup-venv built '$env:SETUP_VENV_PY' but this job runs '$env:VENV_PY'. Point VENV_PY/VENV_GAIA in the job env at setup-venv's venv-name."
            exit 1
          }
          foreach ($exe in @($env:VENV_PY, $env:VENV_GAIA)) {
            if (-not (Test-Path $exe)) {
              Write-Host "::error::$exe is missing - the venv install did not complete. See the Setup Python Environment step."
              exit 1
            }
          }
          & $env:VENV_PY -c "import gaia, gaia_agent, gaia_agent_chat"
          if ($LASTEXITCODE -ne 0) {
            Write-Host "::error::$env:VENV_PY cannot import gaia / gaia_agent / gaia_agent_chat - the editable installs in the Setup Python Environment step are broken."
            exit 1
          }
          $bare = (Get-Command python -ErrorAction SilentlyContinue).Source
          Write-Host "Venv interpreter: $env:VENV_PY"
          Write-Host "Bare 'python' on this runner: $(if ($bare) { $bare } else { 'none' })"

      - name: Set up Node
        uses: actions/setup-node@v7
        with:
          node-version: '22'

      - name: Install Claude Code CLI (the judge)
        run: |
          npm install -g @anthropic-ai/claude-code
          if ($LASTEXITCODE -ne 0) { throw "Claude Code CLI install failed" }
          claude --version
          if ($LASTEXITCODE -ne 0) { throw "claude installed but not runnable on PATH" }

      - name: Start GAIA's embedded Lemonade
        timeout-minutes: 20
        run: |
          $ErrorActionPreference = "Continue"
          & $env:VENV_GAIA lemonade embedded start --timeout 180
          if ($LASTEXITCODE -ne 0) {
            Get-Content "$HOME\.gaia\lemonade\lemond.log" -Tail 60 -ErrorAction SilentlyContinue
            throw "The embedded Lemonade did not start; its log is above."
          }
          & $env:VENV_GAIA lemonade embedded install-backend llamacpp:vulkan
          if ($LASTEXITCODE -ne 0) { throw "Could not install llamacpp:vulkan into the embedded Lemonade." }
          & $env:VENV_GAIA lemonade embedded status

      # This pool is ephemeral, so the model is never cached. Download it here,
      # or the first task's clock pays for a 23 GB download.
      - name: Download the task model
        timeout-minutes: 60
        run: |
          . "$HOME\.gaia\lemonade\env.ps1"
          & $env:VENV_PY -c "import os, sys; from gaia.llm.lemonade_client import LemonadeClient; sys.exit(0 if LemonadeClient().ensure_model_downloaded(os.environ['TASK_MODEL'], timeout=3300) else 1)"
          if ($LASTEXITCODE -ne 0) { throw "Could not download $env:TASK_MODEL into the embedded Lemonade; the error is above." }

      # No credentials in this step's environment: the agent runs shell commands.
      - name: Run the flagship on the task suites
        timeout-minutes: ${{ github.event_name != 'pull_request' && 240 || 60 }}
        run: |
          $ErrorActionPreference = "Continue"
          [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
          . "$HOME\.gaia\lemonade\env.ps1"
          # One suite crashing must not stop the next from being measured.
          $crashed = @()
          foreach ($suite in ($env:TASK_SUITES -split ' ')) {
            & $env:VENV_GAIA eval tasks run --suite $suite --model $env:TASK_MODEL --no-judge --out "$env:TASKS_OUT\$suite"
            if ($LASTEXITCODE -ne 0) { $crashed += "$suite (exit $LASTEXITCODE)" }
          }
          if ($crashed.Count -gt 0) { throw "gaia eval tasks run crashed for: $($crashed -join ', '). A harness failure, not a verdict." }

      - name: Judge quality
        if: ${{ !cancelled() }}
        timeout-minutes: ${{ github.event_name != 'pull_request' && 60 || 30 }}
        env:
          CLAUDE_CODE_OAUTH_TOKEN: ${{ (vars.CLAUDE_ACCOUNT == 'secondary' && secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY || secrets.CLAUDE_CODE_OAUTH_TOKEN) }}
          # Blank when the OAuth token is set: the judge adds --bare only with a key.
          ANTHROPIC_API_KEY: ${{ (vars.CLAUDE_ACCOUNT == 'secondary' && secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY || secrets.CLAUDE_CODE_OAUTH_TOKEN) == '' && secrets.ANTHROPIC_API_KEY || '' }}
        run: |
          [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
          $failed = @()
          foreach ($suite in ($env:TASK_SUITES -split ' ')) {
            $out = "$env:TASKS_OUT\$suite"
            if (-not (Test-Path "$out\scorecard.json")) { $failed += "$suite (no results to judge)"; continue }
            & $env:VENV_GAIA eval tasks judge $out --judge-model $env:TASK_JUDGE_MODEL --judge-attempts 2
            if ($LASTEXITCODE -ne 0) { $failed += "$suite (exit $LASTEXITCODE)" }
          }
          if ($failed.Count -gt 0) { throw "gaia eval tasks judge failed for: $($failed -join ', ')." }

      # Always enforced. With no committed expectations for the model yet it only
      # reports; the proposal in the artifact is the file to commit from main.
      - name: Gate on the committed expectations
        id: gate
        if: ${{ !cancelled() }}
        run: |
          [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
          $failed = @()
          foreach ($suite in ($env:TASK_SUITES -split ' ')) {
            $out = "$env:TASKS_OUT\$suite"
            if (-not (Test-Path "$out\scorecard.json")) { $failed += "$suite (no results - its run crashed)"; continue }
            & $env:VENV_GAIA eval tasks gate $out --enforce --propose "$out\proposed-expectations.json"
            if ($LASTEXITCODE -ne 0) { $failed += "$suite (exit $LASTEXITCODE)" }
          }
          "failed=$($failed -join ', ')" | Out-File -FilePath $env:GITHUB_OUTPUT -Append -Encoding ascii
          if ($failed.Count -gt 0) {
            Write-Host "::error::The outcomes gate failed for: $($failed -join ', '). Each suite's verdict is printed above."
            exit 1
          }

      - name: Stop the embedded Lemonade
        if: always()
        run: '& $env:VENV_GAIA lemonade embedded stop'

      - name: Keep the embedded Lemonade log
        if: always()
        run: |
          $log = "$HOME\.gaia\lemonade\lemond.log"
          if (Test-Path $log) { Copy-Item $log "$env:TASKS_OUT\lemond.log" -Force }

      # Scans EVERY file that uploads, not a hand-kept subset, so the two lists
      # cannot drift apart. The artifact is public and, unlike the job log, not
      # secret-masked.
      - name: Refuse to upload a credential
        if: always()
        env:
          CLAUDE_CODE_OAUTH_TOKEN: ${{ (vars.CLAUDE_ACCOUNT == 'secondary' && secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY || secrets.CLAUDE_CODE_OAUTH_TOKEN) }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          $ErrorActionPreference = "Stop"
          $files = @(Get-ChildItem $env:TASKS_OUT -File -Recurse -ErrorAction SilentlyContinue)
          $secrets = @()
          foreach ($name in @("CLAUDE_CODE_OAUTH_TOKEN", "ANTHROPIC_API_KEY")) {
            $value = [Environment]::GetEnvironmentVariable($name)
            if ($value) { $secrets += $value }
          }
          $leaked = @()
          foreach ($file in $files) {
            if (-not (Test-Path -LiteralPath $file.FullName)) { continue }
            $hit = $false
            foreach ($value in $secrets) {
              if (Select-String -LiteralPath $file.FullName -SimpleMatch -Pattern $value -Quiet) { $hit = $true }
            }
            if (-not $hit) { continue }
            $leaked += $file.FullName
            $text = [System.IO.File]::ReadAllText($file.FullName)
            foreach ($value in $secrets) { $text = $text.Replace($value, "***REDACTED***") }
            [System.IO.File]::WriteAllText($file.FullName, $text, (New-Object System.Text.UTF8Encoding($false)))
          }
          if ($leaked.Count -gt 0) {
            Write-Host "::error::A credential appeared in $($leaked.Count) file(s) bound for the PUBLIC artifact: $($leaked -join ', '). They were redacted in place before upload, but a secret reaching a result file at all means something is printing it - find that and fix it, then rotate the credential."
            exit 1
          }
          if ($files.Count -eq 0) {
            Write-Host "::warning::No files under $env:TASKS_OUT - nothing was scanned, and nothing will upload."
          } else {
            Write-Host "Scanned $($files.Count) file(s): no credential found."
          }

      # This machine is destroyed when the job ends, so the per-task transcripts
      # and the Lemonade log go with it unless they leave here.
      - name: Upload the outcomes results
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: eval-outcomes
          path: eval-out/tasks
          if-no-files-found: warn
          retention-days: 14

  agent-eval:
    name: Scenario eval - ${{ matrix.lane }}
    needs: [eval-gate, lanes]
    strategy:
      # One lane's failure must not cancel the others. They measure different
      # categories on different machines, and a lane that dies is exactly when
      # the other lanes' scorecards are most worth having.
      fail-fast: false
      matrix: ${{ fromJSON(needs.lanes.outputs.matrix) }}
    # Per LANE, not per workflow. CLAUDE.md's "run evals serially" rule is about
    # one Lemonade backend race-evicting its own model slot; lanes get separate
    # machines and therefore separate backends, so they are free to run at once.
    # Within a lane, categories still run one at a time.
    #
    # Must stay keyed on matrix.lane. A group shared across lanes would make the
    # second lane wait for the first and undo the split; a group shared with a
    # CALLER would deadlock outright (the phantom-job bug).
    concurrency:
      group: ${{ github.workflow }}-${{ matrix.lane }}-${{ github.ref }}
      cancel-in-progress: false
    # Fork PRs and drafts never reach the lab pool. A fork PR would schedule
    # untrusted code onto a self-hosted runner and could not pass anyway
    # (ANTHROPIC_API_KEY is not exposed to fork PRs, so the judge preflight
    # fails); a draft would spend a multi-hour single-slot run on work still in
    # progress. A job-level `if` resolves before a runner is assigned, so a
    # skipped job never reaches the pool at all. `ready_for_ci` is the repo-wide
    # escape hatch for a draft that does want CI. Mirrors the guard in
    # test_email_agent_eval.yml.
    #
    # Kept flat-indented on purpose: a YAML folded scalar preserves newlines on
    # more-indented lines, and an `if:` expression must fold to a single line.
    if: >-
      !cancelled() && needs.eval-gate.result == 'success' &&
      needs.eval-gate.outputs.run == 'true' && needs.lanes.result == 'success' &&
      needs.lanes.outputs.has_lanes == 'true' &&
      (github.event_name != 'pull_request' ||
      (github.event.pull_request.head.repo.full_name == github.repository &&
      (github.event.pull_request.draft == false ||
      contains(github.event.pull_request.labels.*.name, 'ready_for_ci'))))
    # `Windows` is load-bearing, not decoration: self-hosted pool labels here also
    # match a Linux box (xsj-aimlab-stxp-04), and every step below is PowerShell +
    # LemonadeServer.exe. Landing on Linux dies with 'powershell: command not
    # found'. Same guard as test_email_agent_eval.yml.
    #
    # `strix-halo` pins the SoC, and that is a correctness requirement, not a
    # preference: the pool spans Strix Point boxes as well (sjlab-stx-1,
    # sjlab-stx-3), which do not have the 128 GB of unified memory this eval's
    # model needs, and scores from two different SoCs are not comparable to each
    # other or to the committed baselines. Do NOT widen this to `stx` — an
    # earlier revision of this comment suggested exactly that, which would have
    # let a scorecard be produced on whichever machine happened to be free.
    #
    # `devlab-dispatch` is the ephemeral pool: every lane gets its own freshly
    # provisioned machine, which is what lets the lanes run at once — separate
    # machines means separate Lemonade backends, and the serial rule is about one
    # backend's model slot. The machine is DESTROYED when the job ends, which is
    # why the upload step at the bottom takes traces and logs and not just scores.
    #
    # `lemonade-eval` is deliberately NOT here. It is the lock label for the one
    # persistent Halo box, and holding it would put every lane back in a single
    # queue. The email and publish workflows still use it; this one must not.
    runs-on: [self-hosted, Windows, devlab-dispatch, strix-halo]
    defaults:
      run:
        # Pinned to Windows PowerShell 5.1, not left to the runner default (pwsh
        # when installed, powershell otherwise). Everything below is written to
        # 5.1's rules — the Tee-Object/`2>&1` and $LASTEXITCODE notes are
        # 5.1-specific — and this file's first run died because nothing declared
        # a shell at all and the steps were bash.
        #
        # KEEP EVERY `run:` BODY PURE ASCII. Actions writes the block to a .ps1
        # with no BOM, and PowerShell 5.1 reads a BOM-less file as ANSI, not
        # UTF-8 — so one em dash inside a `run:` string is three cp1252 bytes
        # mid-token and the whole step dies at parse time before a line executes.
        # (Run 32274575191: a single `—` in an ::error:: message produced
        # "Missing argument in parameter list" 30 lines away from itself.) YAML
        # comments and step `name:` values are fine; they never reach the file.
        shell: powershell
    # BUDGET DERIVATION — measured on this pool, not guessed. Sized from the
    # [TIMING] lines of 19 complete runs (2026-09-17 .. 09-21) plus the
    # per-scenario [DONE] timings inside 8 of them, which is what makes this a
    # per-SCENARIO rate rather than a per-category one:
    #
    #   140 healthy scenarios sampled   4.05 min mean, 6.73 max, 2.00 min
    #   per-category overhead           none measurable — a category's [TIMING]
    #                                   minus the sum of its scenarios: 0.04 min
    #                                   mean, 0.10 max across 33 category-runs
    #
    # A category therefore costs exactly what its scenarios cost, so the lane
    # budget is one multiplication. WHY SCENARIO COUNT IS THE BALANCING UNIT:
    # category means span only 3.86-4.52 min (tool_selection / rag_quality /
    # context_retention, the three CI has ever run), while scenarios INSIDE one
    # category span 2.00-6.73. Scenario identity explains about twice the
    # variance that category does, so minutes-per-category would move these lane
    # boundaries by minutes, not tens of minutes.
    #
    #   tools 18 -> 72 min    retrieval 29 -> 116    memory 25 -> 101
    #   conversation 10 -> 42                        outcomes: its own job
    #   gaia-* lanes 23-34 scenarios each (CI-runnable, after tag exclusions)
    #
    # Lanes grow as scenarios are added, so the step limit is generous rather
    # than a cap to balance against: 360 min covers ~53 scenarios at the worst
    # observed 6.73 min. It exists only so an overrun fails here with partial
    # results still collected; rebalance when a lane's [TIMING] nears it.
    #
    # The ceiling is sized for the LARGEST lane rather than per lane; a smaller
    # lane finishing early costs nothing on a machine that is discarded anyway:
    #
    #   eval step (see above)                                          -> 360
    #   setup on a COLD machine (venv, npm, Lemonade install + model)  ->  45
    #   collect + scrub + integrity + compare + upload                 ->  15
    #   job: 360+45+15                                                 -> 420
    #
    # Serial, all 82 scenarios in one job, this was 347 min expected / 552 ceiling.
    # The split trades that for four ~2 h lanes that finish together.
    #
    # Only 13 of the 82 are priced on a MEASURED category mean; the rest use the
    # global one, because CI has never run their category. Watch `memory` as real
    # numbers arrive: 25 scenarios at the highest turn count in the repo, and
    # turns are the one structural driver that correlates with duration at all.
    # Rebalance by editing eval/ci_lanes.json; the [TIMING] lines and each lane's
    # lane-timing.json exist to supply the real number.
    #
    # The eval step carries its OWN timeout-minutes below so an overrun fails at a
    # NAMED step instead of a silent job-level `cancelled` — a job-level timeout
    # skips the `!cancelled()` collect/integrity/compare steps AND the upload,
    # which on an ephemeral machine destroys the partial results permanently.
    # That is how run 30869194618 threw away 124 min of rag_quality data.
    timeout-minutes: 420
    env:
      # The venv's interpreter and CLI by full path. A bare `python` resolves
      # through PATH, which a runner's own setup can put ahead of the venv.
      VENV_PY: ${{ github.workspace }}\.venv\Scripts\python.exe
      VENV_GAIA: ${{ github.workspace }}\.venv\Scripts\gaia.exe
      # The lane's categories, from eval/ci_lanes.json via the `lanes` job. Every
      # step below reads this one list, so run / collect / gate / compare cannot
      # disagree about what this lane was supposed to measure.
      EVAL_CATEGORIES: ${{ matrix.categories }}
      EVAL_LANE: ${{ matrix.lane }}
      # Targeted dispatch only: "category=id,id category=id". Blank runs every
      # scenario in each category, which is what a tag run always does.
      EVAL_SCENARIOS: ${{ matrix.scenarios }}
      EVAL_SELECTION: ${{ needs.lanes.outputs.selection }}
      EVAL_TARGETED: ${{ needs.lanes.outputs.targeted }}
      EVAL_TIER: ${{ needs.lanes.outputs.tier }}
      # The local model under test, pinned in every category's GAIA home and
      # checked per scenario, and the baseline captured on that model. Both
      # from eval/ci_tiers.json via the lanes job, so they cannot disagree.
      AGENT_MODEL: ${{ needs.lanes.outputs.agent_model }}
      BASELINE_DIR: ${{ needs.lanes.outputs.baseline_dir }}
      # Every run, tag runs included, fails on scenarios that errored before
      # Claude spent a token (an exhausted quota, as on v0.25.1-rc2). That is a
      # run that measured nothing, not a regression, so report mode does not
      # soften it.
      FAIL_ON_ZERO_TOKENS: 'true'
      # Memory ON for the memory lane, OFF everywhere else. Job scope beats the
      # workflow defaults, and everything that reads these is started from here:
      # the backend inherits the job environment through Start-Process, and the
      # MCP server inherits it from the backend. The values are decided in the
      # `lanes` job and arrive as literal strings - see the comment there for why
      # they are not computed in YAML.
      GAIA_MEMORY_DISABLED: ${{ matrix.memory_disabled }}
      GAIA_MEMORY_ADMIN: ${{ matrix.memory_admin }}
      GAIA_MEMORY_MCP_ALWAYS: ${{ matrix.memory_mcp }}
      GAIA_FIXTURES: ${{ matrix.gaia_fixtures }}
      # A tier machine can hold `memory` beside other non-gaia categories. The
      # job runs with memory off, and the run step turns the store on for the
      # categories named here only, each on its own backend and GAIA home.
      EVAL_MEMORY_CATEGORIES: ${{ matrix.memory_categories }}
    steps:
      - name: Show which Claude account this job uses
        run: echo "::notice title=Claude account::${{ vars.CLAUDE_ACCOUNT != 'secondary' && 'This job uses the primary Claude account.' || secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY != '' && 'This job uses the secondary Claude account.' || 'CLAUDE_ACCOUNT is secondary but CLAUDE_CODE_OAUTH_TOKEN_SECONDARY is not set, so this job uses the primary Claude account.' }}"
      - name: Checkout
        uses: actions/checkout@v7

      # The Claude model drives every scenario as well as judging it, so it must
      # be the one the model's baseline was captured under: its meta.json is the
      # single source. A dispatch input overrides it.
      - name: Resolve the eval driver model
        run: |
          $model = $env:EVAL_MODEL_OVERRIDE
          if (-not $model) {
            $meta = "$env:BASELINE_DIR/meta.json"
            $model = (Get-Content -Raw $meta | ConvertFrom-Json).eval_model
            if (-not $model) {
              Write-Host "::error title=No eval model::$meta has no eval_model, so this run cannot know which Claude model the baseline was captured under. | WHAT TO DO: restore eval_model in $meta, or dispatch with the eval_model input."
              exit 1
            }
          }
          "EVAL_MODEL=$model" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding ascii
          Write-Host "Eval driver and judge model: $model"

      # A memory lane with memory switched off does not fail - it scores badly,
      # and the scorecard reads as a quality regression rather than a setup
      # fault. So re-derive what this lane needs from its own category list and
      # refuse to start if the environment disagrees, before anything has run.
      # Independent of the matrix wiring on purpose: the point is to catch that
      # wiring going wrong, so it cannot be checked against itself.
      - name: Check the lane environment matches its categories
        run: |
          $ErrorActionPreference = "Stop"
          $cats = @($env:EVAL_CATEGORIES -split '\s+' | Where-Object { $_ })
          $needsMemory = $cats -contains 'memory'
          $gaiaLane = @($cats | Where-Object { $_ -like 'gaia_*' }).Count -gt 0
          # A tier machine with memory beside other non-gaia categories runs
          # with memory off and turns it on for `memory` alone (run step).
          $mixedMemory = $needsMemory -and $cats.Count -gt 1 -and -not $gaiaLane
          if ($gaiaLane) {
            $want = [ordered]@{ GAIA_MEMORY_DISABLED = '0'; GAIA_MEMORY_ADMIN = '1'; GAIA_MEMORY_MCP_ALWAYS = '0'; GAIA_FIXTURES = '1'; EVAL_MEMORY_CATEGORIES = '' }
            $why = "gaia_* categories must run the way test_gaia_agent_eval.yml runs them and their baselines were captured: memory live with admin seeding, fixtures staged."
          } elseif ($mixedMemory) {
            $want = [ordered]@{ GAIA_MEMORY_DISABLED = '1'; GAIA_MEMORY_ADMIN = '0'; GAIA_MEMORY_MCP_ALWAYS = '0'; GAIA_FIXTURES = '0'; EVAL_MEMORY_CATEGORIES = 'memory' }
            $why = "This machine runs memory beside other categories: the job runs with the store off and the run step turns it on for the memory category alone, so EVAL_MEMORY_CATEGORIES must name exactly memory."
          } elseif ($needsMemory) {
            $want = [ordered]@{ GAIA_MEMORY_DISABLED = '0'; GAIA_MEMORY_ADMIN = '1'; GAIA_MEMORY_MCP_ALWAYS = '1'; GAIA_FIXTURES = '0'; EVAL_MEMORY_CATEGORIES = '' }
            $why = "The memory scenarios call remember/recall and verify through memory_recall, so this lane must run with the store live and its admin and MCP read tools registered."
          } else {
            $want = [ordered]@{ GAIA_MEMORY_DISABLED = '1'; GAIA_MEMORY_ADMIN = '0'; GAIA_MEMORY_MCP_ALWAYS = '0'; GAIA_FIXTURES = '0'; EVAL_MEMORY_CATEGORIES = '' }
            $why = "No category in this lane touches long-term memory, and leaving the store live lets state from one scenario change the next one's score."
          }
          $wrong = @()
          foreach ($name in $want.Keys) {
            $have = [Environment]::GetEnvironmentVariable($name)
            # Windows drops an env var set to '', so unset IS the empty value.
            if ($null -eq $have) { $have = if ($want[$name] -eq '') { '' } else { '<unset>' } }
            if ($have -ne $want[$name]) { $wrong += "$name is '$have' but should be '$($want[$name])'" }
          }
          if ($wrong.Count -gt 0) {
            Write-Host "::error::Lane '$env:EVAL_LANE' runs [$env:EVAL_CATEGORIES] but its environment is wrong: $($wrong -join '; '). $why The per-lane values come from util/eval_lane_matrix.py via the lanes job - fix them there, not in the workflow-level defaults, which would change every lane."
            exit 1
          }
          Write-Host "Lane '$env:EVAL_LANE' environment is correct (memory lane: $needsMemory, gaia_* lane: $gaiaLane)."

      - name: Setup Python Environment
        id: venv
        uses: ./.github/actions/setup-venv
        with:
          python-version: '3.12'
          # [dev] + [eval] for the eval harness and the Anthropic judge deps;
          # [ui] for fastapi/uvicorn + the RAG deps gaia.ui.server imports at boot
          # ([dev] alone fails at import), [api] for the connectors layer.
          # Both hub packages are REQUIRED, and neither is pulled by those extras.
          # Every scenario runs the flagship `gaia` agent (DEFAULT_AGENT_TYPE in
          # src/gaia/eval/config.py), which is not a builtin — it registers only
          # through the `gaia.agent` entry point in gaia-agent-gaia, and GaiaAgent
          # subclasses ChatAgent from gaia-agent-chat (#1102). Miss either and
          # POST /api/sessions 422s for every scenario in this lane. The "default
          # agent is registered" preflight below asserts this at runtime so the
          # gap costs seconds, not a whole lane. Core gaia stays editable so eval
          # fixtures resolve in-repo.
          # `mcp` is not optional for this gate: every scenario is driven through
          # the Agent UI MCP server, and without the extra that server exits with
          # ModuleNotFoundError before writing a protocol frame — which the client
          # reports only as CONNECTION_CLOSED.
          install-package: '-e .[dev,eval,ui,api,mcp] -e hub/agents/chat/python -e hub/agents/gaia/python'

      - name: Check the venv interpreter
        env:
          SETUP_VENV_PY: ${{ steps.venv.outputs.python-path }}
        run: |
          if ($env:SETUP_VENV_PY -ne $env:VENV_PY) {
            Write-Host "::error::setup-venv built '$env:SETUP_VENV_PY' but this job runs '$env:VENV_PY'. Point VENV_PY/VENV_GAIA in the job env at setup-venv's venv-name."
            exit 1
          }
          foreach ($exe in @($env:VENV_PY, $env:VENV_GAIA)) {
            if (-not (Test-Path $exe)) {
              Write-Host "::error::$exe is missing - the venv install did not complete. See the Setup Python Environment step."
              exit 1
            }
          }
          & $env:VENV_PY -c "import gaia, gaia_agent, gaia_agent_chat"
          if ($LASTEXITCODE -ne 0) {
            Write-Host "::error::$env:VENV_PY cannot import gaia / gaia_agent / gaia_agent_chat - the editable installs in the Setup Python Environment step are broken."
            exit 1
          }
          $bare = (Get-Command python -ErrorAction SilentlyContinue).Source
          Write-Host "Venv interpreter: $env:VENV_PY"
          Write-Host "Bare 'python' on this runner: $(if ($bare) { $bare } else { 'none' })"

      # `gaia eval agent` does not merely call the Anthropic API - it shells out
      # to `claude -p` with an MCP config to DRIVE each scenario (runner.py:937),
      # and its own preflight hard-fails with "'claude' CLI not found on PATH"
      # before running anything. ANTHROPIC_API_KEY is necessary but not
      # sufficient; the CLI has to be installed too. Same prerequisite the
      # eval-CI recipe documents (docs/guides/eval-ci.mdx).
      - name: Set up Node
        uses: actions/setup-node@v7
        with:
          node-version: '22'

      - name: Install Claude Code CLI (the eval's scenario driver)
        run: |
          npm install -g @anthropic-ai/claude-code
          if ($LASTEXITCODE -ne 0) { throw "Claude Code CLI install failed" }
          # Fail here, not after the backend has booted.
          claude --version
          if ($LASTEXITCODE -ne 0) { throw "claude installed but not runnable on PATH" }

      - name: Install Lemonade Server
        uses: ./.github/actions/install-lemonade

      - name: Start Lemonade Server (version-matched, warmed)
        # 15 min for the server and Gemma, plus up to 30 min to pull a 23 GB
        # Qwen3.6 onto a fresh machine.
        timeout-minutes: 50
        env:
          HUGGINGFACE_ACCESS_TOKEN: ${{ secrets.HUGGINGFACE_ACCESS_TOKEN }}
          HF_TOKEN: ${{ secrets.HUGGINGFACE_ACCESS_TOKEN }}
        run: |
          # Launches the version-matched server as a persistent Scheduled Task
          # (survives across CI jobs), reuses it if healthy, and warms
          # Gemma-4-E4B, its default. Gemma is pulled whatever the model under
          # test: vision loads it by name, and every PC `gaia init` sets up has
          # it, Qwen machines included.
          powershell -ExecutionPolicy Bypass -File installer\scripts\ensure-lemonade-running.ps1
          if ($LASTEXITCODE -ne 0) { throw "Lemonade server not ready" }
          # Then the model under test, pulled here rather than through a second
          # script call: the script returns early on a healthy server without
          # warming anything, and swallows a failed pull.
          if ($env:AGENT_MODEL -ne 'Gemma-4-E4B-it-GGUF') {
            $sw = [Diagnostics.Stopwatch]::StartNew()
            $base = $env:LEMONADE_BASE_URL.TrimEnd('/') -replace '/api/v1$', ''
            $body = @{ model_name = $env:AGENT_MODEL } | ConvertTo-Json
            try {
              Invoke-RestMethod "$base/api/v1/pull" -Method POST -ContentType "application/json" -Body $body -TimeoutSec 1800 | Out-Null
            } catch {
              throw "Could not pull $env:AGENT_MODEL into Lemonade at $base : $($_.Exception.Message). Check the runner's network and disk, then re-run."
            }
            Write-Host "Pulled $env:AGENT_MODEL in $([math]::Round($sw.Elapsed.TotalMinutes, 1)) min."
          }

      - name: Preflight — judge key, baselines, Lemonade
        env:
          CLAUDE_CODE_OAUTH_TOKEN: ${{ (vars.CLAUDE_ACCOUNT == 'secondary' && secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY || secrets.CLAUDE_CODE_OAUTH_TOKEN) }}
          # Blank when the OAuth token is present, deliberately: runner.py adds
          # `--bare` only when ANTHROPIC_API_KEY is set, and `--bare` restricts
          # auth to the key alone, so leaving both set would ignore the token.
          ANTHROPIC_API_KEY: ${{ (vars.CLAUDE_ACCOUNT == 'secondary' && secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY || secrets.CLAUDE_CODE_OAUTH_TOKEN) == '' && secrets.ANTHROPIC_API_KEY || '' }}
        run: |
          $ErrorActionPreference = "Stop"
          $ProgressPreference = "SilentlyContinue"

          # One probe, so each rung of the ladder below runs an identical check.
          # $LlamacppArgs empty = load the embedder bare, which is ONLY ever used
          # as the discriminator on the last rung -- never to take the job green,
          # because production RAG always passes --ubatch-size 2048
          # (gaia/rag/sdk.py::_load_embedder) and measuring a different load than
          # production performs is exactly the "we did not measure" case the
          # integrity gate exists to refuse.
          function Invoke-EmbedProbe {
            param([string]$LlamacppArgs)
            $ErrorActionPreference = "Continue"
            $global:LASTEXITCODE = 0
            if ($LlamacppArgs) {
              $out = & $env:VENV_PY tests/ci_lemonade_check.py `
                --model $embeddingModel --checkpoint $embeddingCheckpoint `
                --recipe $embeddingRecipe --register-embedding --embeddings `
                --llamacpp-args $LlamacppArgs 2>&1
            } else {
              $out = & $env:VENV_PY tests/ci_lemonade_check.py `
                --model $embeddingModel --checkpoint $embeddingCheckpoint `
                --recipe $embeddingRecipe --register-embedding --embeddings 2>&1
            }
            $code = $LASTEXITCODE
            $ErrorActionPreference = "Stop"
            $out | ForEach-Object { Write-Host "$_" }
            # Assert the EMBEDDINGS line, not the helper's terminal `[ci] OK` -
            # that one prints whether or not --embeddings was passed. `dim=` only
            # prints after a real vector came back. Exit 0 alone is not enough
            # either: a command that fails to LAUNCH leaves $LASTEXITCODE at the
            # reset 0 and would read as a pass.
            return ($code -eq 0 -and [bool]($out -match '\[ci\] embeddings OK \(dim='))
          }

          # Lemonade runs as a SYSTEM Scheduled Task, whose action has no console
          # and whose stdout/stderr the scheduler records nowhere -- so until
          # ensure-lemonade-running.ps1 began redirecting them, llama-server's
          # startup error was discarded by the OS, not merely hard to locate.
          # Every #2920 failure reported "(none found)" from a SYSTEM-profile glob
          # that never had anything to match. Read the redirect target instead.
          function Show-LemonadeTaskLog {
            $dir = "C:\ProgramData\GaiaLemonadeServer"
            Write-Host "=== Lemonade task stdout/stderr (tail) ==="
            $found = $false
            foreach ($f in @("$dir\lemonade-task-stderr.log", "$dir\lemonade-task-stdout.log")) {
              if (Test-Path $f) {
                # Report the size: "file exists but is empty" and "file is absent"
                # are different faults (buffering vs. no redirect) and printing
                # nothing for both makes them indistinguishable.
                $len = (Get-Item $f).Length
                Write-Host "--- $f ($len bytes) ---"
                if ($len -gt 0) {
                  Get-Content $f -Tail 120 -ErrorAction SilentlyContinue | ForEach-Object { Write-Host "$_" }
                  $found = $true
                } else {
                  Write-Host "(empty - server captured nothing on this stream)"
                }
              }
            }
            if (-not $found) {
              Write-Host "(none found in $dir - this runner's Scheduled Task predates the redirect; the -ForceRestart below re-registers it with logging)"
            }
            return $found
          }

          # Lemonade's POST /delete deregisters a model but never removes its
          # files, so the probe's delete + re-pull cannot re-fetch anything: it
          # short-circuits on the blob cache and reloads the same bytes every
          # time. Freeing them needs an OS-level delete, which is this function.
          # Roots follow _huggingface_cache_dir / _lemonade_models_dir in
          # src/gaia/installer/uninstall_command.py; report what was removed, since
          # the sizes are the evidence that a re-pull was ever real.
          function Clear-EmbedderFromDisk {
            $sys = "$env:SystemRoot\System32\config\systemprofile"
            $roots = @(
              "$sys\AppData\Local\huggingface\hub", "$sys\.cache\huggingface\hub",
              "$sys\AppData\Local\lemonade\models",  "$sys\.cache\lemonade\models",
              "$env:LOCALAPPDATA\huggingface\hub",   "$env:USERPROFILE\.cache\huggingface\hub",
              "$env:LOCALAPPDATA\lemonade\models",   "$env:USERPROFILE\.cache\lemonade\models"
            )
            if ($env:HF_HOME) { $roots += (Join-Path $env:HF_HOME "hub") }
            $removed = 0
            foreach ($root in ($roots | Where-Object { $_ } | Select-Object -Unique)) {
              if (-not (Test-Path $root)) { continue }
              Get-ChildItem $root -Filter "*embeddinggemma*" -Directory -ErrorAction SilentlyContinue |
                ForEach-Object {
                  $mb = 0
                  try {
                    $mb = [math]::Round(((Get-ChildItem $_.FullName -Recurse -File -ErrorAction SilentlyContinue |
                          Measure-Object Length -Sum).Sum / 1MB), 1)
                  } catch { }
                  Write-Host "Removing on-disk embedder cache: $($_.FullName) ($mb MB)"
                  Remove-Item $_.FullName -Recurse -Force -ErrorAction SilentlyContinue
                  $removed++
                }
            }
            if ($removed -eq 0) {
              # Do NOT read this as "the delete worked" - it never removes files.
              # Nothing found almost certainly means the cache lives somewhere the
              # roots above do not cover, so the retry below is still loading the
              # same bytes and this probe proved nothing.
              Write-Host "PURGE: found nothing to remove. Lemonade's delete does not remove files, so this most likely means the model cache is NOT under any of these roots, not that the disk was clean: $($roots -join '; ')"
            } else {
              Write-Host "PURGE: removed $removed dir(s). Lemonade's delete only deregisters, so this is the first time the bytes are actually gone and the next pull genuinely re-fetches."
            }
            return $removed
          }


          if (-not $env:CLAUDE_CODE_OAUTH_TOKEN -and -not $env:ANTHROPIC_API_KEY) {
            Write-Host "::error::Neither CLAUDE_CODE_OAUTH_TOKEN nor ANTHROPIC_API_KEY is set. `gaia eval agent` drives AND scores each scenario through ``claude -p`` (src/gaia/eval/runner.py), so without a credential every scenario errors in a few seconds and the scorecard is meaningless. Add either repository secret, or run the eval locally on AMD hardware."
            exit 1
          }

          # Assert the credential is ACCEPTED, not merely present. #3341 sat red
          # for weeks because the key existed and the account behind it was out
          # of credit: the preflight passed and all five scenarios then died in
          # three seconds each with their stdout discarded.
          # A rejected probe has two causes needing two different people: the org is
          # out of budget (billing admin) or the credential is bad (secrets change).
          # The exit code cannot tell them apart, so classify the response body and
          # name the right owner - see util/classify_claude_probe.py (#4080).
          $global:LASTEXITCODE = 0
          $probe = (claude -p --model $env:EVAL_MODEL "reply with: ok" 2>&1 | Out-String)
          $probeExit = $LASTEXITCODE
          $probeFile = Join-Path $env:RUNNER_TEMP "claude_probe.txt"
          Set-Content -Path $probeFile -Value $probe -Encoding UTF8
          & $env:VENV_PY util/classify_claude_probe.py --exit-code $probeExit --text-file $probeFile
          if ($LASTEXITCODE -ne 0) { exit 1 }

          # No baseline is a legitimate state right now, and it is reported, never
          # assumed away. Enforcing against a baseline that does not exist is the
          # pass-by-default trap this gate exists to prevent, so enforce=true with
          # no baseline is a hard error rather than a silent green.
          $categories = ($env:EVAL_CATEGORIES -split '\s+') | Where-Object { $_ }
          if (-not (Test-Path $env:BASELINE_DIR)) {
            if ($env:ENFORCE -eq 'true') {
              Write-Host "::error::enforce=true but no baseline exists at $env:BASELINE_DIR, so there is nothing to enforce against and every category would pass by default. Land a flagship baseline first (run the eval on this hardware and commit the scorecards as $env:BASELINE_DIR\scorecard_<category>.json), or re-dispatch with enforce: false."
              exit 1
            }
            Write-Host "::warning::NO BASELINE COMMITTED. Regression detection is OFF for this run: the eval still measures every category and uploads its scorecards, but nothing compares them to a known-good run. Restore the flagship baseline in $env:BASELINE_DIR."
            "REGRESSION DETECTION OFF: no baseline at ``$env:BASELINE_DIR``. Scorecards are produced and uploaded; nothing is compared." | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
          } else {
            # The baseline must have been captured on the model under test.
            # eval/ci_tiers.json maps each model to its directory and a unit test
            # checks the meta, so this only fires if the two drift at runtime.
            $metaModel = (Get-Content -Raw (Join-Path $env:BASELINE_DIR 'meta.json') | ConvertFrom-Json).model
            if ($metaModel -ne $env:AGENT_MODEL) {
              Write-Host "::error::$env:BASELINE_DIR was captured on '$metaModel', but this run scores '$env:AGENT_MODEL'. Comparing them would report the model swap as regressions. Fix the baselines map in eval/ci_tiers.json."
              exit 1
            }
            $noScorecards = @(Get-ChildItem $env:BASELINE_DIR -Filter 'scorecard_*.json').Count -eq 0
            if ($noScorecards) {
              Write-Host "::warning::NO $env:AGENT_MODEL BASELINE YET: $env:BASELINE_DIR holds no scorecards, so nothing in this run is compared. Every category still runs and uploads its scorecard. Capture the baseline from a full sweep ($env:BASELINE_DIR/README.md)."
              "**No $env:AGENT_MODEL baseline yet** (``$env:BASELINE_DIR``): this run measured quality and compared it to nothing." | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
            }
            foreach ($category in $categories) {
              $baseline = Join-Path $env:BASELINE_DIR "scorecard_$category.json"
              if (-not (Test-Path $baseline)) {
                # A category added since the baseline was captured has none until
                # its first tag run; that is reported, and never enforced against.
                if ($env:ENFORCE -eq 'true') {
                  Write-Host "::error::enforce=true but there is no $env:AGENT_MODEL baseline at $baseline, so '$category' would pass by default. Commit its scorecard from a run on this hardware first, or re-dispatch with enforce: false."
                  exit 1
                }
                if (-not $noScorecards) {
                  Write-Host "::warning::${category}: NOT COMPARED - no $env:AGENT_MODEL baseline at $baseline yet. The category still runs and its scorecard is uploaded; commit that scorecard as the baseline."
                  "``$category``: not compared, no $env:AGENT_MODEL baseline yet. Its scorecard from this run is the baseline candidate." | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
                }
                continue
              }
              Write-Host "baseline ok: $baseline"
            }
          }

          # Resolve the URL and the model under test through the client's own
          # constants rather than hardcoding them here: the default port already
          # moved once (8000 -> 13305 in Lemonade v10.1.0) and a stale literal
          # would fail a healthy runner.
          # The embedder's checkpoint/recipe come from the same MODELS entry
          # gaia.rag.sdk registers it from, so the probe below cannot drift from
          # what RAG will actually ask Lemonade for.
          $global:LASTEXITCODE = 0
          $probe = @(& $env:VENV_PY -c "from gaia.llm.lemonade_client import DEFAULT_EMBEDDING_MODEL, DEFAULT_MODEL_NAME, MODELS, _get_lemonade_config; m = next(x for x in MODELS.values() if x.model_id == DEFAULT_EMBEDDING_MODEL); print(_get_lemonade_config()[2]); print(DEFAULT_MODEL_NAME); print(DEFAULT_EMBEDDING_MODEL); print(m.checkpoint); print(m.recipe)")
          if ($LASTEXITCODE -ne 0 -or $probe.Count -lt 5) {
            Write-Host "::error::Could not resolve the Lemonade URL / models under test - the venv install is broken."
            exit 1
          }
          $lemonadeUrl = $probe[-5]
          # Not DEFAULT_MODEL_NAME ($probe[-4]): the model under test is pinned
          # per run (AGENT_MODEL), and on this pool it is not the code default.
          $modelUnderTest = $env:AGENT_MODEL
          $embeddingModel = $probe[-3]
          $embeddingCheckpoint = $probe[-2]
          $embeddingRecipe = $probe[-1]

          try {
            Invoke-RestMethod -Uri "$lemonadeUrl/health" -TimeoutSec 10 | Out-Null
          } catch {
            Write-Host "::error::Lemonade Server not reachable at $lemonadeUrl. The previous step should have started it; check its log, start it on the runner (installer\scripts\ensure-lemonade-running.ps1), or set LEMONADE_BASE_URL to a reachable instance."
            exit 1
          }
          Write-Host "Lemonade reachable at $lemonadeUrl"

          # Record WHICH machine produced this scorecard. The runs-on labels state
          # the intent; this is the only thing that proves it held at runtime, and
          # a score is not interpretable without it - the pool spans two SoCs and
          # only Strix Halo has the memory this eval assumes. /v1/system-info also
          # carries per-recipe backend state (unsupported / installable /
          # update_required / installed) and compatible devices, which is the
          # diff that explains why this embedder loads on a Strix Point box and
          # not here (#3016). Best-effort and never fatal: it is evidence, not a
          # gate, and must not be able to fail a run on its own.
          $ErrorActionPreference = "Continue"
          Write-Host "=== Lemonade system-info (device + llamacpp backend state) ==="
          & $env:VENV_PY -c "import json; from gaia.llm.lemonade_client import LemonadeClient; si = LemonadeClient().get_system_info(); print(json.dumps({k: v for k, v in si.items() if k != 'recipes'}, indent=2, default=str)[:2000]); print('--- recipes.llamacpp ---'); print(json.dumps(si.get('recipes', {}).get('llamacpp'), indent=2, default=str)[:2000])"
          if ($LASTEXITCODE -ne 0) { Write-Host "(system-info unavailable on this Lemonade build - continuing, this is diagnostic only)" }
          $global:LASTEXITCODE = 0
          $ErrorActionPreference = "Stop"

          # ensure-lemonade-running.ps1 SWALLOWS a failed warm pull ("WARN: warm
          # pull failed") and still exits 0, so a healthy server proves nothing
          # about the model this job exists to measure. `/models` (no show_all)
          # lists DOWNLOADED models only, so this is the assertion that the model
          # is actually on disk. Without it a run can score every scenario against
          # whatever was resident and compare it to the wrong baseline.
          try {
            $downloaded = @((Invoke-RestMethod -Uri "$lemonadeUrl/models" -TimeoutSec 60).data | ForEach-Object { $_.id })
          } catch {
            Write-Host "::error::Could not list downloaded models at $lemonadeUrl/models. Lemonade answered /health but not /models - check the server log on the runner."
            exit 1
          }
          if ($downloaded -notcontains $modelUnderTest) {
            Write-Host "::error::$modelUnderTest is not downloaded on this runner (Lemonade has: $($downloaded -join ', ')). The warm-pull step failed silently. Pull it on the runner (POST $lemonadeUrl/pull with model_name=$modelUnderTest) and re-run; do NOT let this eval score a different model against the $modelUnderTest baseline."
            exit 1
          }
          Write-Host "Model under test present: $modelUnderTest"

          # ...but "present" only means the GGUF is on disk. Nothing in this job
          # has ever LOADED it: ensure-lemonade-running.ps1 warms via /pull, which
          # is a download, and the check above reads /models' `downloaded` flag. So
          # a runner where llama-server cannot start AT ALL looks identical here to
          # a healthy one, and the first thing to actually load a model is the
          # embedder probe below -- which is why an embedder-specific fault and a
          # dead llama-server have been indistinguishable for this gate's whole
          # history (#2920). Load the CHAT model first: it shares the llama-server
          # binary, so if it also fails, the embedder is not the variable.
          $ErrorActionPreference = "Continue"
          $global:LASTEXITCODE = 0
          $chatOut = & $env:VENV_PY tests/ci_lemonade_check.py --model $modelUnderTest --chat 2>&1
          $chatCode = $LASTEXITCODE
          $ErrorActionPreference = "Stop"
          $chatOut | ForEach-Object { Write-Host "$_" }
          if ($chatCode -ne 0 -or -not ($chatOut -match '\[ci\] chat OK')) {
            Write-Host "::error::llama-server cannot start for the CHAT model $modelUnderTest on this runner either, so this is not an embedder fault - no model can be served here and every category would fail. The GGUF is downloaded and Lemonade answers /health, so the server process is up and the binary underneath it is what will not spawn. Check whether an Application Control policy is blocking llama-server.exe on this box (#2826), and read the Lemonade task log below."
            Show-LemonadeTaskLog | Out-Null
            exit 1
          }
          Write-Host "Chat model loads and serves - llama-server works on this runner, so an embedder failure below is embedder-specific."

          # THE RAG CATEGORIES RUN ON A SECOND MODEL. rag_quality and
          # context_retention answer entirely out of chunks the EMBEDDER
          # produced, so an embedder that cannot start turns every doc-grounded
          # scenario into a ~2/10 non-answer that reads exactly like a model
          # regression. ensure-lemonade-running.ps1 warms only the chat model
          # and nothing asserted the embedder, so run 31051933420 spent 3h31m
          # proving it: llama-server returned model_load_error on every index,
          # RAG got 0 chunks, and rag_quality scored 3.1 against a 9.4 baseline
          # captured with the SAME chat model. Probe it here, in seconds.
          #
          # Probed AFTER the chat model is warm on purpose: the embedder is
          # co-resident with it (#1544), so probing an idle server would not
          # reproduce the condition the eval actually runs under.
          #
          # Presence in /models is not enough - it proves the GGUF is on disk,
          # not that llama-server can serve it, and "on disk but unservable" is
          # the exact failure above. tests/ci_lemonade_check.py is the shared CI
          # helper for precisely this (test_embeddings.yml, test_rag.yml,
          # test_api.yml all use it, and it is green on the sibling stx runners),
          # so this reuses it rather than hand-rolling a weaker probe:
          #
          #   --register-embedding  DELETES any prior registration and re-registers
          #     with the embeddings label before pulling. That is the one failure
          #     in this class CI can actually REPAIR: a stale registration left by
          #     an earlier job (wrong recipe, missing label - #1745) makes every
          #     subsequent load fail forever, and ensure_model_downloaded alone
          #     short-circuits on "already downloaded" and never re-applies it.
          #   --llamacpp-args       loads with the SAME flag gaia.rag.sdk passes
          #     (_load_embedder: --ubatch-size 2048). Without it the probe proves
          #     the embedder can be served, not that RAG's load of it succeeds -
          #     which is the difference between a green preflight and a green run.
          #   the client's load retry absorbs the known transient AMD-Vulkan
          #     "llama-server failed to start" fault instead of tripping on it.
          #
          # Continue, not Stop: under Stop the helper's first stderr line becomes
          # a terminating NativeCommandError and the step dies before the
          # ::error:: below. Same reasoning as the eval step's own note.
          # RUNG 1 - the load production performs.
          $embedOk = Invoke-EmbedProbe -LlamacppArgs "--ubatch-size 2048"

          # RUNG 2 - a stale llama.cpp backend cannot spawn and reports exactly
          # this "llama-server failed to start".
          if (-not $embedOk) {
            Show-LemonadeTaskLog | Out-Null

            # A stale llama.cpp backend in Lemonade's bin cache cannot spawn, and
            # reports exactly this "llama-server failed to start". install-lemonade
            # already wipes that cache - but only when it reconciles, so a runner
            # whose VERSION is correct never clears it and can never self-heal
            # (#2920). Do the same wipe here, restart, and re-probe once.
            Write-Host "=== Probe failed; clearing the llama.cpp backend cache and retrying once ==="
            $cacheRoots = @(
              "$env:SystemRoot\System32\config\systemprofile\.cache\lemonade",
              "$env:USERPROFILE\.cache\lemonade"
            )
            if ($env:LEMONADE_CACHE_DIR) { $cacheRoots += $env:LEMONADE_CACHE_DIR }
            $seenRoots = @{}
            foreach ($root in ($cacheRoots | Where-Object { $_ })) {
              $rootKey = $root.ToLowerInvariant()
              if ($seenRoots.ContainsKey($rootKey)) { continue }
              $seenRoots[$rootKey] = $true
              $backendDir = Join-Path $root "bin"
              if (-not (Test-Path $backendDir)) { continue }
              Write-Host "Clearing stale llama.cpp backend cache at $backendDir"
              # Cached ROCm Composable-Kernel headers blow past MAX_PATH, so mirror
              # an empty dir over the tree with robocopy (long-path-safe) first.
              $emptyDir = Join-Path $env:RUNNER_TEMP "lemonade-wipe-empty"
              if (-not (Test-Path $emptyDir)) { New-Item -ItemType Directory -Path $emptyDir -Force | Out-Null }
              robocopy $emptyDir $backendDir /MIR /NFL /NDL /NJH /NJS /NP | Out-Null
              # robocopy: <8 is success. Fail loud - a swallowed wipe leaves the
              # stale backend in place and the retry below repeats the failure.
              if ($LASTEXITCODE -ge 8) { throw "robocopy mirror-wipe of $backendDir failed with exit code $LASTEXITCODE" }
              Remove-Item $backendDir -Recurse -Force -ErrorAction Stop
            }
            $global:LASTEXITCODE = 0

            Write-Host "Restarting Lemonade so it refetches a matching backend..."
            powershell -ExecutionPolicy Bypass -File installer\scripts\ensure-lemonade-running.ps1 -ForceRestart
            $global:LASTEXITCODE = 0

            $embedOk = Invoke-EmbedProbe -LlamacppArgs "--ubatch-size 2048"
            if ($embedOk) {
              Write-Host "Embedder recovered after the backend-cache wipe - the cached llama.cpp backend was stale."
            }
          }

          # STOP HERE ON A NORMAL RUN. Everything above is cheap and read-only:
          # it establishes whether llama-server works at all on this box and
          # whether a stale backend was the problem. Everything below purges the
          # model from disk and unloads every resident model, which was the right
          # thing to do while the cause was unknown and is the wrong thing to
          # repeat on every PR now that it is known (#3016) - it adds ~7 min and
          # leaves the next job on this shared box paying for a cold reload.
          # Re-dispatch with deep_diagnostics: true to run the full ladder.
          # ...and do NOT kill the job here. Exiting left PRs with no regression
          # signal whatsoever; record the state, let the run measure what it can,
          # and let the integrity gate render the honest verdict.
          #
          # How much tool_selection survives a dead embedder is LESS than an
          # earlier revision of this comment claimed. It asserted the category
          # "carries no corpus at all"; three of its five scenarios
          # (data_vs_recall_disambiguation, known_path_read, multi_step_plan) index
          # corpus documents in their `setup:`, so they need the embedder too. Only
          # no_tools_needed and smart_discovery are genuinely corpus-free. Running
          # the category anyway is still worth it - two measured scenarios beat
          # zero - but do not read a green tool_selection here as full coverage;
          # the integrity gate names exactly what was and was not measured.
          #
          # `exit 0` is what makes both of those true, and it is load-bearing:
          # without it this block only PRINTED the promise. Execution fell through
          # into the rungs below and ended at their terminal `exit 1`, which failed
          # the step, which skipped the eval step entirely - so tool_selection was
          # never measured either and the gate blocked PRs while reporting nothing
          # (run 33286140041: 12 min of preflight, every category skipped).
          #
          # The annotation level tracks `enforce` for the same reason the integrity
          # gate's does: #3016 is an environment defect, so on a report-mode PR run
          # a red ::error:: on a green job is noise that trains people to ignore the
          # check. The text is identical either way - only its severity moves.
          if (-not $embedOk -and $env:DEEP_DIAG -ne 'true') {
            Show-LemonadeTaskLog | Out-Null
            "EMBEDDER_OK=false" | Out-File -FilePath $env:GITHUB_ENV -Append -Encoding utf8
            $level = if ($env:ENFORCE -eq 'true') { 'error' } else { 'warning' }
            Write-Host "::${level}::The RAG embedder $embeddingModel will not load on this runner - a known Strix Halo limitation tracked in #3016, NOT a fault in the code under test. The chat model loaded and served over the same llama-server binary earlier in this step, and the same embedder loads fine on the Strix Point boxes, so llama-server works here and the gap is specific to the embeddings path on this SoC. Re-running this gate will not clear it. rag_quality and context_retention cannot be measured at all; tool_selection WILL still run, but only its two corpus-free scenarios (no_tools_needed, smart_discovery) can produce a measurement - the other three index documents and need the same embedder. The deeper rungs (purge the model from disk, load without --ubatch-size 2048, load with everything else unloaded) are skipped by default on purpose because they disturb other jobs on this shared box - re-dispatch with deep_diagnostics: true if you need the full elimination ladder."
            exit 0
          }

          # RUNG 3 - purge the model's BYTES, not just its registration, then
          # retry the production load once. `--register-embedding` already deletes
          # and re-pulls before every attempt above, so a third identical retry
          # would prove nothing; what has never been tried is removing the files
          # underneath, because Lemonade's delete may only drop the registry entry
          # and leave the HF blob cache intact (`gaia uninstall --purge-hf-cache`
          # exists for exactly that). If a corrupt or truncated GGUF is the cause,
          # this is the only rung that can clear it - and it takes the job green,
          # which the diagnostic rung below deliberately never does.
          if (-not $embedOk) {
            Write-Host "=== Backend-cache wipe did not help; purging the embedder from disk and retrying once ==="
            Clear-EmbedderFromDisk | Out-Null
            Write-Host "Restarting Lemonade before the clean-disk retry..."
            powershell -ExecutionPolicy Bypass -File installer\scripts\ensure-lemonade-running.ps1 -ForceRestart
            $global:LASTEXITCODE = 0
            $embedOk = Invoke-EmbedProbe -LlamacppArgs "--ubatch-size 2048"
            if ($embedOk) {
              Write-Host "Embedder recovered after a full on-disk purge - the cached GGUF was bad. Continuing."
            }
          }

          # RUNG 4 - DIAGNOSTIC ONLY, never a pass. Everything shared with the two
          # healthy sibling jobs (test_embeddings.yml, test_rag.yml - same model,
          # same helper, green on their runner) has now been ruled out, and the
          # remaining differences are this job's --ubatch-size 2048 and its
          # co-resident chat model. Loading bare separates those in one run
          # instead of another week of blind re-runs. It cannot take the job
          # green: production RAG always passes the flag (gaia/rag/sdk.py), so a
          # bare load measures something GAIA never does.
          if (-not $embedOk) {
            Show-LemonadeTaskLog | Out-Null
            Write-Host "=== Discriminating probe: loading the embedder WITHOUT --ubatch-size 2048 (diagnostic only) ==="
            $bareOk = Invoke-EmbedProbe
            $logged = Show-LemonadeTaskLog
            $logHint = if ($logged) { "llama-server's own error is in the task-log tail above" } else { "no task log was captured even after a -ForceRestart re-register - check that this runner's ensure-lemonade-running.ps1 carries the stdout/stderr redirect" }

            if ($bareOk) {
              Write-Host "::error::The RAG embedder $embeddingModel loads FINE without ``--ubatch-size 2048`` and fails WITH it, on this runner. That flag is not CI-only - gaia/rag/sdk.py::_load_embedder passes it on every embedder load, so document Q&A and every RAG-backed agent are broken on this hardware the same way, and this eval gate has simply never been able to measure them (0 passes in its history, #2920). Fix the load in gaia/rag/sdk.py - do NOT drop the flag from this probe to go green, which would measure a load GAIA never performs. $logHint"
              exit 1
            }

            # RUNG 5 - the last hypothesis standing. The chat model loaded and
            # served over the same llama-server binary earlier in this step, so
            # the box can spawn it; the flag, the registration, the model's bytes
            # and the backend cache are all eliminated above. What differs from
            # the two green sibling jobs is that they load this embedder on a
            # fresh `-NoModel` server while here a chat model is already resident
            # (#1544 keeps it co-resident by design). Evict everything and load
            # the embedder alone: if that works, co-residency is the cause, and
            # on unified-memory hardware that is a GAIA problem, not a CI one.
            Write-Host "=== Final discriminator: unloading ALL models, then loading the embedder ALONE ==="
            $ErrorActionPreference = "Continue"
            & $env:VENV_PY -c "from gaia.llm.lemonade_client import LemonadeClient; LemonadeClient().unload_model(ignore_if_not_loaded=True); print('[ci] unloaded all models')"
            $ErrorActionPreference = "Stop"
            $aloneOk = Invoke-EmbedProbe -LlamacppArgs "--ubatch-size 2048"
            Show-LemonadeTaskLog | Out-Null
            if ($aloneOk) {
              Write-Host "::error::The RAG embedder $embeddingModel loads ALONE but not while the chat model is resident. This runner cannot hold both at once, and gaia/rag/sdk.py::_load_embedder deliberately keeps the chat model resident (#1544), so on this hardware every RAG-backed agent hits the same wall, and this gate has never been able to measure them. Fix the co-residency in the RAG SDK (evict-then-reload, or a smaller resident context); do NOT paper over it here, since an eval that unloads the chat model measures a configuration GAIA never runs."
              exit 1
            }

            Write-Host "::error::The RAG embedder $embeddingModel cannot be served on this runner under any condition tried. Everything cheap is now eliminated, so do not re-run hoping it clears: the chat model LOADED and served over the same llama-server binary earlier in this step (so the binary spawns fine here and this is not #2826), the load fails identically with and without ``--ubatch-size 2048``, alone and co-resident with the chat model, and the registration, the model's on-disk bytes and the llama.cpp backend cache were each purged and retried. What is left is this specific embedding model on this specific box - most likely the ``--embeddings``/pooling path of the llama.cpp build Lemonade ships here, which is the same class of fault as #1831. Next step is a HUMAN one: on the runner, run Lemonade in a console (not the Scheduled Task, whose stdout/stderr it does not write to) and read llama-server's own failure while reproducing with ``python tests/ci_lemonade_check.py --model $embeddingModel --checkpoint $embeddingCheckpoint --recipe $embeddingRecipe --register-embedding --embeddings --llamacpp-args '--ubatch-size 2048'``. $logHint. Until then rag_quality and context_retention cannot be measured at all - they would score ~2/10 as non-answers against a ~9/10 baseline and read as a model regression."
            exit 1
          }

      # gaia_* scenarios read fixtures under ~/gaia-eval, starter skills and a
      # signed fixture hub. Same script as test_gaia_agent_eval.yml, so the two
      # workflows cannot disagree on what "staged" means. Skills and the hub's
      # trust store go into a TEMPLATE GAIA home that the run step copies into
      # each category's fresh home - never into the runner's real ~/.gaia, where
      # skills captured by an earlier run were still installed (rc16).
      - name: Stage the gaia_* fixtures, skills and fixture hub
        if: env.GAIA_FIXTURES == '1'
        run: |
          $template = Join-Path $env:RUNNER_TEMP "gaia-eval-homes\_staged"
          if (Test-Path $template) { Remove-Item $template -Recurse -Force }
          New-Item -ItemType Directory -Force -Path $template | Out-Null
          & $env:VENV_PY tests/fixtures/gaia/stage_eval_env.py --home "$env:USERPROFILE" --gaia-home $template
          if ($LASTEXITCODE -ne 0) { throw "Staging the gaia_* fixtures failed; the output above says which part." }

      - name: Run this lane's categories serially
        # Own timeout so an overrun fails HERE, by name, with the collect /
        # integrity / compare / upload steps still running on the partial results.
        # A job-level timeout instead CANCELS, which skips every `!cancelled()`
        # step and discards the categories that did finish. See the budget
        # derivation on the job above. Same convention as email_scorecard_refresh.yml.
        #
        # MUST stay below the job's timeout-minutes, or the job cancels first and
        # this is dead code - the 60 min gap is the setup + collect + upload the
        # budget table prices.
        timeout-minutes: 360
        env:
          CLAUDE_CODE_OAUTH_TOKEN: ${{ (vars.CLAUDE_ACCOUNT == 'secondary' && secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY || secrets.CLAUDE_CODE_OAUTH_TOKEN) }}
          # Blank when the OAuth token is present, deliberately: runner.py adds
          # `--bare` only when ANTHROPIC_API_KEY is set, and `--bare` restricts
          # auth to the key alone, so leaving both set would ignore the token.
          ANTHROPIC_API_KEY: ${{ (vars.CLAUDE_ACCOUNT == 'secondary' && secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY || secrets.CLAUDE_CODE_OAUTH_TOKEN) == '' && secrets.ANTHROPIC_API_KEY || '' }}
        run: |
          # Deliberately NOT "Stop": the eval's stderr is piped through `2>&1`
          # below, and under Stop the first stderr line from a native command
          # becomes a terminating NativeCommandError. Every failure here is
          # checked explicitly instead.
          $ErrorActionPreference = "Continue"
          $ProgressPreference = "SilentlyContinue"
          # PYTHONIOENCODING makes the eval emit UTF-8; without this PowerShell
          # decodes it as the OEM codepage and the logs come back as mojibake.
          [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
          New-Item -ItemType Directory -Force -Path eval-out | Out-Null

          function Test-Backend {
            try { Invoke-RestMethod -Uri "$env:BACKEND_URL/api/health" -TimeoutSec 5 | Out-Null; return $true }
            catch { return $false }
          }

          # A leftover backend from a cancelled run still holds :4200 on this
          # persistent runner, and the cleanup below does not fire on job
          # cancellation. Without this check the wait loop can get 200 from that
          # STALE server and evaluate the previous checkout while reporting green.
          if (Test-Backend) {
            Write-Host "::error::$env:BACKEND_URL is already serving - a stale GAIA backend is running on this runner. Kill it (gaia kill) and re-run; otherwise this eval measures the previous checkout, not this one."
            exit 1
          }

          # The GAIA UI backend must live in the SAME shell session as the evals:
          # a backgrounded child does not survive a step boundary. Hence backend
          # start + all this lane's categories in one step.
          #
          # ONE BACKEND PER CATEGORY, each on a fresh, empty GAIA home under the
          # job's temp dir. The runner's real ~/.gaia outlives the job on this
          # pool, and in rc16 the agent recalled "facts" a previous day's memory
          # scenario had stored, and capture scenarios found skills an earlier
          # run had captured. util/eval_gaia_home.py names the variables that move
          # every state root GAIA reads from the environment (memory DB, skills,
          # MCP list, agents, daemon host dir, UI database, config). The shared
          # Lemonade and its models are reached through LEMONADE_BASE_URL and are
          # not touched. A restart per category also keeps one category's memory
          # out of the next one's score.
          $homesRoot = Join-Path $env:RUNNER_TEMP "gaia-eval-homes"
          $stagedHome = Join-Path $homesRoot "_staged"
          New-Item -ItemType Directory -Force -Path $homesRoot | Out-Null

          # The agent under test runs shell commands, so it must not inherit the
          # judge's credentials; only `claude -p` below needs them. Every child
          # this step starts goes through Start-Isolated for that reason.
          # Start-Process resolves relative paths against the .NET process CWD,
          # which is not guaranteed to track the PowerShell location, so callers
          # pass an absolute working directory and absolute redirect targets.
          $judgeCredentials = @{}
          foreach ($name in @("CLAUDE_CODE_OAUTH_TOKEN", "ANTHROPIC_API_KEY")) {
            $judgeCredentials[$name] = [Environment]::GetEnvironmentVariable($name)
          }
          function Start-Isolated($argList, $cwd, $out, $err) {
            foreach ($name in $judgeCredentials.Keys) { Remove-Item "Env:\$name" -ErrorAction SilentlyContinue }
            try {
              return Start-Process -FilePath $env:VENV_PY -ArgumentList $argList `
                -WorkingDirectory $cwd -RedirectStandardOutput $out -RedirectStandardError $err `
                -NoNewWindow -PassThru
            }
            finally {
              foreach ($name in $judgeCredentials.Keys) {
                if ($judgeCredentials[$name]) { Set-Item "Env:\$name" $judgeCredentials[$name] }
              }
            }
          }

          # Stops a category's backend and the gaia daemon it spawned. The daemon
          # lives in the category's GAIA home, so `gaia daemon stop` must run
          # while GAIA_HOME still names that home. taskkill /T takes the
          # backend's children (MCP servers, sidecars) with it; Stop-Process
          # alone orphans them. Returns $false if :4200 is still answering.
          function Stop-CategoryBackend($proc) {
            & $env:VENV_GAIA daemon stop 2>&1 | ForEach-Object { Write-Host "$_" }
            if ($proc -and -not $proc.HasExited) {
              Write-Host "Stopping backend pid $($proc.Id) and its children..."
              & taskkill /F /T /PID $proc.Id 2>&1 | Out-Null
            }
            foreach ($attempt in 1..30) {
              if (-not (Test-Backend)) { return $true }
              Start-Sleep -Seconds 1
            }
            return $false
          }

          # gaia_* lanes: the fixture server, and the env the corpus is written
          # against (tests/fixtures/gaia/README.md), set before any backend
          # starts so the agents it spawns inherit them.
          $fixture = $null
          $backend = $null
          $extraArgs = @()
          $backendCwd = "$PWD"
          if ($env:GAIA_FIXTURES -eq '1') {
            # The backend's working directory is a session's file scope when
            # nothing is attached. The staged fixtures are what the scenarios
            # name; the checkout holds every scenario's expected answer.
            $backendCwd = Join-Path $env:USERPROFILE "gaia-eval"
            $env:GAIA_HUB_URL = "http://127.0.0.1:8765/fixture_hub"
            $env:GAIA_WEB_ALLOWED_HOSTS = "127.0.0.1"
            $env:GAIA_AUTO_APPROVE_TOOLS = "1"
            # The github-triage scenarios are written against the staged fake gh.
            # Without it first on PATH the agent runs the runner's real gh, and
            # auto-approved writes are aimed at real GitHub.
            $env:PATH = (Join-Path $env:USERPROFILE ".gaia-eval-bin") + ";" + $env:PATH
            # gaia_email reads this offline mailbox, never a connected account.
            $env:GAIA_EVAL_MAILBOX = Join-Path $PWD "tests/fixtures/gaia/email/eval_inbox.mbox"
            if (-not (Test-Path (Join-Path $stagedHome "skills"))) {
              Write-Host "::error::$stagedHome\skills is missing - the staging step did not write the template GAIA home the gaia_* categories copy their skills from."
              exit 1
            }
            $fixture = Start-Isolated @("tests/fixtures/gaia/serve_fixtures.py","--port","8765") "$PWD" `
              (Join-Path $PWD "eval-out\fixtures.log") (Join-Path $PWD "eval-out\fixtures.err.log")
            if (-not $fixture) {
              Write-Host "::error::Could not start tests/fixtures/gaia/serve_fixtures.py - the gaia_* web and skill-hub scenarios depend on it."
              exit 1
            }
            # live: hits a real external service. local_blocked_win_shim: kept off
            # until a CI run confirms the staged gh.exe answers (#4427).
            $extraArgs = @('--exclude-tag', 'live', '--exclude-tag', 'local_blocked_win_shim')
          }

          try {
            if ($fixture) {
              $served = $false
              foreach ($attempt in 1..6) {
                if ($fixture.HasExited) { break }
                try { Invoke-RestMethod -Uri "http://127.0.0.1:8765/atlas.html" -TimeoutSec 10 | Out-Null; $served = $true; break }
                catch { Start-Sleep -Seconds 5 }
              }
              if (-not $served) {
                Write-Host "::error::Fixture server not serving on port 8765 - see eval-out\fixtures*.log. The gaia_* web and skill-hub scenarios depend on it."
                Get-Content (Join-Path $PWD "eval-out\fixtures.err.log") -ErrorAction SilentlyContinue
                exit 1
              }
              Write-Host "Fixture server healthy."
            }

            # Deliberately not asserting a literal "gaia": the id is read from
            # gaia.eval.config so the registration check below tracks the default
            # instead of becoming a second place to update.
            $defaultAgent = (& $env:VENV_PY -c "from gaia.eval.config import DEFAULT_AGENT_TYPE; print(DEFAULT_AGENT_TYPE)" 2>&1 | Out-String).Trim()
            if ($LASTEXITCODE -ne 0 -or -not $defaultAgent) {
              Write-Host "::error::Could not read DEFAULT_AGENT_TYPE from gaia.eval.config - the venv install is broken. Output: $defaultAgent"
              exit 1
            }

            # Per-category wall clock. The job timeout was guessed once and cost a
            # 148-min run (30869194618) that produced one category out of three;
            # these lines are what replaces the guess with a measurement. Each is
            # written the moment its category ends - appended to the step summary,
            # not buffered to the end - so a run that overruns or crashes still
            # leaves behind the timings of whatever DID finish. A category's time
            # includes its own backend start.
            #
            # Not `Out-File -Encoding utf8`: on Windows PowerShell 5.1 that
            # prepends a BOM, and a BOM ahead of the leading `##` stops GitHub
            # rendering it as a heading. AppendAllText with UTF8Encoding($false)
            # has an empty preamble, so no BOM on create or append.
            $summaryEnc = New-Object System.Text.UTF8Encoding($false)
            function Add-Summary($line) {
              if ($env:GITHUB_STEP_SUMMARY) {
                [System.IO.File]::AppendAllText($env:GITHUB_STEP_SUMMARY, "$line`r`n", $summaryEnc)
              }
            }
            $swTotal = [Diagnostics.Stopwatch]::StartNew()
            Add-Summary "## Eval wall clock - lane ``$env:EVAL_LANE`` on $env:COMPUTERNAME, model ``$env:AGENT_MODEL``"
            Add-Summary ""
            # category -> "id,id", from a targeted dispatch. Empty on a tag run.
            $selected = @{}
            foreach ($pair in (($env:EVAL_SCENARIOS -split '\s+') | Where-Object { $_ })) {
              $kv = $pair -split '=', 2
              $selected[$kv[0]] = $kv[1]
            }
            if ($env:EVAL_TARGETED -eq 'true') {
              Add-Summary "**$env:EVAL_SELECTION**"
              Add-Summary ""
              foreach ($c in $selected.Keys) { Add-Summary "- ``$c``: $($selected[$c] -replace ',', ', ')" }
              Add-Summary ""
            }
            Add-Summary "| category | minutes |"
            Add-Summary "| --- | ---: |"

            # Serial by construction: a plain foreach in one session, never
            # backgrounded. Two concurrent `gaia eval agent` runs race-evict each
            # other's model out of the single Lemonade slot (CLAUDE.md).
            #
            # `gaia eval agent --category X` exits 0 whatever the scenarios score -
            # it writes the scorecard and returns. So a non-zero exit here is a real
            # harness crash, never a regression; the regression verdict comes from
            # the --compare step below.
            #
            # No --agent-type: the eval already defaults to the flagship
            # (gaia.eval.config.DEFAULT_AGENT_TYPE) and every scenario scores it,
            # so naming one here would only be a second place to drift.
            #
            # Category list comes from this LANE's EVAL_CATEGORIES, handed down
            # from eval/ci_lanes.json by the `lanes` job - do not hardcode one
            # here. That is what let four steps drift apart.
            $all = ($env:EVAL_CATEGORIES -split '\s+') | Where-Object { $_ }
            $categories = $all
            if ($env:EMBEDDER_OK -eq 'false') {
              # INTERSECT with this lane. The no-embedder list is workflow-wide, so
              # taking it whole would make every lane run another lane's category.
              $ok = ($env:EVAL_CATEGORIES_NO_EMBEDDER -split '\s+') | Where-Object { $_ }
              $categories = @($all | Where-Object { $ok -contains $_ })
              if ($categories.Count -eq 0) {
                # Fatal regardless of `enforce`, unlike a PARTIAL lane below.
                # `enforce` governs what a measurement is allowed to say, never
                # whether the harness ran - and an empty foreach exits 0, so this
                # lane's green would be the absence of a measurement dressed as one.
                Add-Summary "| **NOTHING MEASURED** | - |"
                Write-Host "::error::Lane '$env:EVAL_LANE' measured NOTHING. The RAG embedder will not load on this runner (#3016), the run is cut to EVAL_CATEGORIES_NO_EMBEDDER, and none of this lane's categories ($($all -join ', ')) are on that list. This lane's green/red says nothing about quality. Fix the embedder on the devlab-dispatch image, or re-dispatch once #3016 is resolved; the embedder ladder output is in the preflight step above."
                exit 1
              }
              Write-Host "::warning::Embedder unavailable (#3016) - measuring $($categories -join ', ') only. The run is cut to the categories listed in EVAL_CATEGORIES_NO_EMBEDDER; the rest of this lane is not run and the integrity gate below reports it as NOT MEASURED (and fails the job when enforce=true). Not every skipped category needs the embedder - the list is what this runner has been shown to measure without it, not a claim about the others."
            }
            Write-Host "Lane '$env:EVAL_LANE' categories this run: $($categories -join ', ') ($($categories.Count) of $($all.Count))"
            foreach ($category in $categories) {
              Write-Host "=================================================================="
              Write-Host "  gaia eval agent --category $category"
              Write-Host "=================================================================="
              $swCat = [Diagnostics.Stopwatch]::StartNew()

              # A fresh GAIA home for this category: the gaia_* skills template
              # copied in, every state root pointed at it, then checked.
              $catHome = Join-Path $homesRoot $category
              if (Test-Path $catHome) {
                Write-Host "::error::$catHome exists before $category started. Every category must start on a GAIA home nothing has written to."
                exit 1
              }
              New-Item -ItemType Directory -Force -Path $catHome | Out-Null
              if ($env:GAIA_FIXTURES -eq '1') {
                Copy-Item (Join-Path $stagedHome "skills") (Join-Path $catHome "skills") -Recurse -Force
              }
              $isoEnv = @(& $env:VENV_PY util/eval_gaia_home.py env --home $catHome)
              if ($LASTEXITCODE -ne 0 -or $isoEnv.Count -eq 0) {
                Write-Host "::error::util/eval_gaia_home.py env failed for $catHome; the output above says why."
                exit 1
              }
              foreach ($line in $isoEnv) {
                $kv = $line -split '=', 2
                Set-Item "Env:\$($kv[0])" $kv[1]
                Write-Host "  $($kv[0])=$($kv[1])"
              }
              # The pre-run check: every state root resolves into this home and
              # the memory database holds nothing.
              & $env:VENV_PY util/eval_gaia_home.py check --home $catHome
              if ($LASTEXITCODE -ne 0) {
                Write-Host "::error::$category would not start on an isolated, empty GAIA home (NOT ISOLATED lines above). Refusing to score it with state from another run."
                exit 1
              }
              # The model under test is this home's default_model, read back
              # through GAIA's own config. Without it a fresh home resolves to
              # the code default, whatever this pool is meant to measure.
              & $env:VENV_PY util/eval_gaia_home.py pin-model --home $catHome --model $env:AGENT_MODEL
              if ($LASTEXITCODE -ne 0) {
                Write-Host "::error::Could not pin $env:AGENT_MODEL in $catHome for $category (NOT PINNED lines above). Refusing to score whatever model the home would fall back to."
                exit 1
              }
              # A tier machine holding memory beside other categories: the store
              # is live for `memory` alone. Set before this category's backend
              # starts, which is what reads it.
              if ($env:EVAL_MEMORY_CATEGORIES) {
                $memOn = @($env:EVAL_MEMORY_CATEGORIES -split '\s+') -contains $category
                $env:GAIA_MEMORY_DISABLED = if ($memOn) { '0' } else { '1' }
                $env:GAIA_MEMORY_ADMIN = if ($memOn) { '1' } else { '0' }
                $env:GAIA_MEMORY_MCP_ALWAYS = if ($memOn) { '1' } else { '0' }
                Write-Host "  memory for ${category}: $(if ($memOn) { 'live' } else { 'off' })"
              }

              # The acme MCP stub exists only for gaia_mcp; staged for any other
              # category it would widen that category's toolset. It writes the MCP
              # list in this category's GAIA home, which the backend reads.
              $stub = 'tests/fixtures/gaia/mcp_stub/stage_mcp_stub.py'
              if ($category -eq 'gaia_mcp') {
                & $env:VENV_PY $stub install
                if ($LASTEXITCODE -ne 0) { Write-Host "::error::Staging the gaia_mcp stub servers failed; the output above says why."; exit 1 }
              }

              # A leftover backend still holding :4200 would answer the health
              # probe below, and this category would measure the previous
              # checkout (or the previous category's home) while reporting green.
              if (Test-Backend) {
                Write-Host "::error::$env:BACKEND_URL is already serving before $category started - a stale GAIA backend is running on this runner. Kill it (gaia kill) and re-run; otherwise this eval measures the wrong backend."
                exit 1
              }
              $outLog = Join-Path $PWD "eval-out\backend-$category.out.log"
              $errLog = Join-Path $PWD "eval-out\backend-$category.err.log"
              $backend = Start-Isolated @("-m","gaia.ui.server","--port","4200","--host","127.0.0.1") $backendCwd $outLog $errLog
              if (-not $backend) {
                Write-Host "::error::Could not start gaia.ui.server - the venv install is broken or python is not on PATH."
                exit 1
              }

              # 120 x 5s sleeps = ~10 min of waiting, more when a probe itself
              # times out. The job timeout is the real backstop.
              Write-Host "Waiting ~10 min for the GAIA UI backend on $env:BACKEND_URL (pid $($backend.Id))..."
              $ready = $false
              foreach ($attempt in 1..120) {
                if ($backend.HasExited) {
                  Write-Host "::error::gaia.ui.server exited during startup (exit $($backend.ExitCode)). Backend log follows."
                  Get-Content $outLog, $errLog -ErrorAction SilentlyContinue
                  exit 1
                }
                if (Test-Backend) { $ready = $true; break }
                Start-Sleep -Seconds 5
              }
              if (-not $ready) {
                Write-Host "::error::GAIA UI backend did not become healthy at $env:BACKEND_URL/api/health in time. Backend log follows."
                Get-Content $outLog, $errLog -ErrorAction SilentlyContinue
                exit 1
              }
              Write-Host "Backend healthy."

              # Healthy is not the same as "can run these scenarios". The eval asks
              # the backend for DEFAULT_AGENT_TYPE on every scenario, and an agent
              # that ships as a hub package is simply absent when its editable
              # install is missing: POST /api/sessions 422s, every scenario in this
              # lane records INFRA_ERROR, and the eval still exits 0 because it
              # produced a scorecard. Assert the id resolves BEFORE spending the
              # slot.
              try {
                $agentList = Invoke-RestMethod -Uri "$env:BACKEND_URL/api/agents" -TimeoutSec 30 -ErrorAction Stop
              } catch {
                Write-Host "::error::GET $env:BACKEND_URL/api/agents failed after the backend reported healthy: $($_.Exception.Message). Backend log follows."
                Get-Content $outLog, $errLog -ErrorAction SilentlyContinue
                exit 1
              }
              $agentIds = @($agentList.agents | ForEach-Object { $_.id })
              if ($agentIds -notcontains $defaultAgent) {
                Write-Host "::error::The eval's default agent '$defaultAgent' is not registered in this backend, so every scenario would 422 at session creation and record INFRA_ERROR. Registered ids: $($agentIds -join ', '). Fix: add the hub package that ships '$defaultAgent' to install-package in this job's Setup Python Environment step (hub/agents/<id>/python; the flagship is hub/agents/gaia/python). See src/gaia/eval/config.py for the default and src/gaia/agents/registry.py for how entry-point agents register."
                Get-Content $outLog, $errLog -ErrorAction SilentlyContinue
                exit 1
              }
              Write-Host "Default agent '$defaultAgent' is registered. Agents: $($agentIds -join ', ')"

              # GAIA_MEMORY_DISABLED=0 keeps the store alive, but the Agent UI's own
              # memory switch defaults OFF, and with it off every `remember` returns
              # "skipped". Flip it for the lanes that measure memory; the runner's
              # preflight refuses to start a memory scenario while it is still off.
              # Per category, because each category's backend has a fresh database.
              if ($env:GAIA_MEMORY_DISABLED -eq '0') {
                try {
                  $memory = Invoke-RestMethod -Method Put -Uri "$env:BACKEND_URL/api/memory/settings" `
                    -Headers @{ "X-Gaia-UI" = "1" } -ContentType "application/json" `
                    -Body '{"memory_enabled": true}' -TimeoutSec 30 -ErrorAction Stop
                } catch {
                  Write-Host "::error::PUT $env:BACKEND_URL/api/memory/settings failed: $($_.Exception.Message). This lane measures long-term memory and cannot run with it off. Backend log follows."
                  Get-Content $outLog, $errLog -ErrorAction SilentlyContinue
                  exit 1
                }
                if ($memory.memory_enabled -ne $true) {
                  Write-Host "::error::The backend did not switch memory on (settings: $($memory | ConvertTo-Json -Compress)). This lane measures long-term memory and cannot run with it off."
                  exit 1
                }
                Write-Host "Long-term memory switched on in the backend."
              }

              # The same pre-run check, asked of the store the BACKEND opened: it
              # must be empty, and it must be this category's file. A backend that
              # ignored GAIA_MEMORY_DB would pass the check above and still read
              # the runner's real memory.
              try {
                $stats = Invoke-RestMethod -Uri "$env:BACKEND_URL/api/memory/stats" -Headers @{ "X-Gaia-UI" = "1" } -TimeoutSec 30 -ErrorAction Stop
              } catch {
                Write-Host "::error::GET $env:BACKEND_URL/api/memory/stats failed: $($_.Exception.Message). Without it there is no proof $category starts with empty memory."
                exit 1
              }
              $held = [ordered]@{
                knowledge = $stats.knowledge.total
                procedures = $stats.procedures.total
                conversation_turns = $stats.conversations.total_turns
              }
              $notEmpty = @($held.Keys | Where-Object { $held[$_] -ne 0 } | ForEach-Object { "$_=$($held[$_])" })
              if ($notEmpty.Count -gt 0) {
                Write-Host "::error::The backend's memory store is not empty at the start of $category ($($notEmpty -join ', ')). Its answers would draw on another run's memory, as rc16's did. Stats: $($stats | ConvertTo-Json -Compress -Depth 4)"
                exit 1
              }
              if (-not (Test-Path $env:GAIA_MEMORY_DB)) {
                Write-Host "::error::The backend answered /api/memory/stats but did not create $env:GAIA_MEMORY_DB, so the store it opened is somewhere else - not $category's isolated GAIA home."
                exit 1
              }
              Write-Host "Memory at the start of ${category}: empty, at $env:GAIA_MEMORY_DB."

              # Reset first: a command that fails to LAUNCH leaves $LASTEXITCODE
              # at its previous value, so an unset/stale 0 would read as success.
              $global:LASTEXITCODE = 0
              # A targeted run names its scenarios; the CLI refuses any that
              # would not run, so a typo cannot shrink the run silently.
              $pick = @()
              if ($selected.ContainsKey($category)) { $pick = @('--scenario', $selected[$category]) }
              # Not Tee-Object: on Windows PowerShell 5.1 it has no -Encoding and
              # writes UTF-16, which the `Output:` parse in the next step then has
              # to guess at. Echo + Out-File keeps the log UTF-8 and greppable.
              # --agent-model turns any scenario whose session ran another model
              # into INFRA_ERROR and records the model in the scorecard, which is
              # what --compare checks against the baseline's meta.json.
              & $env:VENV_GAIA eval agent --category $category --backend "$env:BACKEND_URL" --model "$env:EVAL_MODEL" --agent-model "$env:AGENT_MODEL" @extraArgs @pick 2>&1 |
                ForEach-Object { Write-Host "$_"; "$_" } |
                Out-File -FilePath "eval-out\$category.log" -Encoding utf8
              # Capture BEFORE anything else can clobber it. .NET calls and
              # Write-Host leave $LASTEXITCODE alone, but reading it first keeps
              # that from being a thing a later edit has to remember.
              $code = $LASTEXITCODE
              if ($category -eq 'gaia_mcp') {
                & $env:VENV_PY $stub remove
                if ($LASTEXITCODE -ne 0) { Write-Host "::error::Removing the gaia_mcp stub servers failed; the output above says why."; exit 1 }
              }

              $stopped = Stop-CategoryBackend $backend
              $backend = $null
              # The daemon's log dies with the home; keep it beside the backend's.
              $daemonLog = Join-Path $catHome "host\daemon.log"
              if (Test-Path $daemonLog) { Copy-Item $daemonLog "eval-out\daemon-$category.log" -Force }
              if (-not $stopped) {
                Write-Host "::error::The backend for $category still answers on $env:BACKEND_URL after being stopped; the next category would talk to it."
                exit 1
              }
              Remove-Item $catHome -Recurse -Force -ErrorAction SilentlyContinue
              if (Test-Path $catHome) {
                Write-Host "::warning::Could not fully delete $catHome after $category (a file is still open). The next category uses its own home, so its score is unaffected; the cleanup step retries."
              }
              $swCat.Stop()

              $mins = [math]::Round($swCat.Elapsed.TotalMinutes, 1)
              Write-Host "[TIMING] ${category}: $mins min (running total $([math]::Round($swTotal.Elapsed.TotalMinutes, 1)) min of the 360-min step budget)"
              Add-Summary "| ``$category`` | $mins |"

              if ($code -ne 0) {
                # Never `exit $LASTEXITCODE` - on a launch failure it can be $null,
                # which PowerShell turns into exit 0 and the step passes GREEN
                # having measured nothing.
                Write-Host "::error::gaia eval agent --category $category crashed with exit $code (a harness failure, not a regression verdict)."
                exit 1
              }
            }

            $swTotal.Stop()
            $totalMins = [math]::Round($swTotal.Elapsed.TotalMinutes, 1)
            Write-Host "[TIMING] lane $env:EVAL_LANE, all $($categories.Count) categories: $totalMins min"
            Add-Summary "| **total** | **$totalMins** |"
            # The aggregate job cannot time a lane it did not run, and the lane's
            # own duration is the only number that tells you which lane to rebalance.
            # BOM-less: PowerShell 5.1's -Encoding UTF8 writes one and json.loads
            # chokes on it, which would blank this out with no error anywhere.
            $timing = @{ lane = $env:EVAL_LANE; total_minutes = $totalMins; categories = $categories } | ConvertTo-Json -Compress
            [System.IO.File]::WriteAllText("$PWD\eval-out\lane-timing.json", $timing, (New-Object System.Text.UTF8Encoding($false)))
          }
          finally {
            # Persistent runner: never leave the stub registered for the next job.
            # Its server list sat in a category home; its activation did not.
            if ($env:GAIA_FIXTURES -eq '1') {
              & $env:VENV_PY tests/fixtures/gaia/mcp_stub/stage_mcp_stub.py remove
            }
            if ($backend) { Stop-CategoryBackend $backend | Out-Null }
            if ($fixture -and -not $fixture.HasExited) {
              Write-Host "Stopping fixture server pid $($fixture.Id)..."
              Stop-Process -Id $fixture.Id -Force -ErrorAction SilentlyContinue
            }
          }

      # Deletes every category's GAIA home, in its own always() step so a step
      # timeout (which kills the shell above before its `finally`) still cleans
      # up. Stops any daemon a home still holds first: a running daemon keeps
      # its host dir open. The runner's real ~/.gaia is never touched.
      - name: Delete the per-category GAIA homes
        if: always()
        run: |
          $ErrorActionPreference = "Continue"
          # The stub's activation lives in the real ~/.gaia; a step timeout skips
          # the run step's `finally`, so deactivate it here too.
          if ($env:GAIA_FIXTURES -eq '1' -and $env:VENV_PY -and (Test-Path $env:VENV_PY)) {
            & $env:VENV_PY tests/fixtures/gaia/mcp_stub/stage_mcp_stub.py remove 2>&1 | ForEach-Object { Write-Host "$_" }
          }
          $homesRoot = Join-Path $env:RUNNER_TEMP "gaia-eval-homes"
          if (-not (Test-Path $homesRoot)) { Write-Host "No GAIA homes to delete."; exit 0 }
          foreach ($dir in @(Get-ChildItem $homesRoot -Directory)) {
            if (Test-Path (Join-Path $dir.FullName "host\instance.json")) {
              $env:GAIA_HOME = $dir.FullName
              $env:GAIA_DAEMON_HOME = Join-Path $dir.FullName "host"
              & $env:VENV_GAIA daemon stop 2>&1 | ForEach-Object { Write-Host "$_" }
            }
          }
          Remove-Item $homesRoot -Recurse -Force -ErrorAction SilentlyContinue
          if (Test-Path $homesRoot) {
            # Not an error: no later category can read it, because each one
            # refuses to start on a home that already exists, and the runner
            # clears its temp dir before the next job.
            Write-Host "::warning::Could not fully delete $homesRoot; a process still holds a file in it. Files left:"
            Get-ChildItem $homesRoot -Recurse -File -ErrorAction SilentlyContinue | Select-Object -First 20 | ForEach-Object { Write-Host "  $($_.FullName)" }
            exit 0
          }
          Write-Host "Deleted $homesRoot."

      - name: Collect scorecards from the printed run directories
        # Not always(): on a concurrency cancel or a job-level timeout this would
        # pile misleading "missing scorecard" errors on top of the cancellation.
        # An eval STEP timeout is deliberately not in that bucket - it fails
        # rather than cancels, so this runs and names exactly which categories
        # went unmeasured, which is the diagnostic you want.
        if: ${{ !cancelled() }}
        run: |
          $ErrorActionPreference = "Stop"
          # This step COLLECTS; it does not judge. A category with no scorecard is
          # reported here and lands in the integrity gate below as "no scorecard
          # produced", which is where the enforce policy is applied - once, in one
          # place. Exiting non-zero here as well would re-introduce a second,
          # enforce-blind blocking decision, which is exactly the bug being fixed.
          # A genuine harness crash still fails at the eval step itself.
          $level = if ($env:ENFORCE -eq 'true') { 'error' } else { 'warning' }

          $all = ($env:EVAL_CATEGORIES -split '\s+') | Where-Object { $_ }
          $categories = $all
          if ($env:EMBEDDER_OK -eq 'false') {
            $ok = ($env:EVAL_CATEGORIES_NO_EMBEDDER -split '\s+') | Where-Object { $_ }
            $categories = @($all | Where-Object { $ok -contains $_ })
          }
          foreach ($category in $categories) {
            $log = "eval-out\$category.log"
            if (-not (Test-Path $log)) {
              Write-Host "::${level}::No log for $category - the chain stopped before it ran."
              continue
            }
            # The runner prints an ABSOLUTE `Output: <run-dir>` line
            # (src/gaia/eval/runner.py::_print_summary) - a Windows path with
            # backslashes. Take the last match so a re-run inside one invocation
            # wins. Never glob by mtime.
            $hit = Select-String -Path $log -Encoding UTF8 -Pattern '^Output:\s*(.+?)\s*$' | Select-Object -Last 1
            $runDir = if ($hit) { $hit.Matches[0].Groups[1].Value } else { $null }
            $scorecard = if ($runDir) { Join-Path $runDir "scorecard.json" } else { $null }
            if (-not $runDir -or -not (Test-Path $scorecard)) {
              $shown = if ($runDir) { $runDir } else { "<none>" }
              Write-Host "::${level}::Could not resolve a scorecard for $category (parsed run dir: '$shown'). The eval did not print an 'Output:' line, or it did not write scorecard.json - see eval-out\$category.log."
              continue
            }
            New-Item -ItemType Directory -Force -Path "eval-out\$category" | Out-Null
            Copy-Item $scorecard "eval-out\$category\scorecard.json"
            # Take the whole run directory, not just the score. This machine is
            # destroyed when the job ends, so a trace left behind is gone for good
            # - and the traces are the transcripts, judge rationales and failure
            # analysis, i.e. everything you need to tell a real regression from a
            # flaky judge. The scrub step below is what makes that safe to publish.
            $summary = Join-Path $runDir "summary.md"
            if (Test-Path $summary) { Copy-Item $summary "eval-out\$category\summary.md" }
            $traces = Join-Path $runDir "traces"
            if (Test-Path $traces) { Copy-Item $traces "eval-out\$category\traces" -Recurse -Force }
            $junit = Join-Path $runDir "results.xml"
            if (Test-Path $junit) { Copy-Item $junit "eval-out\$category\results.xml" }
            Write-Host "${category}: $runDir"
          }

          # Lemonade writes outside the workspace, so it is not swept up by the
          # upload path. Without this the only record of a model that would not
          # load dies with the machine.
          $lemonadeLogs = "C:\ProgramData\GaiaLemonadeServer"
          if (Test-Path $lemonadeLogs) {
            New-Item -ItemType Directory -Force -Path "eval-out\lemonade" | Out-Null
            Get-ChildItem -Path $lemonadeLogs -Filter "lemonade-task-*.log" -ErrorAction SilentlyContinue |
              ForEach-Object { Copy-Item $_.FullName "eval-out\lemonade\$($_.Name)" -Force }
          }

      - name: Integrity gate — did we actually measure?
        if: ${{ !cancelled() }}
        run: |
          $ErrorActionPreference = "Stop"
          # `--compare` only scores scenarios present on BOTH sides, so a run whose
          # scenarios vanished exits 0 and reads as "no regression" (verified: 6 of
          # 7 dropped -> exit 0). This gate asks the other question - is there a
          # measurement at all - and is the only thing between an empty run and a
          # green check. The logic lives in gaia.eval.integrity_gate so it is unit
          # tested (tests/unit/eval/test_integrity_gate.py) rather than being
          # seventy lines of untestable PowerShell.
          #
          # IT HONORS `enforce`, and that is the fix for the state this gate was in.
          # An earlier revision failed UNCONDITIONALLY on the reasoning that report
          # mode may soften "worse" but never "did not measure". Sound in principle;
          # in practice the embedder cannot load on this runner (#3016, an
          # environment defect no PR can fix), so the gate went red on every PR
          # touching its path filter while the job's own summary printed "report
          # mode ... build stays green". A workflow that advertises non-blocking and
          # blocks anyway is worse than either posture: it blocked 7 PRs whose diffs
          # had nothing to do with the eval, and people stop reading a check that is
          # always red. So `enforce` now decides the EXIT CODE for both incomplete
          # and worse-than-baseline; it never decides VISIBILITY. In report mode
          # every finding is still a ::warning:: annotation and still lands in the
          # job summary, so an incomplete run is impossible to mistake for a
          # complete one - it just does not veto an unrelated PR.
          #
          # `skipped` (SKIPPED_NO_DOCUMENT) counts as unmeasured for a subtle reason
          # documented in the module: those scenarios keep their ids, so the
          # missing-scenario check stays clean, and compare files them under
          # "not a quality signal".
          #
          # With no baseline committed the gate keeps working on everything that
          # does not need one - a scorecard has to exist, and its scenarios have
          # to have produced a measurement. Only the "a baseline scenario went
          # missing" check goes dark, and it says so per category.
          $all = ($env:EVAL_CATEGORIES -split '\s+') | Where-Object { $_ }
          $ok = ($env:EVAL_CATEGORIES_NO_EMBEDDER -split '\s+') | Where-Object { $_ }
          $measured = @($all | Where-Object { $ok -contains $_ })
          $gateArgs = @(
            "--baseline-dir", $env:BASELINE_DIR,
            "--results-dir", "eval-out"
          )
          if ($env:EMBEDDER_OK -eq 'false') {
            foreach ($c in $measured) { $gateArgs += @("--category", $c) }
            foreach ($c in ($all | Where-Object { $measured -notcontains $_ })) { $gateArgs += @("--not-measured", $c) }
            # One reason covers every skipped category, so it can only say why
            # the RUN was cut - not why each category needs the embedder, which
            # for several of them it does not.
            $gateArgs += @("--not-measured-reason", "the RAG embedder will not load on this runner (#3016), so the run was cut to EVAL_CATEGORIES_NO_EMBEDDER and this category was not on that list")
          } else {
            foreach ($c in $all) { $gateArgs += @("--category", $c) }
          }
          if ($env:ENFORCE -eq 'true') { $gateArgs += "--enforce" }
          if ($env:FAIL_ON_ZERO_TOKENS -eq 'true') { $gateArgs += "--fail-on-zero-token-errors" }

          # Reset first: a command that fails to LAUNCH leaves $LASTEXITCODE at its
          # previous value, so a stale 0 would read as a complete run.
          $global:LASTEXITCODE = 0
          & $env:VENV_PY -m gaia.eval.integrity_gate @gateArgs
          $code = $LASTEXITCODE
          if ($code -ne 0) {
            Write-Host "::error::Integrity gate failed (exit $code) with enforce=$env:ENFORCE for lane '$env:EVAL_LANE'. Download this run's 'eval-$env:EVAL_LANE' artifact: it carries the traces, the backend logs (backend-<category>.err.log) and the Lemonade task logs from this machine, which no longer exists."
            exit 1
          }

      - name: Compare each category against its committed flagship baseline
        if: ${{ !cancelled() }}
        run: |
          $regressed = @()
          $failedCompare = $false

          # Say it, do not skip quietly. With nothing to compare against, this
          # step has no verdict to give, and a green check that looks like
          # "no regression" when nothing was checked is the failure mode the
          # whole gate exists to avoid. The preflight already hard-failed the
          # enforce=true case, so reaching here with no baseline means report mode.
          if (-not (Test-Path $env:BASELINE_DIR)) {
            Write-Host "::warning::NO REGRESSION VERDICT: no baseline at $env:BASELINE_DIR, so nothing was compared. The scorecards this lane produced are uploaded as the 'eval-$env:EVAL_LANE' artifact - they are the raw material for the first flagship baseline, not a pass."
            "" | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
            "### Regression check, lane ``$env:EVAL_LANE``: NOT PERFORMED" | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
            "" | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
            "No flagship baseline is committed at ``$env:BASELINE_DIR``, so this run measured quality but compared it to nothing. A green check here does NOT mean no regression." | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
            exit 0
          }
          # The model's baseline directory exists but holds no scorecards yet
          # (the Qwen one, until a full sweep is captured). Never fall back to
          # another model's baseline.
          if (@(Get-ChildItem $env:BASELINE_DIR -Filter 'scorecard_*.json').Count -eq 0) {
            Write-Host "::warning::NO $env:AGENT_MODEL BASELINE YET: nothing in lane '$env:EVAL_LANE' was compared. The scorecards are uploaded as the 'eval-$env:EVAL_LANE' artifact. Capture the baseline from a full sweep ($env:BASELINE_DIR/README.md)."
            "" | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
            "### Regression check, lane ``$env:EVAL_LANE``: NO $env:AGENT_MODEL BASELINE YET" | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
            "" | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
            "``$env:BASELINE_DIR`` holds no scorecards, so this run compared nothing. It is not compared with another model's baseline. A green check here does NOT mean no regression." | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Encoding utf8 -Append
            exit 0
          }

          $all = ($env:EVAL_CATEGORIES -split '\s+') | Where-Object { $_ }
          $categories = $all
          if ($env:EMBEDDER_OK -eq 'false') {
            $ok = ($env:EVAL_CATEGORIES_NO_EMBEDDER -split '\s+') | Where-Object { $_ }
            $categories = @($all | Where-Object { $ok -contains $_ })
          }
          foreach ($category in $categories) {
            $current = "eval-out\$category\scorecard.json"
            $baseline = Join-Path $env:BASELINE_DIR "scorecard_$category.json"
            Write-Host ""
            Write-Host "=================================================================="
            Write-Host "  ${category}: $baseline  ->  $current"
            Write-Host "=================================================================="
            if (-not (Test-Path $current)) {
              # NOT $failedCompare: "we did not measure" is the integrity gate's
              # verdict and it has already rendered it under the enforce policy.
              # Counting it again here would fail the job regardless of enforce.
              Write-Host "::warning::${category}: no scorecard produced - nothing to compare (already reported by the integrity gate)."
              continue
            }
            if (-not (Test-Path $baseline)) {
              # Preflight already refused this under enforce=true.
              Write-Host "::warning::${category}: NOT COMPARED - no baseline at $baseline yet (reported at preflight)."
              continue
            }
            # `--compare` exits 2 on status/score/time regressions, 0 otherwise
            # (src/gaia/cli.py). It only DIFFS - the eval above already ran.
            # Reset first: a launch failure would otherwise inherit the PREVIOUS
            # category's 2 and be misreported as this category regressing.
            $global:LASTEXITCODE = 0
            & $env:VENV_GAIA eval agent --compare "$baseline" "$current"
            switch ($LASTEXITCODE) {
              0 { Write-Host "${category}: no regression vs baseline." }
              2 {
                Write-Host "::warning::${category}: regression detected vs $baseline (see the diff above)."
                $regressed += $category
              }
              # Exit 3: the baseline was measured on another agent model or
              # eval model. A config error, never a verdict, so always fatal.
              3 {
                Write-Host "::error::${category}: refused to compare - $baseline was captured on a different agent model or eval model than this run ($env:AGENT_MODEL, $env:EVAL_MODEL). The reason is printed above."
                $failedCompare = $true
              }
              default {
                Write-Host "::error::${category}: compare failed with exit $LASTEXITCODE (not a regression verdict - a harness/IO error)."
                $failedCompare = $true
              }
            }
          }

          Write-Host ""
          Write-Host "=================================================================="
          Write-Host "  SUMMARY - enforce=$env:ENFORCE"
          Write-Host "=================================================================="
          if ($env:EVAL_TARGETED -eq 'true') {
            Write-Host "$env:EVAL_SELECTION"
            Write-Host "Each comparison above judged only the selected scenarios; its NOT RUN list names the rest."
          }
          if ($regressed.Count -gt 0) {
            Write-Host "Regressed categories: $($regressed -join ' ')"
          } else {
            Write-Host "Regressed categories: none"
          }
          Write-Host "Model under test: $env:AGENT_MODEL"
          Write-Host "Baselines: $env:BASELINE_DIR (captured on the same model)"
          Write-Host "Scorecards, traces and logs uploaded as the 'eval-$env:EVAL_LANE' artifact."
          Write-Host ""
          Write-Host "Read the per-scenario diff above before treating a regression as a bug:"
          Write-Host "about 18% of scenarios flip between two runs of the same code (#5493)."
          Write-Host ""
          Write-Host "TIME regressions (>2x per-scenario wall clock) count toward the same"
          Write-Host "verdict, and are the weakest signal here. Each lane runs on a freshly"
          Write-Host "provisioned devlab-dispatch machine, so a baseline captured on a"
          Write-Host "different box of the same SoC carries that box's noise. Read a time"
          Write-Host "verdict as a prompt to look, not as a result."
          Write-Host "=================================================================="

          # A compare that exits with something other than 0 or 2 is the TOOL
          # failing, not a verdict - `enforce` governs what a measurement is
          # allowed to say, never whether the tool ran. Always fatal.
          if ($failedCompare) {
            Write-Host "::error::One or more comparisons could not be performed - 'gaia eval agent --compare' itself failed. This is a harness/IO error, not a regression verdict, so it fails regardless of enforce."
            exit 1
          }
          if ($regressed.Count -gt 0) {
            if ($env:ENFORCE -eq "true") {
              Write-Host "::error::Regressions in: $($regressed -join ' '). enforce=true - failing the build. Fix the prompt/tooling and re-run, or (if the regression is intentional) regenerate the baseline with --save-baseline and call it out in the PR description."
              exit 1
            }
            Write-Host "Report mode (enforce=false): regressions logged as warnings, build stays green."
            Write-Host "Re-dispatch this workflow with enforce=true to make them blocking."
          }

          # THIS LINE IS THE REPORT MODE. GitHub appends
          # `if (Test-Path variable:\LASTEXITCODE) { exit $LASTEXITCODE }` to every
          # PowerShell step, so falling off the end here would re-export the LAST
          # `--compare` exit code: a report-mode regression in the final category
          # would fail the build anyway, and whether it did would depend on which
          # category regressed. Same guard as install-lemonade/action.yml.
          $global:LASTEXITCODE = 0

      # Runs LAST of the run steps, and scans EVERY file under eval-out - the same
      # tree the upload takes, so the two cannot drift into "scanned a subset of
      # what we published". The artifact is public and, unlike the job log, not
      # secret-masked.
      - name: Refuse to upload a credential
        if: always()
        env:
          CLAUDE_CODE_OAUTH_TOKEN: ${{ (vars.CLAUDE_ACCOUNT == 'secondary' && secrets.CLAUDE_CODE_OAUTH_TOKEN_SECONDARY || secrets.CLAUDE_CODE_OAUTH_TOKEN) }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          HF_TOKEN: ${{ secrets.HUGGINGFACE_ACCESS_TOKEN }}
        run: |
          $ErrorActionPreference = "Stop"
          $files = @(Get-ChildItem eval-out -Recurse -File -ErrorAction SilentlyContinue)
          $secrets = @()
          foreach ($name in @("CLAUDE_CODE_OAUTH_TOKEN", "ANTHROPIC_API_KEY", "HF_TOKEN")) {
            $value = [Environment]::GetEnvironmentVariable($name)
            if ($value) { $secrets += $value }
          }
          $leaked = @()
          foreach ($file in $files) {
            if (-not (Test-Path -LiteralPath $file.FullName)) { continue }
            $hit = $false
            foreach ($value in $secrets) {
              if (Select-String -LiteralPath $file.FullName -SimpleMatch -Pattern $value -Quiet) { $hit = $true }
            }
            if (-not $hit) { continue }
            $leaked += $file.FullName
            # Redacted, not deleted. These files ARE the diagnostic - deleting the
            # log that explains the failure defeats the reason it is uploaded - and
            # the step still exits 1, so a real leak is never quiet.
            $text = [System.IO.File]::ReadAllText($file.FullName)
            foreach ($value in $secrets) { $text = $text.Replace($value, "***REDACTED***") }
            [System.IO.File]::WriteAllText($file.FullName, $text, (New-Object System.Text.UTF8Encoding($false)))
          }
          if ($leaked.Count -gt 0) {
            Write-Host "::error::A credential appeared in $($leaked.Count) file(s) bound for the PUBLIC artifact: $($leaked -join ', '). They were redacted in place before upload, but a secret reaching a trace or log at all means something is printing it - find that and fix it, then rotate the credential."
            exit 1
          }
          Write-Host "Scanned $($files.Count) file(s) under eval-out: no credential found."

      # Last step, always, so the aggregate can tell "this lane failed" from "this
      # lane never reported" even when everything above went wrong.
      - name: Record how this lane ended
        if: always()
        run: |
          # eval-out may not exist yet if the lane died before the eval step - this
          # file is how the aggregate tells "died early" from "had nothing to say".
          New-Item -ItemType Directory -Force -Path "eval-out" | Out-Null
          $status = @{
            lane = $env:EVAL_LANE
            job_status = "${{ job.status }}"
            embedder_ok = ($env:EMBEDDER_OK -ne 'false')
            runner = $env:COMPUTERNAME
            enforce = ($env:ENFORCE -eq 'true')
            # Whether the agent under test could remember anything, recorded next
            # to the scores so a memory result is never read without it.
            memory_enabled = ($env:GAIA_MEMORY_DISABLED -ne '1')
            memory_categories = $env:EVAL_MEMORY_CATEGORIES
            tier = $env:EVAL_TIER
            agent_model = $env:AGENT_MODEL
          }
          if ($env:EVAL_TARGETED -eq 'true') {
            $status.targeted = $true
            $status.selection = $env:EVAL_SELECTION
            $status.scenarios = $env:EVAL_SCENARIOS
          }
          $status = $status | ConvertTo-Json -Compress
          [System.IO.File]::WriteAllText("$PWD\eval-out\lane-status.json", $status, (New-Object System.Text.UTF8Encoding($false)))

      # Scores AND evidence, because this pool is ephemeral: the machine that
      # produced them is destroyed when the job ends, so "reproduce it locally"
      # is not advice anyone can follow for a lane that only fails in CI.
      - name: Upload this lane's scorecards, traces and logs
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: eval-${{ matrix.lane }}
          path: eval-out
          if-no-files-found: warn
          retention-days: 14

  # Nothing else sees the whole run. Each lane knows only its own categories, and
  # a lane whose machine died reports nothing at all - which on the Actions page
  # is a red X easy to read as "one flaky lane" rather than "those categories
  # were never measured". This job is the one place the run is described as a
  # whole.
  #
  # It REPORTS; it does not judge. Every lane already applied the enforce policy
  # to its own result, and re-deciding here would make one regression read as two.
  aggregate:
    name: Scenario eval - all lanes
    needs: [lanes, agent-eval, flagship-tasks]
    # Not on a run where the lanes were deliberately skipped (`[skip eval]` RC):
    # a table of empty lanes reads as "nothing was measured" when the truth is
    # "nothing was meant to be".
    if: >-
      always() && needs.lanes.result == 'success' &&
      needs.agent-eval.result != 'skipped'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v7

      - name: Download every lane's artifact
        uses: actions/download-artifact@v8
        with:
          pattern: eval-*
          path: lane-artifacts
        continue-on-error: true

      - name: Summarise the run
        env:
          OUTCOMES_RESULT: ${{ needs.flagship-tasks.result }}
          LANES_RESULT: ${{ needs.agent-eval.result }}
          TARGETED: ${{ needs.lanes.outputs.targeted }}
          MATRIX: ${{ needs.lanes.outputs.matrix }}
          SELECTION: ${{ needs.lanes.outputs.selection }}
          TIER: ${{ needs.lanes.outputs.tier }}
          AGENT_MODEL: ${{ needs.lanes.outputs.agent_model }}
          BASELINE_DIR: ${{ needs.lanes.outputs.baseline_dir }}
        run: |
          set -euo pipefail
          # The report covers the lanes this run's matrix held: tier machines
          # are not ci_lanes.json lanes, and the full sweep drops the backlog's
          # scenarios. A partial run (smoke, backlog, targeted) gets a banner.
          scope=(--matrix-json "${MATRIX}")
          if [ "${TARGETED}" = "true" ]; then
            scope+=(--selection "${SELECTION}")
          fi
          # Run as a file, not `-m gaia.eval.lane_summary`: this is a hosted runner
          # with no gaia install, and the module is stdlib-only by design.
          python src/gaia/eval/lane_summary.py \
            --lanes-file eval/ci_lanes.json \
            --artifacts-dir lane-artifacts \
            --baseline-dir "${BASELINE_DIR}" \
            "${scope[@]}" \
            | tee -a "$GITHUB_STEP_SUMMARY"
          {
            echo ""
            echo "| job | result |"
            echo "| --- | --- |"
            echo "| tier | \`${TIER}\` |"
            echo "| model under test | \`${AGENT_MODEL}\` (baseline \`${BASELINE_DIR}\`) |"
            echo "| scenario lanes (matrix) | \`${LANES_RESULT}\` |"
            echo "| outcomes (\`gaia eval tasks\`) | \`${OUTCOMES_RESULT}\` |"
            echo ""
            echo "Every lane uploads its transcripts, judge rationales and backend"
            echo "and Lemonade logs. The machines are ephemeral, so those artifacts"
            echo "are the only copy - download them before they expire (14 days)."
          } >> "$GITHUB_STEP_SUMMARY"

      # What this run was, kept 90 days. A final tag's verify-final job looks
      # for the `flagship-tier-full` one on its commit or its RCs' commits.
      - name: Record the tier
        env:
          TIER: ${{ needs.lanes.outputs.tier }}
          AGENT_MODEL: ${{ needs.lanes.outputs.agent_model }}
          LANES_RESULT: ${{ needs.agent-eval.result }}
          OUTCOMES_RESULT: ${{ needs.flagship-tasks.result }}
        run: |
          mkdir -p tier
          jq -n --arg tier "$TIER" --arg model "$AGENT_MODEL" --arg sha "$GITHUB_SHA" \
            --arg ref "$GITHUB_REF" --arg lanes "$LANES_RESULT" --arg outcomes "$OUTCOMES_RESULT" \
            '{tier: $tier, agent_model: $model, sha: $sha, ref: $ref, lanes: $lanes, outcomes: $outcomes}' \
            > tier/tier.json
          cat tier/tier.json
      - name: Upload the tier marker
        uses: actions/upload-artifact@v7
        with:
          name: flagship-tier-${{ needs.lanes.outputs.tier }}
          path: tier/tier.json
          retention-days: 90

  # The weekly backlog's whole point is its trend: one run of a scenario that
  # always fails says nothing new. Picks up the previous week's trend artifact,
  # appends this run, and re-uploads it, so the history rolls forward. A gap
  # longer than the 90-day retention starts a new trend, and says so.
  backlog-trend:
    name: Weekly backlog trend
    needs: [lanes, agent-eval]
    if: >-
      always() && needs.lanes.result == 'success' &&
      needs.lanes.outputs.tier == 'backlog' && needs.agent-eval.result != 'skipped'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      actions: read
    steps:
      - name: Checkout
        uses: actions/checkout@v7

      - name: Download the backlog lanes' artifacts
        uses: actions/download-artifact@v8
        with:
          pattern: eval-backlog-*
          path: lane-artifacts
        continue-on-error: true

      - name: Fetch the previous trend
        id: previous
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          set -euo pipefail
          # Newest first across the repo, this run's own not uploaded yet.
          found="$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts?name=flagship-backlog-trend&per_page=1" \
            --jq '.artifacts[0] | select(. != null) | "\(.workflow_run.id) \(.expired)"')"
          if [ -z "${found}" ]; then
            echo "::notice title=Backlog trend::No earlier trend artifact exists. This run starts the trend."
            echo "path=" >> "$GITHUB_OUTPUT"
            exit 0
          fi
          read -r run expired <<< "${found}"
          if [ "${expired}" = "true" ]; then
            echo "::warning title=Backlog trend::The last trend artifact (run ${run}) has expired, so the history before it is gone. This run starts a new trend."
            echo "path=" >> "$GITHUB_OUTPUT"
            exit 0
          fi
          gh run download "${run}" --repo "${GITHUB_REPOSITORY}" -n flagship-backlog-trend -D previous
          echo "path=previous/trend.json" >> "$GITHUB_OUTPUT"

      - name: Append this week and summarise
        env:
          PREVIOUS: ${{ steps.previous.outputs.path }}
          AGENT_MODEL: ${{ needs.lanes.outputs.agent_model }}
          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
        run: |
          set -euo pipefail
          previous=()
          if [ -n "${PREVIOUS}" ]; then previous=(--previous "${PREVIOUS}"); fi
          python util/eval_backlog_trend.py --artifacts-dir lane-artifacts \
            "${previous[@]}" --out trend/trend.json --run-url "${RUN_URL}" \
            --sha "${GITHUB_SHA}" --date "$(date -u +%F)" --model "${AGENT_MODEL}" \
            | tee -a "$GITHUB_STEP_SUMMARY"

      - name: Upload the trend
        if: always() && hashFiles('trend/trend.json') != ''
        uses: actions/upload-artifact@v7
        with:
          name: flagship-backlog-trend
          path: trend/trend.json
          retention-days: 90

  # A final tag does not re-run the eval: the release train promotes the RC
  # that was measured (#5492). This checks that measurement exists - a
  # full-sweep run (its `flagship-tier-full` marker) of this commit or of one of
  # its RCs' commits - and fails loudly when it does not. It does not judge the
  # results; that gate is #5497.
  verify-final:
    name: Final tag - a full sweep measured this release
    needs: [eval-gate]
    if: >-
      !cancelled() && needs.eval-gate.result == 'success' &&
      needs.eval-gate.outputs.run == 'true' && needs.eval-gate.outputs.tier == 'verify'
    runs-on: ubuntu-latest
    permissions:
      contents: read
      actions: read
    steps:
      - name: Checkout
        uses: actions/checkout@v7
      - name: Find the full sweep
        env:
          GH_TOKEN: ${{ github.token }}
        run: python util/eval_tiers.py verify-final

  # A release or RC tag run that does not pass files an issue, one per tag.
  # The nightly that used to catch environment drift (a Lemonade bump, a
  # re-quantised model, a runner-image change) is gone, so the tag run is now the
  # only recurring measurement - and its annotations would otherwise go unread.
  # It does not block publish.yml: the release proceeds, the issue records why
  # the eval did not pass. A dispatch does not file one; the dispatcher is
  # watching. A skipped RC (`[skip eval]`) files nothing.
  #
  # This job claims NO lane's concurrency group. A caller that holds the group
  # its callee needs deadlocks the callee before a job record exists, so
  # `needs.*.result` reads `failure` for a job that never ran and the issue filed
  # describes nothing. The now-deleted weekly caller did exactly that, every run
  # for ten days, filing issues about a job that never started.
  #
  # A job a tier deliberately leaves out reads `skipped` (outcomes on an RC
  # smoke, every lane on a final), so only `failure` and `cancelled` count.
  # A final tag files one when verify-final finds no full sweep.
  release-escalation:
    name: File an issue when the release eval does not pass
    needs: [eval-gate, lanes, agent-eval, flagship-tasks, aggregate, verify-final]
    if: >-
      always() && github.event_name == 'push' && github.ref_type == 'tag' &&
      needs.eval-gate.outputs.run == 'true' &&
      (contains(fromJSON('["failure", "cancelled"]'), needs.lanes.result) ||
      contains(fromJSON('["failure", "cancelled"]'), needs.agent-eval.result) ||
      contains(fromJSON('["failure", "cancelled"]'), needs.flagship-tasks.result) ||
      contains(fromJSON('["failure", "cancelled"]'), needs.verify-final.result))
    runs-on: ubuntu-latest
    permissions:
      contents: read
      issues: write
    steps:
      - name: File (or update) the issue for this tag
        env:
          GH_TOKEN: ${{ github.token }}
          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
          TAG: ${{ github.ref_name }}
          LANES_RESULT: ${{ needs.agent-eval.result }}
          OUTCOMES_RESULT: ${{ needs.flagship-tasks.result }}
          OUTCOMES_FAILED: ${{ needs.flagship-tasks.outputs.failed_suites }}
          TIER: ${{ needs.eval-gate.outputs.tier }}
          VERIFY_RESULT: ${{ needs.verify-final.result }}
        run: |
          set -euo pipefail
          # This job has no checkout, so every gh call names the repo; without it gh
          # dies with "not a git repository" and the alert is never filed.
          # DEDUP KEY - do not reword. The lookup below searches this exact title,
          # so a re-run of the same tag comments on its issue instead of filing another.
          TITLE="Flagship eval did not pass on ${TAG}"
          gh label create eval-failure --repo "$GITHUB_REPOSITORY" --color B60205 \
            --description "Automated eval failure" --force
          BODY="$(cat <<EOF
          The flagship eval on release tag \`${TAG}\` did not pass. Publishing is not blocked by it.

          - Run: ${RUN_URL}
          - Tier: \`${TIER}\` (eval/ci_tiers.json)
          - Scenario lanes: \`${LANES_RESULT}\`
          - Outcomes (\`gaia eval tasks\`): \`${OUTCOMES_RESULT}\`${OUTCOMES_FAILED:+ - gate failed for: ${OUTCOMES_FAILED}}
          - Final tag, full sweep found for this commit or an RC: \`${VERIFY_RESULT}\` (\`skipped\` on an RC)

          Start with the run's summary page: it names which lanes measured, which measured nothing, and
          how long each took. Then download the failing lane's \`eval-<lane>\` artifact - it carries the
          transcripts, judge rationales, backend log and Lemonade logs. Those machines are ephemeral and
          already destroyed, so the artifact is the only copy, and it expires in 14 days.

          Compare with the previous tag's run first. A failure no change since then explains usually means
          the environment moved, not the code: a Lemonade version bump, a re-quantised model, a driver or
          runner-image change.

          \`cancelled\` rather than \`failure\` means no measurement happened at all - a manual cancel, or a
          lane that waited out GitHub's 24h queue limit. Re-dispatch the workflow on the tag.
          EOF
          )"
          EXISTING="$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --label eval-failure \
            --search "${TITLE} in:title" --json number,title \
            | jq -r --arg t "${TITLE}" 'map(select(.title == $t)) | .[0].number // empty')"
          if [ -n "${EXISTING}" ]; then
            gh issue comment "${EXISTING}" --repo "$GITHUB_REPOSITORY" --body "${BODY}"
            echo "commented on existing issue #${EXISTING}"
          else
            gh issue create --repo "$GITHUB_REPOSITORY" --title "${TITLE}" --body "${BODY}" \
              --label eval-failure
          fi