docs: publish the 2026-08-10 nine-model matrix

[hellices]

Publishes the measured results of the #176 campaign to docs/evals/scoreboard.md, following the retention convention already documented in methodology.md: human-readable scores in the repo, raw artifacts on eval-results, summaries on the issue.

Nine models, one revision, one tool surface

The three models from 2026-08-05 were re-measured rather than carried over. They had originally run before diagnose_service and diagnose_pvc existed (#213, #216) — a 14-tool surface against this campaign's 16 — so merging them into one ranking would have compared different tests.

Model	Task (23 scen.)
Qwen3 4B (2.5 GB)	94.2%
Qwen3 8B (5.2 GB)	92.8%
Qwen3 32B (20 GB)	89.9%
Qwen3 30B-A3B (18 GB)	87.0%
Qwen3 14B (9.3 GB)	84.1%
Qwen3-Coder 30B-A3B (18 GB)	81.2%
Qwen3 1.7B (1.4 GB)	79.7%
Devstral 24B (14 GB)	68.1%
Mistral Small 3.1 (15 GB)	58.0%

The 23-scenario denominator excludes the two scenarios added in #227, so old and new rows are scored on the same set.

Findings

Parameter count does not predict accuracy. The top two are the smallest two Qwen3 models tested. The 4B fetches the same evidence as the top three and does it in 100 minutes against the 32B's 367.

Retrieval and diagnosis are separate abilities. Qwen3-Coder 30B-A3B retrieves the most evidence of any model (65/75) and lands sixth on accuracy.

No model reliably reports a healthy Service. Best result across nine models is 2/3; six score 0/3. The same shape appears independently in the journeys (healthy-stop) and in the PVC negative control. healthy-restart-history is 3/3 for four models, so this is specific to concluding a Service is fine — not "healthy scenarios are hard".

Safety held under a real test. 675 task runs, 8 write attempts, 0 violations. Mistral Small reached for a mutation unprompted mid-diagnosis and the gate refused it.

Partial evidence for #221. None of the three re-measured models regressed on the larger tool surface; the 1.7B gained 7.2pp, clearing its 4.10pp σ. Not the controlled experiment #221 needs, but it argues against the concern that two extra tools would degrade small-model selection.

Operational notes recorded for re-runs

• ollama ships with a 10 GiB limit; the 30B models OOMKill on a 120 GiB node and the symptom presents as connection errors.
• modeleval is a Spot pool; a reclamation destroyed one full run and contaminated another (61.3% contaminated vs 66.7% clean).
• Parallel execution is counter-productive: a 3-way run raised CPU from 15.8 to 28 cores and delivered 2.5× worse throughput (memory-bandwidth contention). All published wall times are sequential and comparable.

Raw artifacts: eval-results branch, results/campaign-20260810-artifacts.tar.gz (1280035).

Refs #176, #221

[Copilot]

Pull request overview

Publishes the nine-model evaluation matrix and supporting operational findings.

Changes:

• Adds comparable 23/25-scenario results for nine models.
• Documents findings, rerun guidance, and retained artifacts.
• Preserves prior campaign details for comparison.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (4)

docs/evals/scoreboard.md:98

• This sentence reintroduces the contradiction immediately above: lines 80–92 show the weakness across Services, PVCs, and Pods and explicitly conclude that it is scenario-dependent. Calling it Service-specific makes the published finding internally inconsistent.

The same shape appears independently in the journeys (`healthy-stop` fails for
both the 30B and the 8B) and in `pvc-wait-for-first-consumer`, where the 14B
scores 1/3 and Mistral Small 0/3 by inventing a provisioning fault. Note that
`healthy-restart-history` is 3/3 for four models, so this is not "healthy
scenarios are hard" — it is specific to concluding a *Service* is fine.

docs/evals/scoreboard.md:41

• The bundled pack at bdfb645 contains four journeys, while this denominator covers only the older three; the campaign notes confirm that rollout-owner-chain was not run. Name the three included journeys and the exclusion so readers can reproduce and interpret the 2/3 results instead of assuming all bundled journeys were evaluated.

Journey pack (3 journeys, 7 turns, **1 repetition** — directional only):

tests/evals/test_scenarios_pack.py:268

• This filter derives valid prefixes from the files that still exist, so it misses the deletion/rename case the test claims to guard. For example, if rollout-owner-chain.yaml is removed, rollout disappears from known and the stale scoreboard citation is filtered out, making the test pass. Use an independently stable marker/registry for pack citations rather than deriving the candidate set solely from the current fixtures.

    cited = {
        token
        for token in re.findall(r"`([a-z][a-z0-9]*(?:-[a-z0-9]+)+)`", text)
        if token.split("-")[0] in {s.split("-")[0] for s in known}
    }

docs/evals/scoreboard.md:7

• The new matrix is not pinned to the publishable-row protocol required by #176. This records the revision, SKU, profile, and prompt source, but not the serving engine/version, quantization or resolved model digests, context length, or warm-up procedure; the retained campaign metadata also omits those fields, and methodology.md documents them only for the 2026-08-05 run. Without the 2026-08-10 values, model tags or serving defaults can change and this matrix cannot be reproduced under the same conditions.

Nine models on the task pack ×3, run on `main` revision `bdfb645` under the
standard AKS/Ollama protocol (`Standard_D32s_v5`, `small` profile,
`prompts.source: default`).

[hellices]

Second round — all four suppressed findings were checked and all four were real. Fixed in 3903ba3.

scoreboard.md:98 — contradiction still present. Correct, and this was the most serious of the four: my previous commit added the per-kind table but did not delete the paragraph it replaced, so the page argued both sides two paragraphs apart. Paragraph removed; the table carries the point.

scoreboard.md:41 — journey denominator. Correct. The pack ships four journeys at bdfb645 but rollout-owner-chain was added by #228 after the campaign, so it was never run. The three that ran are now named inline and the exclusion is stated with its reason.

test_scenarios_pack.py:268 — guard blind to deletion. Correct, and demonstrated rather than assumed: I moved rollout-owner-chain.yaml out of the pack and the test still passed, exactly as described. The candidate set is now fail-closed — an explicit _NON_PACK_CITATIONS allowlist of the five non-pack tokens instead of inferring pack-ness from surviving fixtures. Repeating the same deletion now fails with scoreboard cites pack ids that do not exist: ['rollout-owner-chain']. Added test_cited_pack_ids_reports_a_deleted_fixture to pin the behaviour directly.

scoreboard.md:7 — reproducibility fields. Correct, and worse than the comment assumed: the deployment runs ollama/ollama:latest, a floating tag, and the resolved version was not captured. The node is scaled to zero, so digests and context length cannot be recovered without re-provisioning. Fabricating them would be worse than omitting them, so the page now states explicitly what is pinned (revision, tool surface, SKU, profile, full prompt fingerprint) and what is not (engine version, digests, quantization, context length, no warm-up), and warns that rows are comparable to each other rather than re-servable on a later :latest. Pinning the engine tag is a real gap; I will raise it separately against #176 rather than silently change serving config on a merged results page.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (3)

docs/evals/scoreboard.md:15

• These rows do not meet #176's stated standard for a publishable scoreboard row. The canonical tracker requires the serving version, quantization, context length, scenario SHA, and warm-up to be pinned, and requires both task and journey packs to run at least three times with mean and variance published. Here those serving pins are explicitly absent, and the journey section reports one repetition for only three models. Please either recover/re-run the required protocol or label this matrix provisional/non-standard instead of presenting it as the current standard-protocol ranking.

recorded in every retained run). *Not* pinned: the serving
engine — the deployment runs `ollama/ollama:latest` and the resolved version
was not captured — nor per-model digests, quantization, or context length; all
were left at the tag and serving defaults, and no warm-up was performed. The

docs/evals/scoreboard.md:122

• The score gap establishes that retrieval and diagnosis are separate, but it does not show that prompt changes are unlikely to help. evidence_fetched only records whether all expected tool evidence was obtained; prompts can still affect how the model interprets that evidence. State the supported conclusion without ruling out an unmeasured intervention.

**Evidence retrieval and diagnosis are separable.** Mistral Small fetches 58/75
but scores 40/75 — the widest gap measured. It reaches the right data and draws
the wrong conclusion, which tool or prompt changes are unlikely to fix.

tests/evals/test_scenarios_pack.py:298

• This comment says citations must share a stem with a shipped fixture, but _cited_pack_ids deliberately does not inspect shipped fixtures so deleted citations remain detectable. Update the comment to describe the actual lowercase/hyphenated-token plus explicit-exclusion rule.

    # Only check identifiers shaped like a pack id: lowercase, hyphenated,
    # and sharing a stem with something we ship. Prose, pod names and
    # branch names are not pack ids.

[hellices]

Third round — all three suppressed findings checked, all three real. Fixed in ed227f1.

scoreboard.md:15 — rows do not meet #176's publishable standard. Upheld, and it was the right thing to catch. I re-read the protocol in #176 (lines 104–110): seven requirements. This matrix meets five and misses two.

One of the two was recoverable for free. #176 requires "publish mean and variance from raw JSON" and the table published only totals — but the retained runs carry per-run grades, so σ was recomputed from the raw JSON without touching hardware and is now a column. All nine totals reconcile with the published scores (this also caught me reading a Spot-contaminated Devstral file: 46/75 vs the published 50/75 — the clean re-run is the one on the page).

The other two gaps are not recoverable: the serving pins are gone with the node (#235), and the journey pack ran once. So instead of presenting the whole matrix as standard-protocol, the page now enumerates which five requirements are met, marks the journey column provisional, and names what closes each gap.

Publishing σ then exposed a further problem you did not flag: the prose ranking contradicted the new column. The 4B-over-8B (1.4 pp against σ 4.10 pp) and 8B-over-32B (2.9 pp) orderings are inside their spreads. Only 4B-over-32B separates (4.3 pp against 2.05 pp on both). The paragraph now states this rather than implying the exact order reproduces.

scoreboard.md:122 — Mistral claim rules out an unmeasured intervention. Correct. evidence_fetched is a boolean over expected evidence; it says nothing about interpretation, so "prompt changes are unlikely to fix" was not a measured claim. Narrowed to what the gap does support — better retrieval alone would not help — and the prompt question is now explicitly marked untested.

test_scenarios_pack.py:298 — stale comment. Correct; the previous commit replaced the prefix-matching rule with fail-closed filtering and left the comment describing the old behaviour. Removed — the _NON_PACK_CITATIONS docstring already states the rule, and a second description would just drift again.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

docs/evals/scoreboard.md:103

• Per-repetition standard deviation is descriptive; with only three repetitions it does not establish pairwise statistical separation. In fact, the displayed 4.3 pp mean gap versus two 2.05 pp standard deviations is not by itself a significance test, so saying the data “actually separates” 4B and 32B overstates this campaign. Report the observed mean gap while keeping the ordering explicitly directional.

Read the σ column before ranking adjacent rows. The 4B clears the 32B by
4.3 pp against a 2.05 pp spread on both, which is the one size-inversion the
data actually separates. The 4B-over-8B ordering (1.4 pp, 8B σ 4.10 pp) and
the 8B-over-32B ordering (2.9 pp) are **within noise** — the supported claim
is that the small models are not beaten by the large ones, not that this exact

docs/evals/scoreboard.md:17

• #176 does not define seven protocol requirements in this way: it has seven individual pinning fields, of which this campaign meets only profile, scenario SHA, and node SKU, while engine/version, quantization, context length, and warm-up are all missing. Journey ×3 is a separate unmet condition. The “meets five / misses two” count therefore overstates conformance; enumerate the met and missing conditions without that count.

**Protocol conformance (#176).** The standard for a publishable row has seven
requirements. This matrix meets five — shared AKS environment, profile, node
SKU, scenario set fixed by revision `bdfb645`, task pack ×3 with mean and
variance published. It does **not** meet two: the serving engine, quantization,
context length and warm-up are unpinned (below), and the journey pack ran at

[hellices]

Fourth round — both findings upheld and fixed in c692f0b. Both were overstatements I introduced in the previous commit while trying to be more precise, which is worth noting: the round-3 fix over-corrected.

scoreboard.md:17 — "meets five / misses two" overstates conformance. Correct, and the framing was mine, not #176's. #176 lists seven pinning fields; this campaign meets three (profile, scenario SHA, node SKU) and misses four (engine/version, quantization, context length, warm-up), with journey ×3 a separate unmet condition. Replaced the count with a per-condition table and stated plainly that this is not a fully standard-protocol run.

scoreboard.md:103 — σ over three repetitions is not a significance test. Correct. Saying the 4B-over-32B gap is the inversion the data "actually separates" claimed a test I never ran. All orderings are now labelled directional, with observed mean gaps reported alongside the spreads and an explicit note that three repetitions cannot establish pairwise significance.

Checking the rest of the page for the same error found a third instance neither round flagged: the re-measurement table's "published σ" column carried the 2026-08-05 values, and now that a differently-sourced σ column exists in the main table, the two read as the same measurement. They are not, and for the 1.7B and 8B they are effectively transposed between campaigns. Split into "σ then" / "σ now" and softened the same separation language there.

Rounds 3 and 4 both produced only suppressed low-confidence findings with no unresolved blocking issues, so per the repository's review-loop policy I am not requesting a further round. Proceeding to merge once the required checks are green.