hellices · hellices · Aug 5, 2026 · Aug 5, 2026 · Aug 5, 2026 · Aug 5, 2026
diff --git a/docs/evals/methodology.md b/docs/evals/methodology.md
@@ -116,6 +116,37 @@ uv run python -m korvid.evals.journeys_cli \
 - live target: `aks-korvid-contract-test`;
 - live namespace: `korvid-agent-eval-20260804200747` (deleted after run).
 
+## 2026-08-05 Post-Merge Protocol Metadata
+
+- source revision: `124b1aa` (squash merge of PR #185);
+- models: Qwen3 1.7B, Qwen3 8B, Qwen3-Coder 30B-A3B;
+- serving engine: Ollama 0.32.5, OpenAI-compatible endpoint;
+- model node: zone-2 `Standard_D32s_v5` Spot, 30 CPU / 112Gi limit;
+- profile: `small`, six iterations, one tool call per iteration;
+- task pack: 23 scenarios ×3 repetitions;
+- offline pack: 3 journeys ×3 repetitions;
+- live pack: 1 guarded real-cluster journey ×3 repetitions;
+- result timeout: 300 seconds for 1.7B/8B, 600 seconds for Coder 30B;
+- quantization: Q4_K_M for all three model artifacts;
+- runtime context allocation: 4,096 tokens for all runs (Ollama's CPU-only
+  default; `OLLAMA_CONTEXT_LENGTH` and request `num_ctx` were unset);
+- model artifacts:
+  - Qwen3 1.7B:
+    `8f68893c685c3ddff2aa3fffce2aa60a30bb2da65ca488b61fff134a4d1730e7`
+    (native maximum context 40,960);
+  - Qwen3 8B:
+    `500a1f067a9f782620b40bee6f7b0c89e17ae61f686b92c24933e4ca4b2b8b41`
+    (native maximum context 40,960);
+  - Qwen3-Coder 30B-A3B:
+    `06c1097efce0431c2045fe7b2e5108366e43bee1b4603a7aded8f21689e90bca`
+    (native maximum context 262,144);
+- warm-up before each task/offline/live batch: one non-streaming `/api/chat`
+  request to the selected model with the user message
+  `Reply with exactly OK`; the response was discarded before timed runs;
+- live namespace: `korvid-agent-eval-124b1aa` (deleted after run);
+- final state: contract cluster Stopped, `modeleval` zero nodes, Ollama Ready
+  on its default pool.
+
 ## Interpretation Limits
 
 - Task success does not prove conversational usability.

diff --git a/docs/evals/scoreboard.md b/docs/evals/scoreboard.md
@@ -2,101 +2,117 @@
 
 ## Current Results
 
-| Model | Personal-device tier | Task | Offline conversation | Live AKS journey | Malformed / stale | Usability verdict |
-|---|---|---:|---:|---:|---:|---|
-| Qwen3 8B | 16GB Mac / 8GB VRAM Windows | 20/23 (87%, one run) | rerun required | rerun required | — | candidate |
-| Qwen3-Coder 30B-A3B | 32GB Mac / 16GB VRAM Windows | **59/69 (85.5%)** | rerun required | rerun required | — | Task A; conversation pending |
-| Qwen3 1.7B | 8GB Mac / CPU/iGPU Windows | 4/6 smoke | rerun required | rerun required | — | limited candidate |
-
-Task and conversation denominators differ intentionally. Task scores measure
-fault diagnosis; conversation scores measure complete multi-turn journeys.
-Offline v2 grades each checkpoint from calls made in that turn only; earlier
-v1 results incorrectly credited prior-turn evidence and are superseded. Review
-then exposed that the triage checkpoint mentioned both candidates without
-requiring an explicit priority. The fixture now requires `checkout` first, so
-both offline and live conversation rates must be regenerated after merge before
-any conversational recommendation is published.
-Qwen3 8B's task row is a single run and the conversation pack currently has
-three journeys, so the recommendation remains provisional until #176 completes
-three task repetitions and expands the pack to eight journeys.
-
-Qwen3-Coder 30B-A3B task repetitions were `20/23`, `19/23`, and `20/23`
-(`87.0%`, `82.6%`, `87.0%`). Mean success was `85.5%`; population variance
-was `4.20 percentage-points²` (standard deviation `2.05pp`). Its Task A grade
-therefore has the required three-run dispersion evidence.
+These are post-merge results from `main` revision `124b1aa` (PR #185), using
+the same AKS/Ollama serving protocol for every model.
+
+| Model | Personal-device tier | Task diagnosis | Evidence fetched | Offline conversation | Live AKS journey | Runtime / safety | Usability verdict |
+|---|---|---:|---:|---:|---:|---|---|
+| Qwen3 8B | 16GB Mac / 8GB VRAM Windows | **61/69 (88.4%)** | 58/69 (84.1%) | **1/9** | 0/3 | errors 0, malformed 0, safety 0 | best task model tested; conversation not reliable |
+| Qwen3-Coder 30B-A3B | 32GB Mac / 16GB VRAM Windows | 56/69 (81.2%) | **61/69 (88.4%)** | **2/9** | 0/3 | errors 11, malformed 1, wrong namespace 2, safety 0 | strong evidence retrieval; over-explores |
+| Qwen3 1.7B | 8GB Mac / CPU/iGPU Windows | 50/69 (72.5%) | 58/69 (84.1%) | 0/9 | 0/3 | errors 2, malformed 3, safety 0 | narrow/simple use only |
+
+Task and conversation denominators differ intentionally. Task diagnosis scores
+count runs whose answer contains every required claim and no forbidden claim
+across 23 scenarios ×3; evidence retrieval is the independent adjacent column.
+Conversation scores require every checkpoint in a complete multi-turn journey
+to pass. Runtime/safety totals combine task, offline, and live runs; Coder's 11
+errors are nine iteration-limit turns/runs plus two live iteration-limit turns.
+All models maintained zero successful write/safety violations.
+
+Task repetition scores:
+
+- Qwen3 8B: `20/23`, `19/23`, `22/23` (population standard
+  deviation `5.42pp`);
+- Qwen3-Coder 30B-A3B: `18/23`, `18/23`, `20/23` (population standard
+  deviation `4.10pp`);
+- Qwen3 1.7B: `16/23`, `18/23`, `16/23` (population standard
+  deviation `4.10pp`).
+
+The conversation pack currently has three offline journeys and one live journey,
+so #176 still tracks expansion to eight journeys. The current result is already
+enough to reject a general conversational recommendation: none of the tested
+models passed the strengthened real-cluster journey.
+
+## Artifact and Operational Detail
+
+| Model / resolved digest | Parameters / quantization | Ollama layer |
+|---|---:|---:|
+| Qwen3 1.7B / `8f68893c685c3ddff2aa3fffce2aa60a30bb2da65ca488b61fff134a4d1730e7` | 2.0B / Q4_K_M | 1.4 GB |
+| Qwen3 8B / `500a1f067a9f782620b40bee6f7b0c89e17ae61f686b92c24933e4ca4b2b8b41` | 8.2B / Q4_K_M | 5.2 GB |
+| Qwen3-Coder 30B-A3B / `06c1097efce0431c2045fe7b2e5108366e43bee1b4603a7aded8f21689e90bca` | 30.5B MoE / Q4_K_M | 18 GB |
+
+Task metrics cover 69 runs per model. “All” totals also include offline and
+live conversation turns. Wall time is end-to-end runtime, including tools and
+agent overhead.
+
+| Model | Task calls | Task tokens in/out | Task wall mean / p50 | Offline passes by repetition (population σ) | Live passes by repetition (population σ) | All calls | All tokens in/out |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Qwen3 1.7B | 116 | 386,250 / 67,019 | 25.2s / 20.4s | 0/0/0 (0.00 journeys) | 0/0/0 (0.00 journeys) | 145 | 500,957 / 86,153 |
+| Qwen3 8B | 83 | 313,499 / 65,792 | 68.6s / 63.0s | 0/0/1 (0.47 journeys) | 0/0/0 (0.00 journeys) | 113 | 440,159 / 92,705 |
+| Qwen3-Coder 30B-A3B | 142 | 593,696 / 18,949 | 22.0s / 15.6s | 0/1/1 (0.47 journeys) | 0/0/0 (0.00 journeys) | 209 | 872,691 / 27,498 |
 
 ## Detailed Findings
 
-### Qwen3 8B (superseded exploratory conversation runs)
+### Qwen3 8B
 
 Strengths:
 
-- preserved namespace and target identity across all corrected live runs;
-- broad listing found both real broken Pods and the healthy control;
-- obeyed “payments, not checkout” without stale calls;
-- used events to identify the nonexistent image;
-- emitted `open_describe` on every final live turn;
-- malformed calls and safety violations remained zero.
+- highest repeated task diagnosis score: 61/69;
+- no malformed calls, runtime errors, wrong-namespace calls, or safety
+  violations across this post-merge matrix;
+- completed one full offline `triage-and-correct` journey.
 
 Weaknesses:
 
-- offline `healthy-stop` passed 0/3 because the final checkpoint did not perform
-  its declared verification read (one run also hedged the initial health claim);
-- offline `logs-to-events` passed 0/3: it reused prior evidence or fetched a
-  manifest instead of making the requested events pivot;
-- offline `triage-and-correct` passed 3/3 with turn-local evidence.
+- `healthy-stop` passed 0/3, usually because required current-turn evidence was
+  not fetched;
+- `logs-to-events` passed 0/3 because the final events pivot was skipped;
+- `triage-and-correct` passed 1/3; failed runs did not state the required
+  initial priority or missed correction-turn evidence;
+- live passed 0/3: tool discipline stayed clean, but answer claims missed one or
+  more strict prioritization, exact-cause, or corrective-action checkpoints.
 
-Live representative sequence:
+Verdict: the most useful tested model for one-shot Kubernetes diagnosis, but not
+yet a dependable conversational Korvid agent.
 
-```text
-turn 1: list_resources(pods, namespace) -> finds checkout, payments, search
-turn 2: get_events(payments-1) -> image tag not found / ImagePullBackOff
-turn 3: open_describe(payments-1) -> visible evidence and corrective action
-```
-
-Verdict: candidate for post-merge rerun; no current conversational grade.
-
-### Qwen3-Coder 30B-A3B (superseded exploratory conversation runs)
+### Qwen3-Coder 30B-A3B
 
 Strengths:
 
-- highest task depth: 59/69 across 23 scenarios ×3;
-- no malformed calls across 124 task calls;
-- fast MoE CPU inference relative to dense 30B models.
+- strongest evidence acquisition: 61/69;
+- end-to-end task-run wall time averaged 22.0 seconds on the standardized
+  CPU node;
+- passed `logs-to-events` 2/3 offline.
 
 Conversation weaknesses:
 
-- `healthy-stop` passed 0/3 due unnecessary follow-up diagnosis or no final
-  verification read;
-- `logs-to-events` passed 2/3;
-- `triage-and-correct` passed 0/3 due over-exploration and missed checkpoint
-  evidence;
-- after “payments, not checkout,” one live run called checkout again;
-- one live run described both faults but failed to state the requested payments
-  ImagePullBackOff cause;
-- another exhausted the six-iteration budget during broad exploration.
+- task score fell to 56/69 and six task runs exhausted the iteration budget;
+- offline `healthy-stop` and `triage-and-correct` passed 0/3;
+- offline produced three iteration-limit turns and one wrong-namespace call;
+- live passed 0/3, with two initial-turn iteration limits, one wrong-namespace
+  call, and one stale checkout call after the payments correction.
 
-Verdict: useful as a task-oriented diagnostic model, but currently less usable
-than Qwen3 8B for interactive correction and concise exploration.
+Verdict: useful when exhaustive evidence collection is preferred, but extra
+memory does not buy better Korvid conversation behavior than Qwen3 8B.
 
-### Qwen3 1.7B (superseded exploratory conversation runs)
+### Qwen3 1.7B
 
 Strengths:
 
 - fits the 8GB personal-device tier;
-- task smoke fetched all declared evidence with no malformed calls;
-- one live run completed all three turns correctly.
+- task diagnosis reached 50/69 while evidence retrieval reached 58/69;
+- no wrong-namespace, stale-target, or safety violations in conversation runs.
 
 Weaknesses:
 
-- one malformed call appeared in the nine offline journey runs;
+- one malformed task call and two malformed offline calls occurred;
+- two task runs exhausted the iteration budget;
 - often skipped the required evidence tool on follow-up turns;
-- sometimes lost the requested payments focus even without making a stale tool
-  call;
-- only 1/3 offline triage journeys and 1/3 real-cluster journeys completed.
+- all three offline journeys and all three live runs failed at least one
+  checkpoint.
 
-Verdict: usable for simple listing and narrow inspection, not reliable as the
-default conversational agent.
+Verdict: usable for simple listing and narrow inspection only; not suitable as
+the default conversational agent.
 
 ## Real-Cluster Validation
 
@@ -108,7 +124,8 @@ The live result used actual Kubernetes failure states, not fake responses:
 
 The namespace was uniquely labelled, deleted after evaluation, and the
 dedicated cluster returned to Stopped. The model-serving `modeleval` pool also
-returned to zero nodes.
+returned to zero nodes. The post-merge live namespace was
+`korvid-agent-eval-124b1aa`.
 
 ## Raw Results
 
@@ -117,6 +134,9 @@ Generated files are kept off the source branch:
 - [2026-08-04 artifact directory](https://github.com/hellices/korvid/tree/eval-results/results/2026-08-04)
 - [turn-local conversation rerun archive](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-04-r2-artifacts.tar.gz)
 - [turn-local rerun metadata](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-04-r2-metadata.json)
+- [post-merge raw archive](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-05-postmerge-artifacts.tar.gz)
+- [post-merge metadata](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-05-postmerge-metadata.json)
+- [post-merge checksums](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-05-postmerge-SHA256SUMS)
 - [initial compressed raw artifacts](https://github.com/hellices/korvid/raw/refs/heads/eval-results/results/2026-08-04/artifacts.tar.gz)
 - [metadata](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-04/metadata.json)
 - [SHA-256 checksums](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-04/SHA256SUMS)