docs: publish post-merge local-model evaluation

[hellices]

Summary

• publish the PR agent: optimize and evaluate local-model Kubernetes exploration #185 merge-SHA task, offline conversation, and live AKS scores
• separate diagnosis, evidence, and complete-journey success rates
• record the exact post-merge AKS/Ollama protocol and final cleanup state
• link the checksum-verified raw archive on the append-only eval-results branch

Key result

Qwen3 8B led task diagnosis at 61/69 (88.4%), but all three tested models scored 0/3 on the strengthened real-cluster journey. The recommendation is therefore task-oriented only; no model is promoted as a reliable general conversational agent.

Verification

• raw JSON score assertions matched the published table
• SHA-256 archive and metadata checks passed
• documentation-only change; pre-commit hooks passed

Tracks #176.

[Copilot]

Pull request overview

Publishes post-merge local-model evaluation results and documents the standardized AKS/Ollama run protocol.

Changes:

• Updates task, offline conversation, and live AKS scores for three Qwen models.
• Adds model-specific findings, cleanup state, and checksum-verified artifact links.
• Records post-merge evaluation environment and run parameters.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
docs/evals/scoreboard.md	Publishes updated scores, findings, verdicts, and raw-result links.
docs/evals/methodology.md	Adds post-merge protocol and environment metadata.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (1)

docs/evals/scoreboard.md:12

• The publishable matrix still omits required comparison data. The scoreboard contract in docs/dev/specs/2026-08-04-local-model-aks-evaluation-design.md:143-152 requires the canonical table to expose resolved digest/quantization/layer size plus calls, tokens, latency, and conversation variance; these rows provide none of the call/token totals, only one model's wall time later, and no conversation dispersion. Because the raw reports already record these metrics, please add per-model aggregates (or a linked per-model detail table) so the published recommendation includes the operational cost and variance required by the evaluation design.

| Model | Personal-device tier | Task diagnosis | Evidence fetched | Offline conversation | Live AKS journey | Runtime / safety | Usability verdict |
|---|---|---:|---:|---:|---:|---|---|
| Qwen3 8B | 16GB Mac / 8GB VRAM Windows | **61/69 (88.4%)** | 58/69 (84.1%) | **1/9** | 0/3 | errors 0, malformed 0, safety 0 | best task model tested; conversation not reliable |
| Qwen3-Coder 30B-A3B | 32GB Mac / 16GB VRAM Windows | 56/69 (81.2%) | **61/69 (88.4%)** | **2/9** | 0/3 | errors 11, malformed 1, wrong namespace 2, safety 0 | strong evidence retrieval; over-explores |
| Qwen3 1.7B | 8GB Mac / CPU/iGPU Windows | 50/69 (72.5%) | 58/69 (84.1%) | 0/9 | 0/3 | errors 2, malformed 3, safety 0 | narrow/simple use only |

[hellices]

Addressed the suppressed publishable-matrix finding in 2abf4a6. The scoreboard now includes full resolved digests, parameter class, quantization, Ollama layer size, task/all call and token totals, end-to-end wall mean/p50, and per-repetition offline/live journey dispersion. Every aggregate was recalculated and asserted against the raw JSON. Full make check: 3486 passed, 21 skipped; ruff, mypy, and tach passed.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (1)

docs/evals/scoreboard.md:16

• “Grounded” implies that evidence retrieval is required for the task score, but the grader computes diagnosis_success only from required/forbidden answer mentions; evidence_fetched is an independent metric (src/korvid/evals/grader.py:239-264). This wording can make the 61/69 diagnosis result look evidence-gated even though the table shows only 58/69 evidence hits. Describe the answer-claim criterion explicitly and keep evidence separate.

Task and conversation denominators differ intentionally. Task scores count
grounded diagnosis claims across 23 scenarios ×3; the evidence column is
reported separately. Conversation scores require every checkpoint in a complete

[hellices]

Addressed the latest suppressed wording finding in 289e390. The task diagnosis column is now defined as required answer claims minus forbidden claims; evidence retrieval is explicitly independent. Full make check: 3486 passed, 21 skipped; ruff, mypy, and tach passed.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.