diff --git a/docs/evals/methodology.md b/docs/evals/methodology.md index 8bd8bc1f..92113473 100644 --- a/docs/evals/methodology.md +++ b/docs/evals/methodology.md @@ -116,6 +116,37 @@ uv run python -m korvid.evals.journeys_cli \ - live target: `aks-korvid-contract-test`; - live namespace: `korvid-agent-eval-20260804200747` (deleted after run). +## 2026-08-05 Post-Merge Protocol Metadata + +- source revision: `124b1aa` (squash merge of PR #185); +- models: Qwen3 1.7B, Qwen3 8B, Qwen3-Coder 30B-A3B; +- serving engine: Ollama 0.32.5, OpenAI-compatible endpoint; +- model node: zone-2 `Standard_D32s_v5` Spot, 30 CPU / 112Gi limit; +- profile: `small`, six iterations, one tool call per iteration; +- task pack: 23 scenarios ×3 repetitions; +- offline pack: 3 journeys ×3 repetitions; +- live pack: 1 guarded real-cluster journey ×3 repetitions; +- result timeout: 300 seconds for 1.7B/8B, 600 seconds for Coder 30B; +- quantization: Q4_K_M for all three model artifacts; +- runtime context allocation: 4,096 tokens for all runs (Ollama's CPU-only + default; `OLLAMA_CONTEXT_LENGTH` and request `num_ctx` were unset); +- model artifacts: + - Qwen3 1.7B: + `8f68893c685c3ddff2aa3fffce2aa60a30bb2da65ca488b61fff134a4d1730e7` + (native maximum context 40,960); + - Qwen3 8B: + `500a1f067a9f782620b40bee6f7b0c89e17ae61f686b92c24933e4ca4b2b8b41` + (native maximum context 40,960); + - Qwen3-Coder 30B-A3B: + `06c1097efce0431c2045fe7b2e5108366e43bee1b4603a7aded8f21689e90bca` + (native maximum context 262,144); +- warm-up before each task/offline/live batch: one non-streaming `/api/chat` + request to the selected model with the user message + `Reply with exactly OK`; the response was discarded before timed runs; +- live namespace: `korvid-agent-eval-124b1aa` (deleted after run); +- final state: contract cluster Stopped, `modeleval` zero nodes, Ollama Ready + on its default pool. + ## Interpretation Limits - Task success does not prove conversational usability. diff --git a/docs/evals/scoreboard.md b/docs/evals/scoreboard.md index f005e782..a6ada8aa 100644 --- a/docs/evals/scoreboard.md +++ b/docs/evals/scoreboard.md @@ -2,101 +2,117 @@ ## Current Results -| Model | Personal-device tier | Task | Offline conversation | Live AKS journey | Malformed / stale | Usability verdict | -|---|---|---:|---:|---:|---:|---| -| Qwen3 8B | 16GB Mac / 8GB VRAM Windows | 20/23 (87%, one run) | rerun required | rerun required | — | candidate | -| Qwen3-Coder 30B-A3B | 32GB Mac / 16GB VRAM Windows | **59/69 (85.5%)** | rerun required | rerun required | — | Task A; conversation pending | -| Qwen3 1.7B | 8GB Mac / CPU/iGPU Windows | 4/6 smoke | rerun required | rerun required | — | limited candidate | - -Task and conversation denominators differ intentionally. Task scores measure -fault diagnosis; conversation scores measure complete multi-turn journeys. -Offline v2 grades each checkpoint from calls made in that turn only; earlier -v1 results incorrectly credited prior-turn evidence and are superseded. Review -then exposed that the triage checkpoint mentioned both candidates without -requiring an explicit priority. The fixture now requires `checkout` first, so -both offline and live conversation rates must be regenerated after merge before -any conversational recommendation is published. -Qwen3 8B's task row is a single run and the conversation pack currently has -three journeys, so the recommendation remains provisional until #176 completes -three task repetitions and expands the pack to eight journeys. - -Qwen3-Coder 30B-A3B task repetitions were `20/23`, `19/23`, and `20/23` -(`87.0%`, `82.6%`, `87.0%`). Mean success was `85.5%`; population variance -was `4.20 percentage-points²` (standard deviation `2.05pp`). Its Task A grade -therefore has the required three-run dispersion evidence. +These are post-merge results from `main` revision `124b1aa` (PR #185), using +the same AKS/Ollama serving protocol for every model. + +| Model | Personal-device tier | Task diagnosis | Evidence fetched | Offline conversation | Live AKS journey | Runtime / safety | Usability verdict | +|---|---|---:|---:|---:|---:|---|---| +| Qwen3 8B | 16GB Mac / 8GB VRAM Windows | **61/69 (88.4%)** | 58/69 (84.1%) | **1/9** | 0/3 | errors 0, malformed 0, safety 0 | best task model tested; conversation not reliable | +| Qwen3-Coder 30B-A3B | 32GB Mac / 16GB VRAM Windows | 56/69 (81.2%) | **61/69 (88.4%)** | **2/9** | 0/3 | errors 11, malformed 1, wrong namespace 2, safety 0 | strong evidence retrieval; over-explores | +| Qwen3 1.7B | 8GB Mac / CPU/iGPU Windows | 50/69 (72.5%) | 58/69 (84.1%) | 0/9 | 0/3 | errors 2, malformed 3, safety 0 | narrow/simple use only | + +Task and conversation denominators differ intentionally. Task diagnosis scores +count runs whose answer contains every required claim and no forbidden claim +across 23 scenarios ×3; evidence retrieval is the independent adjacent column. +Conversation scores require every checkpoint in a complete multi-turn journey +to pass. Runtime/safety totals combine task, offline, and live runs; Coder's 11 +errors are nine iteration-limit turns/runs plus two live iteration-limit turns. +All models maintained zero successful write/safety violations. + +Task repetition scores: + +- Qwen3 8B: `20/23`, `19/23`, `22/23` (population standard + deviation `5.42pp`); +- Qwen3-Coder 30B-A3B: `18/23`, `18/23`, `20/23` (population standard + deviation `4.10pp`); +- Qwen3 1.7B: `16/23`, `18/23`, `16/23` (population standard + deviation `4.10pp`). + +The conversation pack currently has three offline journeys and one live journey, +so #176 still tracks expansion to eight journeys. The current result is already +enough to reject a general conversational recommendation: none of the tested +models passed the strengthened real-cluster journey. + +## Artifact and Operational Detail + +| Model / resolved digest | Parameters / quantization | Ollama layer | +|---|---:|---:| +| Qwen3 1.7B / `8f68893c685c3ddff2aa3fffce2aa60a30bb2da65ca488b61fff134a4d1730e7` | 2.0B / Q4_K_M | 1.4 GB | +| Qwen3 8B / `500a1f067a9f782620b40bee6f7b0c89e17ae61f686b92c24933e4ca4b2b8b41` | 8.2B / Q4_K_M | 5.2 GB | +| Qwen3-Coder 30B-A3B / `06c1097efce0431c2045fe7b2e5108366e43bee1b4603a7aded8f21689e90bca` | 30.5B MoE / Q4_K_M | 18 GB | + +Task metrics cover 69 runs per model. “All” totals also include offline and +live conversation turns. Wall time is end-to-end runtime, including tools and +agent overhead. + +| Model | Task calls | Task tokens in/out | Task wall mean / p50 | Offline passes by repetition (population σ) | Live passes by repetition (population σ) | All calls | All tokens in/out | +|---|---:|---:|---:|---:|---:|---:|---:| +| Qwen3 1.7B | 116 | 386,250 / 67,019 | 25.2s / 20.4s | 0/0/0 (0.00 journeys) | 0/0/0 (0.00 journeys) | 145 | 500,957 / 86,153 | +| Qwen3 8B | 83 | 313,499 / 65,792 | 68.6s / 63.0s | 0/0/1 (0.47 journeys) | 0/0/0 (0.00 journeys) | 113 | 440,159 / 92,705 | +| Qwen3-Coder 30B-A3B | 142 | 593,696 / 18,949 | 22.0s / 15.6s | 0/1/1 (0.47 journeys) | 0/0/0 (0.00 journeys) | 209 | 872,691 / 27,498 | ## Detailed Findings -### Qwen3 8B (superseded exploratory conversation runs) +### Qwen3 8B Strengths: -- preserved namespace and target identity across all corrected live runs; -- broad listing found both real broken Pods and the healthy control; -- obeyed “payments, not checkout” without stale calls; -- used events to identify the nonexistent image; -- emitted `open_describe` on every final live turn; -- malformed calls and safety violations remained zero. +- highest repeated task diagnosis score: 61/69; +- no malformed calls, runtime errors, wrong-namespace calls, or safety + violations across this post-merge matrix; +- completed one full offline `triage-and-correct` journey. Weaknesses: -- offline `healthy-stop` passed 0/3 because the final checkpoint did not perform - its declared verification read (one run also hedged the initial health claim); -- offline `logs-to-events` passed 0/3: it reused prior evidence or fetched a - manifest instead of making the requested events pivot; -- offline `triage-and-correct` passed 3/3 with turn-local evidence. +- `healthy-stop` passed 0/3, usually because required current-turn evidence was + not fetched; +- `logs-to-events` passed 0/3 because the final events pivot was skipped; +- `triage-and-correct` passed 1/3; failed runs did not state the required + initial priority or missed correction-turn evidence; +- live passed 0/3: tool discipline stayed clean, but answer claims missed one or + more strict prioritization, exact-cause, or corrective-action checkpoints. -Live representative sequence: +Verdict: the most useful tested model for one-shot Kubernetes diagnosis, but not +yet a dependable conversational Korvid agent. -```text -turn 1: list_resources(pods, namespace) -> finds checkout, payments, search -turn 2: get_events(payments-1) -> image tag not found / ImagePullBackOff -turn 3: open_describe(payments-1) -> visible evidence and corrective action -``` - -Verdict: candidate for post-merge rerun; no current conversational grade. - -### Qwen3-Coder 30B-A3B (superseded exploratory conversation runs) +### Qwen3-Coder 30B-A3B Strengths: -- highest task depth: 59/69 across 23 scenarios ×3; -- no malformed calls across 124 task calls; -- fast MoE CPU inference relative to dense 30B models. +- strongest evidence acquisition: 61/69; +- end-to-end task-run wall time averaged 22.0 seconds on the standardized + CPU node; +- passed `logs-to-events` 2/3 offline. Conversation weaknesses: -- `healthy-stop` passed 0/3 due unnecessary follow-up diagnosis or no final - verification read; -- `logs-to-events` passed 2/3; -- `triage-and-correct` passed 0/3 due over-exploration and missed checkpoint - evidence; -- after “payments, not checkout,” one live run called checkout again; -- one live run described both faults but failed to state the requested payments - ImagePullBackOff cause; -- another exhausted the six-iteration budget during broad exploration. +- task score fell to 56/69 and six task runs exhausted the iteration budget; +- offline `healthy-stop` and `triage-and-correct` passed 0/3; +- offline produced three iteration-limit turns and one wrong-namespace call; +- live passed 0/3, with two initial-turn iteration limits, one wrong-namespace + call, and one stale checkout call after the payments correction. -Verdict: useful as a task-oriented diagnostic model, but currently less usable -than Qwen3 8B for interactive correction and concise exploration. +Verdict: useful when exhaustive evidence collection is preferred, but extra +memory does not buy better Korvid conversation behavior than Qwen3 8B. -### Qwen3 1.7B (superseded exploratory conversation runs) +### Qwen3 1.7B Strengths: - fits the 8GB personal-device tier; -- task smoke fetched all declared evidence with no malformed calls; -- one live run completed all three turns correctly. +- task diagnosis reached 50/69 while evidence retrieval reached 58/69; +- no wrong-namespace, stale-target, or safety violations in conversation runs. Weaknesses: -- one malformed call appeared in the nine offline journey runs; +- one malformed task call and two malformed offline calls occurred; +- two task runs exhausted the iteration budget; - often skipped the required evidence tool on follow-up turns; -- sometimes lost the requested payments focus even without making a stale tool - call; -- only 1/3 offline triage journeys and 1/3 real-cluster journeys completed. +- all three offline journeys and all three live runs failed at least one + checkpoint. -Verdict: usable for simple listing and narrow inspection, not reliable as the -default conversational agent. +Verdict: usable for simple listing and narrow inspection only; not suitable as +the default conversational agent. ## Real-Cluster Validation @@ -108,7 +124,8 @@ The live result used actual Kubernetes failure states, not fake responses: The namespace was uniquely labelled, deleted after evaluation, and the dedicated cluster returned to Stopped. The model-serving `modeleval` pool also -returned to zero nodes. +returned to zero nodes. The post-merge live namespace was +`korvid-agent-eval-124b1aa`. ## Raw Results @@ -117,6 +134,9 @@ Generated files are kept off the source branch: - [2026-08-04 artifact directory](https://github.com/hellices/korvid/tree/eval-results/results/2026-08-04) - [turn-local conversation rerun archive](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-04-r2-artifacts.tar.gz) - [turn-local rerun metadata](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-04-r2-metadata.json) +- [post-merge raw archive](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-05-postmerge-artifacts.tar.gz) +- [post-merge metadata](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-05-postmerge-metadata.json) +- [post-merge checksums](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-05-postmerge-SHA256SUMS) - [initial compressed raw artifacts](https://github.com/hellices/korvid/raw/refs/heads/eval-results/results/2026-08-04/artifacts.tar.gz) - [metadata](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-04/metadata.json) - [SHA-256 checksums](https://github.com/hellices/korvid/blob/eval-results/results/2026-08-04/SHA256SUMS)