agent: optimize and evaluate local-model Kubernetes exploration

docs/agent.md

@@ -287,3 +287,24 @@ Custom scenario packs can be pointed at with `--scenarios DIR`, and
 descriptions, 6-iteration budget, small system prompt — see
 [Capability profiles](#capability-profiles)) so before/after numbers for a
 small model come from the same pack.
+
+### Conversational journeys
+
+`korvid.evals.journeys_cli` keeps one runtime alive across multiple user turns,
+measuring broad discovery, corrections, evidence pivots, stopping behavior, and
+UI intent:
+
+```sh
+export KORVID_EVAL_BASE_URL=http://localhost:11434/v1
+export KORVID_EVAL_MODEL=qwen3:8b
+export KORVID_EVAL_TIMEOUT_SECONDS=300
+
+uv run python -m korvid.evals.journeys_cli \
+  --profile small --reps 3 \
+  --out journeys.md --json journeys.json
+```
+
+The guarded `--live` mode reads real faults only from the dedicated
+`aks-korvid-contract-test` context and a namespace beginning with
+`korvid-agent-eval-`. See [evaluation methodology](evals/methodology.md),
+[scenario catalog](evals/scenarios.md), and the [model scoreboard](evals/scoreboard.md).

docs/dev/specs/2026-08-04-local-model-aks-evaluation-design.md

@@ -0,0 +1,178 @@
+# Local Model AKS Evaluation Design
+
+## Goal
+
+Evaluate models that individuals can install and use routinely on
+Apple-silicon MacBooks or consumer Windows PCs under one standardized AKS
+serving protocol. Publish task-diagnostic
+and conversational-exploration scores separately, assign practical grades, and
+recommend models by MacBook unified-memory tier.
+
+## Scope
+
+The model is the comparison unit. Developer laptop versus AKS is not a
+leaderboard dimension. Laptop runs may guide iteration, but a model receives a
+published score only after it runs on the shared AKS evaluation environment.
+
+The primary matrix covers practical personal-device tiers:
+
+| Mac unified memory | Windows baseline | Candidate class |
+|---:|---|---|
+| 8GB | CPU/iGPU, 8GB system RAM | 0.6B–3B; 4B only with short context |
+| 16GB | 8GB VRAM and 16GB+ system RAM | 4B–8B |
+| 24GB | 12GB VRAM and 24GB+ system RAM | 8B–14B |
+| 32GB | 16GB VRAM and 32GB+ system RAM | 14B–24B |
+| 64GB | 24GB VRAM and 48GB+ system RAM | 30B MoE or 32B dense |
+
+The tiers are practical minimum recommendations, not claims that the model
+weights are the only memory consumer. Ollama, KV cache, context length, and the
+operating system need headroom.
+
+Primary candidates are Qwen3 0.6B/1.7B/4B/8B/14B/30B-A3B, Llama 3.2
+1B/3B, Granite 3.2 2B, Phi-4 Mini, Llama 3.1 8B, Mistral Small 3.1,
+Devstral, and Qwen3-Coder 30B-A3B. The 8GB smoke result promotes Qwen3
+1.7B (4/6 with all evidence fetched) and conditionally Qwen3 4B when context
+is kept short. Qwen3 0.6B, Llama 3.2 1B/3B, Granite 3.2 2B, and Phi-4 Mini
+do not reliably complete evidence-driven tool tasks in the current profile.
+Qwen3 32B is an enthusiast reference. Llama 3.3 70B is excluded from
+personal-device recommendations and retained only as a server-class comparison.
+
+## Evaluation Strategy
+
+Use a two-stage promotion gate.
+
+### Stage 1: smoke matrix
+
+Every candidate runs:
+
+- six task scenarios covering pod failure, scheduling, storage, rollout,
+  healthy negative control, and safety-sensitive behavior;
+- three conversational journeys covering broad discovery, user correction,
+  and UI-follow/evidence narrowing;
+- one repetition after an explicit warm-up request.
+
+A candidate advances when it achieves all of:
+
+- at least 4/6 task scenarios;
+- at least 2/3 conversational journeys;
+- zero malformed calls;
+- zero safety violations;
+- native structured tool calls that the runtime can parse.
+
+### Stage 2: publishable matrix
+
+Promoted candidates run:
+
+- all 23 task scenarios, three repetitions;
+- the initial three conversational journeys, three repetitions;
+- the same profile, prompt SHA, scenario SHA, serving engine, quantization,
+  context length, and warm-up procedure.
+
+Raw JSON records every run. Generated documentation contains means and
+variance; it does not copy ad hoc numbers from issue comments.
+
+The initial three-journey pack validates broad triage/correction, evidence
+pivoting, and healthy stopping. Expanding it to eight journeys (including
+ownership traversal, cross-namespace comparison, and RBAC-limited exploration)
+remains follow-up work tracked by #176; scores in this PR are an initial,
+explicitly partial conversation benchmark.
+
+## Conversational Journey Evaluation
+
+Task scenarios remain one-turn diagnostic checks. A separate journey format
+models the interactive product:
+
+1. broad namespace triage and candidate discovery;
+2. compare abnormalities and prioritize one;
+3. correct the target mid-investigation;
+4. preserve name/namespace identity across turns;
+5. pivot from logs to events or manifests when logs are insufficient;
+6. traverse workload ownership;
+7. mirror evidence in the TUI;
+8. verify a healthy environment and stop.
+
+Each journey uses deterministic fake-cluster state and scripted user turns.
+Checkpoints grade evidence, target identity, conversational pivots, redundant
+calls, clarification quality, TUI-follow behavior, and the terminal answer.
+Safety and malformed-call invariants remain identical to task evaluation.
+
+## Grades
+
+| Grade | Task success | Conversation success | Required invariants |
+|---|---:|---:|---|
+| S | at least 90% | at least 80% | malformed below 1%; safety violations 0 |
+| A | at least 80% | at least 70% | malformed below 1%; safety violations 0 |
+| B | at least 65% | at least 55% | malformed below 1%; safety violations 0 |
+| C | below B | or below B | no safety exception |
+
+A model with a safety violation is ungraded and not recommended regardless of
+answer quality.
+
+## AKS Serving Environment
+
+Use the existing `aks-shared-runners` cluster and `ollama/ollama` deployment.
+
+- Expand `ollama-models` from 30GiB to 200GiB. The `default` StorageClass uses
+  Azure Disk CSI and supports online expansion.
+- Add a retained zone-2 `Standard_D32s_v5` Spot user node pool named
+  `modeleval`. Zone 2 is required by the existing model PVC.
+- Keep the node-pool resource after the evaluation track. Scale it to one node
+  during runs and zero while idle; do not delete it after each run.
+- Bind Ollama to the current `modeleval` node explicitly while evaluating.
+  The cluster uses Node Auto Provisioning, whose scheduler rejected values
+  from a static node-pool selector during the initial experiment.
+- Restore Ollama to its original scheduling and CPU/memory settings before
+  scaling the evaluation pool to zero.
+- Use `kubectl port-forward`; the existing public LoadBalancer is not reachable
+  from the development network.
+
+GPU T4 SKUs are present in Korea Central, but the subscription's
+`Standard NCASv3_T4 Family` quota is zero and an automatic quota request
+returned `QuotaNotAvailableForResource`. A future approved GPU quota replaces
+the CPU pool without changing the scoring protocol.
+
+## Model Storage
+
+The 200GiB PVC can retain the initial matrix. Pull candidates serially and
+record the resolved Ollama digest and model-layer size. If total storage
+approaches 85%, remove only candidates that failed the smoke gate; retain
+published and pending candidates.
+
+## Scoreboard
+
+The canonical table contains:
+
+- model and resolved digest;
+- parameter class and quantization;
+- Ollama layer size;
+- practical MacBook unified-memory tier;
+- task score and variance;
+- conversation score and variance;
+- evidence, malformed, safety, calls, tokens, and latency;
+- grade and recommendation.
+
+The environment appears once in the evaluation-protocol header, not as a row
+dimension.
+
+## Error Handling
+
+- A cold-load timeout is not a model-quality failure. Warm-up is mandatory and
+  timed separately.
+- A model that cannot emit parseable native tool calls fails the smoke gate
+  with the reason recorded.
+- Spot eviction marks the run invalid and resumable; it does not count as a
+  model failure.
+- Endpoint, port-forward, or node failures produce nonzero runner exit status
+  and never create a publishable row.
+
+## Acceptance Criteria
+
+- All listed candidates receive a smoke result on AKS.
+- Every published task grade uses three task-pack runs.
+- Initial conversation results use three journeys × three repetitions and stay
+  labelled partial until #176 expands the pack to eight journeys.
+- Results include practical 8GB, 16GB, 24GB, 32GB, and 64GB MacBook tiers and
+  corresponding consumer Windows VRAM/system-memory guidance.
+- The node pool remains available and can scale to zero after evaluation.
+- The scoreboard is generated from raw results and issue #176 is its single
+  tracking location.

docs/evals/methodology.md

@@ -0,0 +1,138 @@
+# Agent Evaluation Methodology
+
+## Purpose
+
+Korvid evaluates whether a locally installable model can use Kubernetes tools
+to explore evidence, maintain conversational context, and reach a safe,
+grounded conclusion. Model quality and serving hardware are separate:
+publishable model comparisons use the same AKS serving protocol.
+
+## Three Evaluation Tiers
+
+### 1. Task diagnostics
+
+The 23 YAML scenarios under `src/korvid/evals/scenarios/` test one user question
+against a deterministic fake cluster. The model and Korvid runtime are live;
+only Kubernetes responses are fixtures.
+
+Each run records:
+
+- required and forbidden claims in the final answer;
+- whether the model fetched the declared ground-truth evidence;
+- resolvable and on-target calls;
+- malformed calls, write attempts, and safety violations;
+- iterations, tokens, and wall time.
+
+### 2. Offline conversational journeys
+
+The YAML files under `src/korvid/evals/journeys/` share one fake cluster across
+multiple scripted user turns. One `AgentRuntime` persists for the complete
+conversation, so history, corrections, stale targets, and stopping behavior are
+real.
+
+Each turn grades:
+
+- answer claims and evidence fetched during that turn (prior-turn evidence
+  cannot satisfy a later checkpoint);
+- calls made during that turn;
+- forbidden stale targets after a user correction;
+- maximum useful-call budget;
+- UI intent such as `open_describe`;
+- malformed calls and runtime errors.
+
+The current pack has three journeys. It is an initial benchmark, not the final
+eight-journey coverage target tracked in #176.
+
+### 3. Live AKS journeys
+
+The live journey targets actual Kubernetes resources in the dedicated
+`aks-korvid-contract-test` cluster. Model serving remains isolated in
+`aks-shared-runners/ollama`.
+
+Safety controls:
+
+- context must be exactly `aks-korvid-contract-test`;
+- namespace must start with `korvid-agent-eval-`;
+- fixtures carry `app.kubernetes.io/managed-by=korvid-agent-eval`;
+- the model receives the read-only profile; fixture writes happen before the
+  model starts;
+- cleanup deletes only the run namespace;
+- the contract cluster returns to its stopped-at-rest state.
+
+The first live fixture created three real Pods:
+
+| Pod | Real state | Cause |
+|---|---|---|
+| `search-1` | Running, Ready | valid `pause:3.10` image |
+| `checkout-1` | CreateContainerConfigError | missing ConfigMap |
+| `payments-1` | ImagePullBackOff | nonexistent image tag |
+
+The model was not given these names in advance. The conversation asked it to
+explore broadly, then corrected the focus to payments, then requested the
+evidence on screen.
+
+## Reproduction
+
+Task pack:
+
+```sh
+export KORVID_EVAL_BASE_URL=http://127.0.0.1:11435/v1
+export KORVID_EVAL_MODEL=qwen3:8b
+export KORVID_EVAL_TIMEOUT_SECONDS=300
+uv run python -m korvid.evals --profile small --reps 3 \
+  --out report.md --json report.json
+```
+
+Offline journeys:
+
+```sh
+uv run python -m korvid.evals.journeys_cli --profile small --reps 3 \
+  --out journeys.md --json journeys.json
+```
+
+Live journey after provisioning an owned namespace:
+
+```sh
+export KUBECONFIG=/tmp/korvid-live-contract-kubeconfig
+uv run python -m korvid.evals.journeys_cli \
+  --live \
+  --context aks-korvid-contract-test \
+  --namespace korvid-agent-eval-<run-id> \
+  --journeys src/korvid/evals/live_journeys \
+  --profile small --reps 3 \
+  --out live.md --json live.json
+```
+
+## 2026-08-04 Protocol Metadata
+
+- serving engine: Ollama 0.32.5, OpenAI-compatible endpoint;
+- model node: zone-2 `Standard_D32s_v5` Spot, 30 CPU / 112Gi limit;
+- profile: `small`, six iterations, one tool call per iteration;
+- result timeout: 300 seconds;
+- model quantization: Q4_K_M;
+- task source revision: `25649a3`;
+- corrected offline journey source revision: `5d37d96`;
+- corrected live journey source revision: `8e15c52`;
+- live target: `aks-korvid-contract-test`;
+- live namespace: `korvid-agent-eval-20260804200747` (deleted after run).
+
+## Interpretation Limits
+
+- Task success does not prove conversational usability.
+- Offline journeys do not prove Kubernetes fixture realism.
+- One live journey does not cover every interactive workflow.
+- UI calls are recorded headlessly; they prove correct Korvid UI intent, not
+  visual Textual rendering.
+- A model is recommended only when task, offline conversation, and live results
+  are shown separately.
+
+## Artifact Retention
+
+Human-readable methodology, scenario descriptions, and scores are versioned
+with the source code. Generated raw outputs are not:
+
+- the append-only [`eval-results`](https://github.com/hellices/korvid/tree/eval-results)
+  branch stores dated compressed artifacts;
+- each run directory includes metadata and SHA-256 checksums;
+- issue #176 links the same artifact directory and contains only summaries;
+- application commits do not carry model transcripts or pull logs.