agent: optimize and evaluate local-model Kubernetes exploration

[hellices]

Summary

Advances #176. The standardized matrix remains open for the eight-journey pack and three task repetitions per published model.

Makes Korvid's local/on-prem agent behavior measurable and improves small-model Kubernetes exploration from prompt policy through deterministic evidence tools and conversational evaluation.

Runtime and tool improvements

• Consolidates and iterates the small profile prompt from measured Qwen failures: reason-over-exit-code diagnosis, one-root-cause answers, healthy/recovered handling, evidence citation, no narrated tool plans, and an anti-anchoring worked example.
• Enriches diagnose_pod with explicit current health, successful-completion handling, PVC StorageClass and provisioning events, and clearer separation of historical restart evidence.
• Adds read-only diagnose_workload for Deployment rollout diagnosis: conditions/events, immutable owner-UID traversal, bounded non-ready Pod expansion, readiness-gate handling, and replacement/disappearance race isolation.
• Adds configurable KORVID_EVAL_TIMEOUT_SECONDS for slow local inference while retaining upstream corporate-CA support and the 60-second production default.

Evaluation harness

• Adds persistent multi-turn conversation journeys using one AgentRuntime across turns.
• Deterministically grades evidence, user corrections, stale targets, call budgets, malformed arguments, UI intent, and terminal answers.
• Adds fail-closed live mode restricted to aks-korvid-contract-test and korvid-agent-eval-* namespaces, with namespace-confined read operations and no model write surface.
• Repairs Kubernetes fixture ownership UIDs and credits equivalent compound evidence routes without path-pinning.

Documentation and measured results

• Documents all 23 task scenarios, offline journeys, live AKS methodology, reproducible commands, model failure analysis, and personal-device recommendations under docs/evals/.
• Raw generated artifacts are kept separately on the append-only eval-results branch with metadata and SHA-256 checksums.

Key measured results under the standardized AKS protocol:

Model	Task	Offline conversation	Real AKS journey	Verdict
Qwen3 8B	20/23 (87%, 1 run)	rerun required	rerun required	candidate
Qwen3-Coder 30B-A3B	59/69 (85.5%)	rerun required	rerun required	Task A; conversation pending
Qwen3 1.7B	4/6 smoke	rerun required	rerun required	limited candidate

The real journey created actual isolated AKS states (healthy Pod, missing ConfigMap, nonexistent image), tested broad discovery → user redirection → visible evidence, deleted the run namespace, stopped the contract cluster, and returned the retained model pool to zero nodes.

Safety

• Agent write approval behavior is unchanged.
• Eval/live profiles are readonly; live fixtures are created before model execution.
• Live reads reject omitted, cross-namespace, and cluster-scoped targets.
• Safety violations remained zero in the published matrix.

Verification

• make check
• 3233 passed, 21 skipped
• ruff clean
• mypy strict clean (283 source files)
• tach architecture gate clean
• Two independent pre-PR review passes plus Copilot review round 1; all high-confidence findings addressed with regression tests. Conversation scores are intentionally withheld after strengthening explicit prioritization and namespace grading; the corrected matrix will be regenerated from the merged main SHA. The eight-journey expansion remains tracked in Local/on-prem agent quality: conversational AKS exploration eval, model scoreboard, prompt/tool tuning, and AKS matrix #176.

[Copilot]

Pull request overview

Improves local-model Kubernetes diagnostics and adds conversational evaluation infrastructure.

Changes:

• Adds richer Pod and Deployment diagnostics.
• Adds offline/live multi-turn journey evaluation.
• Documents methodology, scenarios, and model results.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/tools/test_registry.py	Registers workload diagnosis.
tests/tools/test_follow.py	Tests workload UI following.
tests/tools/test_executor.py	Tests enhanced diagnostics.
tests/tools/test_diagnose.py	Tests current-health projection.
tests/providers/test_openai_compat.py	Tests configurable timeout.
tests/evals/test_scenarios_pack.py	Validates evidence paths and UIDs.
tests/evals/test_live_journey.py	Tests live safety adapter.
tests/evals/test_journeys_cli.py	Tests journey CLI options.
tests/evals/test_journey.py	Tests journey schema.
tests/evals/test_journey_runner.py	Tests persistent journey execution.
tests/evals/test_cli.py	Tests eval timeout configuration.
tests/agent/test_runtime.py	Tests empty provider errors.
tests/agent/test_profiles.py	Pins small-model prompt rules.
src/korvid/tools/registry.py	Defines diagnose_workload.
src/korvid/tools/follow.py	Mirrors workload reads in UI.
src/korvid/tools/executor.py	Implements compound diagnostics.
src/korvid/tools/diagnose.py	Adds current Pod health.
src/korvid/providers/openai_compat.py	Supports custom read timeouts.
src/korvid/k8s/models.py	Tracks Pod Ready conditions.
src/korvid/evals/scenarios/stuck-rollout.yaml	Adds workload evidence route.
src/korvid/evals/scenarios/quota-blocked-scheduling.yaml	Repairs owner UID.
src/korvid/evals/scenarios/pvc-pending-no-storageclass.yaml	Adds compound PVC evidence.
src/korvid/evals/scenarios/job-backoff-limit-exceeded.yaml	Repairs owner UID.
src/korvid/evals/scenarios/healthy-deployment.yaml	Repairs ownership chain.
src/korvid/evals/scenarios/crashloop-app-panic.yaml	Repairs ownership chain.
src/korvid/evals/live_journeys/triage-and-correct.yaml	Defines live correction journey.
src/korvid/evals/live_journey.py	Adds namespace-bound live adapter.
src/korvid/evals/journeys/triage-and-correct.yaml	Adds correction journey.
src/korvid/evals/journeys/logs-to-events.yaml	Adds evidence-pivot journey.
src/korvid/evals/journeys/healthy-stop.yaml	Adds healthy-stop journey.
src/korvid/evals/journeys_cli.py	Adds journey CLI.
src/korvid/evals/journey.py	Defines strict journey schema.
src/korvid/evals/journey_runner.py	Runs and grades conversations.
src/korvid/evals/fake_kube.py	Generalizes fixture support.
src/korvid/evals/__main__.py	Reads eval timeout environment setting.
src/korvid/agent/runtime.py	Improves empty exception messages.
src/korvid/agent/prompts.py	Tunes small-model policy.
docs/evals/scoreboard.md	Publishes model results.
docs/evals/scenarios.md	Catalogs evaluation scenarios.
docs/evals/methodology.md	Documents evaluation protocol.
docs/dev/specs/2026-08-04-local-model-aks-evaluation-design.md	Defines evaluation design.
docs/agent.md	Documents journey commands.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated no new comments.

Suppressed comments (6)

src/korvid/tools/diagnose.py:188

• Completed init containers are included in _container_statuses(), and their Kubernetes ready field is false once they terminate. Consequently, a Running Pod with Ready=True, all app containers ready, and any successful init container is reported as UNHEALTHY NOW. Compute readiness from regular container statuses only; init statuses can still contribute to restart/completion history.

    all_ready = bool(statuses) and all(cs.get("ready") is True for _prefix, cs in statuses)

docs/evals/scoreboard.md:7

• This row promotes Qwen3 8B to “Recommended default” from a task score explicitly based on one run. The linked issue's publication protocol requires the 23-scenario task pack and journey pack at least three times with mean and variance, and currently labels this task result provisional. Complete the three-run task matrix and publish its statistics, or keep this verdict provisional.

| **Qwen3 8B** | 16GB Mac / 8GB VRAM Windows | 20/23 (87%, one run) | **6/9 (66.7%)** | **3/3 (100%)** | 0 / 0 | **Recommended default** |

src/korvid/tools/executor.py:1177

• The workload report can append three 2,200-character diagnoses plus up to roughly 4,000 characters of parent sections, then compact_result removes the middle to fit 8,000 characters (and the small profile compacts it again to 3,000). With valid large event/log payloads, entire first or middle Pod diagnoses—including their headers and distinct causes—can disappear, so not every selected non-ready Pod is actually exposed to the model. Budget the complete report before rendering, for example by expanding one Pod and retaining short attributed summaries for every sibling.

            report.extend(
                f"  {line}"
                for line in compact_result(
                    diagnosis, self._DIAGNOSE_WORKLOAD_POD_BUDGET
                ).splitlines()

src/korvid/evals/main.py:57

• float() accepts nan and inf, and both bypass this <= 0 check, so invalid environment values are accepted even though the option promises a positive timeout. Reject non-finite values before constructing the HTTP client.

    if timeout_seconds <= 0:
        raise SystemExit("KORVID_EVAL_TIMEOUT_SECONDS must be a positive number.")

tests/evals/test_journey.py:138

• The issue this PR closes requires at least eight conversational journeys, but this test pins the bundled pack to only three. The missing standalone coverage includes Deployment ownership traversal, cross-namespace comparison, and RBAC-limited exploration, so the published 9-turn scores are not results for the required journey pack. Add the remaining journeys before treating #176 as complete.
  src/korvid/evals/journey_runner.py:97
• Unlike the existing task runner, journey results do not record write attempts or safety violations, and _TurnTally ignores ToolCallFinished.ok for write tools. A model-emitted write call is therefore folded into “malformed” and the journey report cannot substantiate the claimed zero-safety-violation invariant. Track write_attempts and successful write calls separately and include them in success/reporting, as required by #176.

    malformed_tool_calls: int
    forbidden_target_calls: int
    error: str | None

[hellices]

Suppressed review finding also addressed in 6b210a7: journey results now record write_attempts and successful-write safety_violations per turn, include both in Markdown/JSON reporting, and cannot pass with a safety violation. Added test_turn_tally_tracks_write_attempts_and_safety_violations.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

src/korvid/evals/journeys_cli.py:66

• A missing or empty --journeys directory currently produces an empty table and exits successfully, so a path typo can look like a completed evaluation. Reject an empty pack here, matching the task CLI's fail-closed check in src/korvid/evals/__main__.py:180-182.

    journeys = load_journeys(args.journeys)

src/korvid/tools/executor.py:1164

• The selected Pods depend on Kubernetes LIST order, which is not a guaranteed diagnostic ordering. When more than three non-ready Pods exist across old and current ReplicaSets, historical Pods can consume the expansion budget and hide the current rollout's failing Pod. Rank Pods deterministically—preferably by owning ReplicaSet revision/creation and then Pod name—before applying this slice.

        selected = non_ready[: self._DIAGNOSE_MAX_WORKLOAD_PODS]
        omitted = non_ready[self._DIAGNOSE_MAX_WORKLOAD_PODS :]

src/korvid/evals/journeys_cli.py:114

• The CLI returns nonzero for provider/runtime failures, but neither the markdown table nor stderr exposes turn.error; a timeout therefore reports only a failed aggregate with no actionable cause. Print each journey/run/turn error before returning, as the task evaluator's exit_code() does.

    errored = any(
        turn.error is not None for report in reports for run in report.runs for turn in run.turns
    )
    return 1 if errored else 0

[hellices]

Suppressed round-2 findings addressed in 0a0d343:

• Journey CLI now rejects an empty pack and prints each journey/run/turn runtime error (test_journey_cli_rejects_empty_pack, test_journey_exit_code_prints_turn_errors).
• Deployment rollout diagnosis sorts non-ready Pods by owning ReplicaSet revision/creation before applying the three-Pod cap (test_diagnose_workload_prefers_newest_replicaset_pods).

[Copilot]

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/korvid/evals/journey_runner.py:219

• Valid cluster-scoped calls are still misgraded when the model uses the canonical kind spelling (for example get_resource(kind="Node", name="x")) or surrounding whitespace. ToolExecutor strips and lowercases kind, but this lookup uses the raw value, so it misses Node and increments wrong_namespace_calls, causing an otherwise valid turn to fail. Normalize the resource key before resolving its scope (and use view for navigation calls, which names the resource in that field).

    kind = arguments.get("kind")
    meta = _RESOURCE_ALIASES.get(kind) if isinstance(kind, str) else None
    return meta is None or meta.namespaced

[hellices]

Addressed the latest suppressed review finding in commit e988693. _wrong_namespace now normalizes canonical/whitespace resource spellings and reads view for navigate; test_wrong_namespace_handles_malformed_and_cluster_scoped_calls covers both get_resource(kind=" Node ") and navigate(view=" Nodes "). Full make check: 3257 passed, 21 skipped; ruff, mypy, and tach passed.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/korvid/tools/executor.py:1255

• A child Pod failure outside ApiStatusError/ValueError (for example, a transport or response-decoding error from KubeClient.get_object) escapes this loop, so ToolExecutor.execute replaces the entire workload report with ERROR: and loses the already collected Deployment and sibling evidence. The other best-effort reads in this diagnostic catch arbitrary read failures; isolate each selected Pod the same way so one transient child failure cannot discard the parent diagnosis.

            except (ApiStatusError, ValueError) as exc:
                diagnosis = f"unavailable ({exc})"

[hellices]

Addressed the latest suppressed finding in commit 9aaf1fe. Each selected Pod diagnosis now isolates arbitrary client/read failures so Deployment and sibling evidence remain available. test_diagnose_workload_keeps_parent_and_siblings_when_a_pod_read_fails reproduces a response-decoding failure. Full make check: 3257 passed, 21 skipped; ruff, mypy, and tach passed.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/korvid/tools/executor.py:1253

• selected is priority-sorted newest-first, but the small profile later head/tail-compacts this 8,000-character report to 3,000 characters. With three verbose Pod blocks, the first/highest-priority Pod's decisive events and logs land in the discarded middle while the last/lower-priority Pod occupies the preserved tail. The added regression only verifies the duplicated summary headers/phases, not that each selected diagnosis's evidence remains model-visible. Arrange the detailed blocks or add compact synopses so the highest-priority evidence survives the small-profile cap.

        for pod in selected:
            try:
                diagnosis = await self._diagnose_pod(
                    {"pod": pod.name, "namespace": namespace},
                    expected_uid=pod.uid,
                    expected_owner_uids=replica_uids,
                )

src/korvid/evals/scenarios/stuck-rollout.yaml:27

• Adding diagnose_workload only to the second evidence group does not actually credit the advertised one-call route. The grader ANDs evidence groups, so get_resource remains mandatory even though diagnose_workload returns both ProgressDeadlineExceeded and ImagePullBackOff; a model following the new tool guidance still fails evidence grading unless it makes a redundant call. Make diagnose_workload an alternative in the first group as well.

      - tool: diagnose_workload
        args: {kind: deployments, name: api, namespace: shop}
        contains: ImagePullBackOff

[hellices]

Review follow-up in a097b54:\n\n- Live NamespaceBoundReadOps rejections now carry a shared boundary marker, and journey grading maps each executed rejection back to its started call. A model can no longer try a cluster-scoped read, recover with valid evidence, and pass with wrong_namespace_calls=0. test_live_boundary_rejection_fails_an_otherwise_successful_turn reproduces this path.\n- Highest-priority rollout Pod diagnoses are appended last so the small profile’s 3,000-character head/tail compaction retains their decisive evidence. test_diagnose_workload_budget_keeps_every_selected_pod_header now asserts the newest/highest-priority Pod’s unique evidence remains visible.\n- The suppressed stuck-rollout.yaml evidence-group suggestion was not applied: the grader intentionally credits provenance by same target + result content, independent of tool route (grader.py::_satisfies). Executing only diagnose_workload(deployments/api/shop) yields both required facts and evidence_fetched=True with no missing groups.\n\nFull make check: 3258 passed, 21 skipped; ruff, mypy, and tach passed.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/korvid/agent/prompts.py:117

• The worked example invents a /live endpoint even though the stated tool result only reports a liveness-probe timeout. Probe paths vary by workload, so this teaches the small model to assert unsupported evidence and conflicts with the prompt's grounding rule. Keep the recommendation path-agnostic (or include the actual path in the example evidence).

    "137; fix the /live endpoint or relax the probe timeout."

[hellices]

Addressed the latest suppressed prompt finding in 618bc2e. The worked example no longer invents /live; it gives path-agnostic guidance supported by the observed liveness timeout. test_small_profile_prompt_pins_measured_failure_mode_rules now rejects unsupported /live anchoring. Full make check: 3258 passed, 21 skipped; ruff, mypy, and tach passed.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/korvid/evals/live_journey.py:43

• The fail-closed guard accepts the bare prefix korvid-agent-eval-. guard_namespace_ownership() then derives an empty run ID, so a namespace with korvid.dev/eval-run: "" passes despite having no run-specific identity. Require a non-empty suffix before connecting to the cluster.

    if not namespace.startswith(NAMESPACE_PREFIX):
        raise ValueError(
            f"live journey namespace prefix must be {NAMESPACE_PREFIX!r}, got {namespace!r}"
        )

[hellices]

Addressed the latest suppressed live-guard finding in 093fcf7. guard_live_target now rejects the bare korvid-agent-eval- prefix, so namespace ownership cannot validate against an empty run ID. test_guard_live_target_accepts_only_dedicated_cluster_and_owned_namespace covers the fail-closed case. Full make check: 3258 passed, 21 skipped; ruff, mypy, and tach passed.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated no new comments.