Measure whether diagnose_service/diagnose_pvc belong on the small-model tool surface

[hellices]

Problem

diagnose_service and diagnose_pvc were added to all three surfaces, including small_agent. Measured cost on the small profile today:

	tools	schema
current	16	8,107 chars
without the two	14	7,046 chars

That is +14% tools and +15% schema, retransmitted on every request. diagnose_pvc now has the longest description of all 16 tools (233 chars), and the four diagnose_* tools occupy the top four places by description length.

This was never costed against the small profile's design premise.

Why it matters

The small profile exists to shrink the surface for 3B-14B local models. Its own module docstring cites BFCL: small models are competitive on simple single-function calls but fall behind sharply on multi-function selection, and degrade with context length well below their advertised windows. Its budgets are sized for a ~4k-token serving context.

Adding two tools to that surface may be a net win — a single bounded call can replace a multi-step read sequence, which is exactly what a small model is bad at — or a net loss, because the selection space grew and every request pays for the schema. Both are plausible. Neither has been measured.

Published work says the schema is the high-leverage variable, so this is not a rounding error: EasyTool and Tool Documentation both find tool-description quality dominates prompt preamble for tool-selection accuracy.

Proposal

Fold this into the matched baseline-versus-diagnostic evaluation already planned in #176, as a third arm rather than a separate study:

1. baseline — ordinary read tools only
2. diagnostic (full surface) — current small surface, 16 tools
3. diagnostic (full/mcp only) — the two tools removed from small_agent, 14 tools

Metrics that decide it: tool-selection accuracy on tasks where the diagnostic tool is not the right answer (does the bigger surface cause misselection?), evidence coverage, tool-call count, and tokens per turn.

Outcomes:

• arm 3 wins → restrict surfaces to _FULL_AND_MCP, a one-line registry change;
• arm 2 wins → keep as-is, and the cost is justified by measurement rather than assumed;
• no measurable difference → keep as-is and record that the surface size is not the binding constraint at this scale.

Not urgent

Nothing is broken. This is a question the evaluation should answer while it is already set up to answer the adjacent one. Blocked on the same eval-harness gaps that block #176: the Service scenarios seed legacy Endpoints while diagnose_service reads EndpointSlice, and no bundled scenario exercises either tool.

Context: #191 (closed as not planned), #213, #216.

[hellices]

Partial evidence: the 2-tool addition did not hurt, and may have helped

Two of the three models from the 2026-08-05 campaign have now been re-measured on the current revision, which is the first time the same model has been scored on both tool surfaces.

Model	14 tools (124b1aa)	16 tools (bdfb645)	Δ
Qwen3 8B	61/69 (88.4%)	64/69 (92.8%)	+4.4pp
Qwen3 1.7B	50/69 (72.5%)	55/69 (79.7%)	+7.2pp

Both on the 23-scenario denominator, so the #227 scenario additions are excluded and this compares like with like on scenarios.

Read this cautiously

This is not the controlled experiment #221 asks for. The two runs differ in more than the tool surface:

• diagnose_service and diagnose_pvc were added (agent: add deterministic Service endpoint diagnosis #213, agent: add deterministic PVC binding diagnosis #216) — the variable of interest;
• the grader gained identity aliases and kind matching (fix(evals): credit diagnose_service and diagnose_pvc as evidence #219) and the dash fix (eval: add Service and PVC diagnostic scenarios, and fix dash negation #227) — though the dash fix was separately proven to cause 0 grade flips on 369 retained runs;
• the ollama memory limit went from 10 GiB to 100 GiB, and CPU from 3.5 to 31 cores;
• sampling is non-deterministic (the eval CLI sends no temperature, so the OpenAI-compatible default applies), and the published run-to-run σ for these models was 4-5pp.

The 1.7B's +7.2pp exceeds its published σ of 4.10pp; the 8B's +4.4pp is within its 5.42pp. So the direction is consistent across both models but only one gap clears its own noise floor.

What it does establish

The concern behind #221 was that adding two tools to a 14-tool surface would degrade small-model tool selection, per BFCL's finding on multi-function selection. On these two models, at this scale, it did not. Neither regressed; both improved. That is enough to say the surface addition is not obviously harmful, and to lower the urgency of the question.

It is not enough to close #221. A proper answer still needs the same binary, same revision, same grader, differing only in whether the two tools are offered — which needs a CLI flag to drop them that does not exist yet.

Qwen3-Coder 30B-A3B is re-running now and will add a third data point.

Context: #176, #213, #216.

[hellices]

Harness support for the controlled arms is merged (#244). This issue stays open — the measurement has not been run.

--without-tool NAME (repeatable) drops a tool from the measured surface:

python -m korvid.evals --profile small --json arm2.json
python -m korvid.evals --profile small \
  --without-tool diagnose_service --without-tool diagnose_pvc --json arm3.json

Review turned up two ways an arm could have silently lied about itself, both fixed before merge and worth recording because they are the kind of thing that invalidates a campaign rather than degrading it:

Hiding a schema is not removing a tool. AgentRuntime dispatches whatever name the provider returns and ToolExecutor resolves every registry read, so a model that remembered diagnose_pvc still got its result — and the run credited the call, because the metrics classified against the global read set rather than the surface the run offered. Reproduced: a scripted reduced-surface run came back evidence_fetched=True. Omitted calls are now refused at execution and counted as malformed, which is itself the signal — whether a small model reaches for a tool it was not given is the question.

A refused call must not look like one that landed. ToolOutcome.error defaults to False, so an omitted write call was recorded as a write that succeeded and as a safety violation, for a call the executor never saw. That would have manufactured the most load-bearing number on the scoreboard.

Also: an unknown or UI-only name is refused rather than ignored (a typo would publish the full surface as the reduced arm), the omission reaches the prompt fingerprint so two arms cannot share a digest, and meta.tools records omitted and count by name so the arm is readable from the artifact instead of a digest lookup table.

Remaining work is cluster time: run arms 2 and 3 on the same models with --warmup and the pinned engine from #235, and decide on tool-selection accuracy where the diagnostic tool is not the right answer.

[hellices]

Closing this as not-now, with the engineering half delivered.

What shipped

--without-tool NAME (repeatable) removes a tool from the measured surface, so the three arms this issue describes are one command each:

python -m korvid.evals --profile small --warmup --json arm1.json
python -m korvid.evals --profile small --warmup \
  --without-tool diagnose_service --without-tool diagnose_pvc --json arm3.json

The arm is recorded in meta.tools and reaches the prompt fingerprint, so two arms cannot share a digest. Review during #244 caught two ways an arm could have silently lied about itself — a hidden schema is not a removed tool, and a refused call must not be recorded as one that landed — both fixed before merge.

Why it is closing unanswered

What remains is not code. It is a cluster campaign: the modeleval pool is at 0 nodes by design, and a publishable run means provisioning, warming each model, and three arms per model. The answer it produces is also perishable and model-specific — it would describe the models loaded that week, not a property of korvid.

Nothing is blocked on it. The small profile ships both tools today, no user has reported the surface as a problem, and the decision to drop a tool can be taken whenever the question becomes live — at which point the measurement is the command above, not a project.

Reopen when there is a concrete reason to re-tune the small-model surface. The harness, the methodology, and the pinned serving environment (#235) will all still be here.