Skip to content

Add eval scenarios that exercise the Service and PVC diagnostic contracts #225

Description

@hellices

Problem

No bundled scenario exercises diagnose_service or diagnose_pvc:

두 도구를 증거 경로로 선언한 시나리오: 없음

The 23-scenario pack grades ordinary get_resource / get_events / diagnose_pod paths. Nothing puts a model in a situation the two deterministic tools were built for.

#176 plans a matched baseline-versus-diagnostic comparison to decide whether those tools are worth keeping. Without scenarios covering their contracts, that comparison has almost nothing to measure.

What is already fixed

The related grader defects are closed by #219, which is a useful floor:

  • identity arguments (service, pvc) fold onto name, so a model using either tool is credited for the evidence it fetched;
  • diagnose_service implies {services, endpoints, endpointslices}, so it satisfies the existing Service scenario's kind: endpoints evidence — verified on main.

So the harness can now measure a model that reaches for these tools on the existing pack. What is missing is scenarios that make reaching for them the right move, and that cover the states the analyzers actually distinguish.

Scope

Add scenarios covering the two contracts, per #176's list:

Service — ready endpoints; matching slices with none ready; no slices at all; stale slices; an RBAC-denied EndpointSlice read producing an explicit evidence gap. Include a healthy control. Do not credit a selector-mismatch conclusion unless both the selector and the pod labels were observed.

PVC — Bound; Lost; Pending with a provisioning-failure event; named StorageClass missing; no default class; multiple defaults; WaitForFirstConsumer; static/pre-bound; an RBAC-denied StorageClass read.

Constraints

  • Evidence must stay route-agnostic. Express expected evidence as the fact and its target, not as "you must call diagnose_service". A baseline run without the diagnostic tools has to be able to satisfy the same evidence through ordinary reads, or the comparison measures tool availability instead of diagnosis quality.
  • The Service fixtures currently seed legacy Endpoints while diagnose_service reads EndpointSlice. New Service scenarios should seed EndpointSlice; whether to also keep Endpoints for the legacy read path is a judgement call worth making explicitly.
  • test_bundled_evidence_is_reachable_through_the_real_tools must keep passing — every declared evidence item has to be genuinely fetchable from the fixture.

Acceptance criteria

Context: #176, #191 (closed), #213, #216, #219, #221.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions