[infra] Golden eval harness gives false assurance: one hardcoded log set + dead confidence check

[rahulkatiyar19955]

Severity

Medium

Summary

The golden eval harness does not actually verify what it claims. In mock mode every case resolves citations against a single hardcoded lidar log set regardless of bag, so "100% grounding" is met by one canonical dataset; and the confidence threshold (confidence_min) is dead config because confidence is never propagated or asserted. Both let real regressions pass.

Findings

• Citation grounding is satisfied by one hardcoded log set — backend/tests/eval/conftest.py:20-45, backend/tests/eval/runner.py:104-115, backend/tests/fixtures/mock_llm.py:43-78, backend/tests/eval/golden.yaml — the seed_canonical_logs autouse fixture stubs mock_neo4j.run_query to always return the canonical lidar_failure logs (l_5,l_6,l_8,l_9) for every case, and the mock specialists return the same canonical lidar findings regardless of bag. Only lidar_why declares citations_must_include_log_ids; the nav_drift_* and controller_* cases never exercise per-bag retrieval or distinct causal chains. So the grounding assertion in test_golden.py:56-65 is met by a single dataset; real per-bag grounding is only tested under LIVE_LLM=1, which CI never runs. fix: give each golden bag its own canonical log set + mock responses, add citations_must_include_log_ids to every case and assert it for all cases, and run the eval in CI in mock mode as a required gate.
• Confidence assertions are silent no-ops — backend/tests/eval/runner.py:78-89, backend/tests/eval/test_golden.py, golden.yaml confidence_min — EvalRun.confidence_floor loops over audit_trail with a bare pass and then returns 1.0 whenever any findings exist (else 0.0), so it never reflects real specialist confidence; and test_golden.py never reads the confidence_min values present in every golden case. The threshold is dead config and confidence regressions cannot be detected. fix: propagate specialist confidence into the envelope/audit trail, implement confidence_floor for real, and assert confidence_min per case in the golden test.

Project-rule reference

AGENT.md ## Testing Expectations (line 283) — "Run the golden eval harness ... Citation grounding must be 100%"; ### Citation & Causality (line 260).