Pin every journey turn in both directions, and stop crediting "it is fine"

[hellices]

Closes #251.

The finding

I filed this issue expecting to write test scaffolding. Running it found real defects in four journeys that shipped long before the pinning rule — and one of them is systemic.

A required group naming a topic is satisfied by ruling that topic out.

"The liveness probe is fine; gateway-1 restarts for another reason."

That satisfied must_mention: [liveness probe, probe failure] — a group whose entire purpose is to require the model to report the probe failure. The same phrasing passed image pull groups in tui-follow and rollout-owner-chain. The grader already refused not X; declaring X healthy is the same assertion in positive grammar, and nothing caught it.

The fix, in two parts

Central. The grader now treats a copula plus an all-clear adjective (is fine, looks normal, seems correct) as negation, scoped to the clause like the existing negators. The adjective list is deliberately narrow — unaffected, serving and ready are required claims elsewhere in the pack, so admitting them would reject correct answers. Two tests guard that over-reach: "The image pull failed and the node is fine" must still pass, and so must "api-5c2f is unaffected".

Per journey, for the shapes no central rule can catch:

journey	hole	fix
triage-and-correct t1	required "checkout first" but never forbade the reverse	mirror prohibition
triage-and-correct t2	named payments, blamed checkout-1 — forbidden_targets stops the read, not the attribution	prohibit checkout
triage-and-correct t3	banned two spellings of the wrong remedy; "needs more memory" was neither	broadened
rollout-owner-chain t2	"api-7b9d-x1 is still running, while api-5c2f has been scaled away" — keyword groups cannot bind a claim to a subject	ban the false claim
namespace-triage t2	bare noun environment variable	split into the thing and its absence

The rule

test_every_journey_turn_pins_an_accepting_and_a_rejecting_answer refuses to let a turn ship without both directions, and counts a turn as covered only when both sides are non-empty — an entry with an empty side pins nothing. Mutation-verified: emptying one side fails the test.

18 turns, all pinned. Gate green: 4741 passed.

[Copilot]

Pull request overview

Adds bidirectional grading pins for every bundled journey turn and strengthens detection of misleading all-clear claims.

Changes:

• Adds accepting/rejecting journey fixtures and coverage enforcement.
• Introduces all-clear predicate handling in the grader.
• Tightens grading rules for several journeys.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/korvid/evals/grader.py	Adds all-clear predicate detection.
src/korvid/evals/journeys/namespace-triage.yaml	Requires configuration absence.
src/korvid/evals/journeys/rollout-owner-chain.yaml	Rejects false scale-down claims.
src/korvid/evals/journeys/triage-and-correct.yaml	Strengthens ordering, attribution, and remedy rules.
tests/evals/test_grader.py	Tests exculpatory predicates and scope.
tests/evals/test_journey.py	Pins both grading directions for every turn.
docs/evals/methodology.md	Documents bidirectional pinning.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (6)

src/korvid/evals/grader.py:164

• working is not unconditionally all-clear: a correct diagnosis such as “the liveness probe is working too slowly and timing out” is currently exculpated as soon as working is seen, so the required liveness probe group becomes missing. Account for fault qualifiers after this token, or narrow this set so valid failure claims are not suppressed.

        "passing",
        "succeeding",
        "working",
        "good",

src/korvid/evals/grader.py:378

• Disabling exculpation for the whole scenario also disables it for must_not_mention, where it must remain active. For a healthy scenario that requires healthy and forbids image pull, “Everything is healthy; the image pull looks normal” is now rejected as a positive image-pull diagnosis, contrary to the symmetric polarity contract. Disable exculpation only while evaluating required groups in negative controls; forbidden groups should always treat all-clear predicates as rule-outs.

    # A scenario with no fault is graded on an all-clear, so "the endpoints
    # are healthy" is its answer rather than a way of dodging the question.
    exculpation = scenario.root_cause != "none"

    def claims(keyword: str) -> bool:
        return _mentions_positively(keyword, answer_tokens, clause_ids, exculpation=exculpation)

tests/evals/test_journey.py:684

• This coverage check can pass even though the phrase test does not use each journey's actual polarity. _turn_scenario() hardcodes root_cause="r", so the newly added healthy-stop cases are graded as fault scenarios even though production loads them with root_cause: none; that now changes exculpation behavior. Construct the test scenario with journey.root_cause so these pins exercise the real grading configuration.

def test_every_journey_turn_pins_an_accepting_and_a_rejecting_answer() -> None:

src/korvid/evals/journeys/triage-and-correct.yaml:21

• The reverse list omits the exact mirror of the accepted checkout should be inspected first wording. Consequently, “checkout should be inspected first, but payments should be inspected first instead” still satisfies every required group without hitting this prohibition. Add the symmetric phrasing and pin it as a rejecting case.

        - [payments first, prioritize payments, start with payments,
           inspect payments first, payments is worse,
           payments is more urgent, payments takes priority]

src/korvid/evals/journeys/triage-and-correct.yaml:51

• This still permits the wrong-subject attribution through an unlisted grammar. For example, “payments-1 is failing; checkout-1 suffers an authentication failure” satisfies [payments] and [authentication] but matches none of these checkout is/has phrases. Bind the required cause to payments (or otherwise cover checkout-bound authentication claims) and pin this form, rather than only prohibiting one sentence shape.

        - [checkout is the one, checkout-1 is the one, checkout is the cause,
           checkout-1 is the cause, cause is checkout, blame checkout,
           checkout is unauthorized, checkout-1 is unauthorized,
           checkout has the registry, checkout-1 has the registry,
           checkout has the unauthorized, checkout-1 has the unauthorized,

src/korvid/evals/journeys/namespace-triage.yaml:60

• Common contraction forms are not accepted here: tokenization turns “DSN isn't set” into dsn isn t set, which does not match dsn is not set. That makes a correct diagnosis fail despite expressing the required subject-bound absence. Include the contraction variants and pin one as accepting.

        - [dsn is missing, missing dsn, dsn is not set, dsn is unset,
           database_dsn is missing, missing database_dsn,
           database_dsn is not set, database url is missing,

[hellices]

Round 2: six suppressed findings, all six upheld and fixed in ba88532. Two were structural, and I would not have found either.

Polarity is not symmetric. Disabling exculpation for a negative control disabled it for must_not_mention too, so "Everything is healthy; the image pull looks normal" was rejected as a positive image-pull diagnosis — the answer punished for telling the truth. Required claims and forbidden claims now go through claims and diagnoses respectively. Test: test_grade_exculpation_stays_on_for_forbidden_keywords.

The pins were not exercising production. _turn_scenario hardcoded root_cause="r", so every journey pin was graded as a fault scenario — including healthy-stop, which production loads with root_cause: none. The moment polarity started affecting grading, those pins were certifying a configuration that does not exist. It now takes journey.root_cause.

The other four are the recurring lesson that a list cannot bound a sentence:

finding	evidence
working is not all-clear	"The liveness probe is working too slowly and timing out" lost its own diagnosis. Removed, with good.
missing ordering mirror	"checkout should be inspected first, but payments should be inspected first instead" passed
unlisted attribution grammar	"checkout-1 suffers an authentication failure" walked past every enumerated shape
contractions	"DATABASE_DSN isn't set" tokenizes to dsn isn t set

The attribution fix is the interesting one: instead of enumerating ways to blame the wrong pod, the required cause is now bound to payments, which closes the class. That immediately over-narrowed — "payments has an unauthorized ..." missed by one token and broke test_journey_runner. Article variants are spelled out, which is bounded in a way sentence shapes are not.

Every reviewed phrasing is pinned. Gate green: 4744 passed.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

src/korvid/evals/grader.py:388

• Applying exculpation to every forbidden alternative also suppresses alternatives that already encode a fault. For the bundled healthy-service-endpoints scenario, "Everything is healthy; having no endpoints is normal." satisfies the required all-clear, but no endpoints is discarded here because it is followed by is normal, so the factually wrong answer passes. Exculpation needs to distinguish neutral topics such as image pull from polarized claims such as no endpoints (and similar missing/failure alternatives).

    def diagnoses(keyword: str) -> bool:
        return _mentions_positively(keyword, answer_tokens, clause_ids, exculpation=True)