Split out of #176 so it is a task rather than a standing programme.
The 8-journey pack currently only runs against a live model, which means the
grading spec is exercised on the cluster and nowhere else. Running the pack
against a scripted provider costs nothing and would catch the failure mode
that dominated #249's review: keyword groups that grade something other than
what their comment claims.
Four rounds of review on #249 found, in journeys I had just written:
- a positive phrase (
access to the log) sitting in the group that requires
the model to report a denial, so "I have access to the log" satisfied it;
- a prohibition on
the log shows that rejected the correct answer;
- a "reason" group that accepted the bare object name
endpoints, so the
right pick with the exactly inverted rationale passed;
- an ordering requirement with no reverse-order prohibition.
Every one was invisible to the existing tests and visible immediately once a
specific phrasing was run through the grader.
What to build
test_new_journey_keywords_discriminate already pins accept/reject phrasings
per turn. Extend that idea to a coverage rule: every journey turn must have
at least one pinned accepting and one pinned rejecting phrasing, enforced by
a test so a new journey cannot ship without them.
Optionally also run the full pack end to end against a ScriptedProvider
playing a known-good trace, asserting every turn passes — which pins the
fixture and the grading together.
Out of scope
Model runs of any kind. This is entirely offline.
Split out of #176 so it is a task rather than a standing programme.
The 8-journey pack currently only runs against a live model, which means the
grading spec is exercised on the cluster and nowhere else. Running the pack
against a scripted provider costs nothing and would catch the failure mode
that dominated #249's review: keyword groups that grade something other than
what their comment claims.
Four rounds of review on #249 found, in journeys I had just written:
access to the log) sitting in the group that requiresthe model to report a denial, so "I have access to the log" satisfied it;
the log showsthat rejected the correct answer;endpoints, so theright pick with the exactly inverted rationale passed;
Every one was invisible to the existing tests and visible immediately once a
specific phrasing was run through the grader.
What to build
test_new_journey_keywords_discriminatealready pins accept/reject phrasingsper turn. Extend that idea to a coverage rule: every journey turn must have
at least one pinned accepting and one pinned rejecting phrasing, enforced by
a test so a new journey cannot ship without them.
Optionally also run the full pack end to end against a
ScriptedProviderplaying a known-good trace, asserting every turn passes — which pins the
fixture and the grading together.
Out of scope
Model runs of any kind. This is entirely offline.