Repository navigation
The mock answers a judge with a verdict object - #1682
Merged
Muizzkolapo merged 3 commits intoNov 8, 2026
Merged
Muizzkolapo merged 3 commits into
Muizzkolapo merged 3 commits into
Conversation
Muizzkolapo
force-pushed
the
mock-answers-in-the-shape-a-prompt-offers
branch
2 times, most recently
from
November 8, 2026 01:16
58c948f to
f13e157
Compare
Muizzkolapo
marked this pull request as ready for review
November 8, 2026 09:59
Muizzkolapo
force-pushed
the
mock-answers-in-the-shape-a-prompt-offers
branch
from
November 8, 2026 13:41
f13e157 to
b3dcb5a
Compare
Muizzkolapo
force-pushed
the
mock-answers-in-the-shape-a-prompt-offers
branch
from
November 8, 2026 14:41
b3dcb5a to
555268f
Compare
An llm_judge asks in text mode for one of two JSON objects. The agac-provider mock answered every text prompt with prose, so every judged outcome failed with "judge response was not a verdict object". A judged rule could not be tested offline. In text mode the mock now reads the prompt's last non-empty line. When that line is only JSON objects joined by " or ", the mock returns one of them, chosen by its seeded generator, with each string value replaced by a generated sentence. Any other text prompt still gets prose.
The mock answered with a JSON object whenever a text prompt's last line was JSON objects joined by " or ". A single object matched too, so a text action whose prompt ends with a rendered record got an object back in place of prose. The mock now needs two or more objects on that line. A judge prompt offers two, so it still gets a verdict.
The mock seeded an offered answer with the prompt and the attempt number. Every judge call of one action shares one attempt counter, so the verdict for a value depended on how many judge calls came before it: adding a record, a second judged rule or a vote changed the verdicts of later records. The docs promise one verdict for one rule and value. The offered answer is now seeded by the prompt alone. Two tests pin it: a value keeps its verdict after other judge calls, and the order of values does not change their verdicts. Two more cases check that two offered objects with extra text on the line still get prose.
Muizzkolapo
force-pushed
the
mock-answers-in-the-shape-a-prompt-offers
branch
from
November 8, 2026 14:54
555268f to
589abde
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
llm_judgeasks in text mode for one of two JSON objects. Theagac-providermock answered every text prompt with prose, so every judged outcome failed with "judge response was not a verdict object".or, the mock returns one of them. Any other text prompt still gets prose.Verification
tests/unit/llm/providers/test_mock_follows_an_offered_answer_shape.pyon the parent)