Make grader response parsing robust to reasoning and echoed rubrics

[dan-s-w]

Every grader took the first regex match in the judge's response. That's only safe if the judge emits nothing but its verdict, and the prompts don't all guarantee that.

SimpleQA — the worst one

match = re.search(r"(A|B|C)", grading_response)

Unanchored, so it matches a capital A, B or C anywhere, including inside a word. A judge response opening "Based on the gold target..." matches the B in "Based" and scores the answer INCORRECT — whatever the actual verdict was.

The prompt does say "Just return the letters, with no text around it," so this needs a chatty judge to bite. But nothing enforces it, and the failure is silent and looks exactly like a wrong answer.

FRAMES — matched against its own prompt

The prompt asks for this shape:

"Explanation:" (How you made the decision?)
"Decision:" ("TRUE" or "FALSE")

Explanation first. So re.search(r"(TRUE|FALSE)", ...) reads whichever appears in the reasoning, not the decision. "It is FALSE to say these differ... Decision: TRUE" scored FALSE.

A response with no match was worse:

grade_letter = match.group(0) if match else None
score_name = {"TRUE": ..., "FALSE": ...}[grade_letter]   # KeyError on None

which surfaced as a FAILED row rather than a parse problem.

FinSearchComp

Took the first answer_score, and its prompt embeds worked examples an echoing judge can reproduce.

Change

All four now read the last match, prefer an explicitly labelled decision where the prompt defines one (Decision: for FRAMES, correct: for BrowseComp), and tolerate case and spacing. Falling back to a default now logs the unparsed response — previously a default was indistinguishable from a genuine wrong answer.

Tests

tests/test_grader_parsing.py — the judge is stubbed, so no network or API keys. Covers each grader's happy path, the specific corruptions above, and the unparseable case.

Six of the thirteen fail against the previous implementation:

FAILED test_simpleqa_ignores_capital_inside_a_word
FAILED test_simpleqa_reads_the_verdict_after_an_echoed_rubric
FAILED test_frames_reads_labelled_decision_not_the_explanation
FAILED test_frames_unparseable_does_not_raise
FAILED test_browsecomp_is_case_and_space_tolerant
FAILED test_fin_search_reads_the_last_score_when_an_example_is_echoed

Worth discussing

This changes grading, so scores can move — most likely on FRAMES, where the prompt actively invites reasoning before the verdict. If any published FRAMES number was affected by first-match parsing, it would have been biased downward. Might be worth a re-run on a sample to see whether the delta is material.

🤖 Generated with Claude Code