Skip to content

fix(mmlu_pro): extract "Answer: X" and final-letter responses instead of marking them invalid - #4417

Open
yucheng-du wants to merge 2 commits into
EleutherAI:mainfrom
yucheng-du:fix/3110-mmlu-pro-answer-extraction
Open

yucheng-du wants to merge 2 commits into
EleutherAI:mainfrom
yucheng-du:fix/3110-mmlu-pro-answer-extraction

Conversation

@yucheng-du

Copy link
Copy Markdown

Addresses #3110 (answer extraction only)
Refs #2702 (item 2)

MMLU-Pro's custom-extract filter only matches answer is (X), so responses like Answer: C, or ones ending in a bare letter, become [invalid] and score 0. For The answer is (C)., Answer: C and ANSWER: H, main extracts ['C', '[invalid]', '[invalid]']; this PR gives ['C', 'C', 'H'].

The new ExtractAnswerFilter in mmlu_pro/utils.py follows the reference implementation in order: answer is first, then Answer:, then the last standalone letter A to J. Unmatched responses stay [invalid] (no random guess). Answers matched by the first pattern are unchanged, so scores can only stay the same or go up (not measured on model outputs). The task version goes from 3.1 to 4.1, with a README changelog line.

Tests: 28 new cases in tests/test_mmlu_pro.py, 13 of which fail on main. With this change, the MMLU-Pro, filter and KorMedMCQA tests (51) pass, and Ruff passes.

Out of scope: prompts, max_gen_toks, and the ignore_case filters suggested in #3110.

Prepared with OpenAI Codex and Claude Code; tests run locally.

@CLAassistant

CLAassistant commented Oct 19, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants