Skip to content

fix(afrobench): pass the document to the XL-Sum prompts - #4411

Open
alleyxd wants to merge 1 commit into
EleutherAI:mainfrom
alleyxd:fix/afrobench-xlsum-document
Open

alleyxd wants to merge 1 commit into
EleutherAI:mainfrom
alleyxd:fix/afrobench-xlsum-document

Conversation

@alleyxd

@alleyxd alleyxd commented Oct 18, 2026

Copy link
Copy Markdown

All 36 AfroBench XL-Sum tasks (xlsum_<lang>_prompt_{1,2,3}, 12 languages) ask the model to summarize the word text instead of the article. The generator in afrobench/xlsum/utils.py builds the prompts with r"{{'text'}}\n": the quotes make Jinja render the string literal 'text' rather than the text field, and the raw string keeps \n as two characters.

Rendering xlsum_swahili_prompt_2 with apply_template and a doc whose text is THE ARTICLE BODY, at d6de816:

'Summarize the document below in triple backticks and return only the summary and nothing else.\n```text```\\n'

After:

'Summarize the document below in triple backticks and return only the summary and nothing else.\n```THE ARTICLE BODY```\n'

The fix is in the generator (all five prompt templates, including the two that have no YAMLs checked in); the 36 YAMLs were regenerated with python utils.py --mode prompt_1 prompt_2 prompt_3, so only doc_to_text changes in each. Since every score from these tasks changes, the three prompt_*/xlsum bases go from version 1.0 to 2.0, with a changelog line in the README.

Verification

$ pytest tests/test_afrobench_xlsum.py -q
37 passed
# fix reverted, test kept: 36 failed, 1 passed

$ pytest tests/test_tasks.py -q -k "afrobench or xlsum"
37 passed, 16 deselected

pre-commit run passes on the changed files.

@CLAassistant

CLAassistant commented Oct 18, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants