Fix evaluate crash on tasks with no stop sequences - #1664
Open
choipilkyu wants to merge 1 commit into
Open
Conversation
`_rstrip_until` calls `min()` on the list of stop-sequence positions, which raises `ValueError: min() iterable argument is empty` when a task specifies none. lm-eval's `ifeval` does exactly that -- `until: []`, because the completion is meant to run to `max_gen_toks` -- as do all of its multilingual variants (`ifeval_ca`, `ifeval_es`, ...). `mlx_lm.evaluate --task ifeval` therefore fails outright rather than producing a score. With no stop sequences there is nothing to truncate at, so the completion should be returned whole. `min(f, default=l)` does that and leaves the non-empty case unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Author
|
Reproduction against from lm_eval.tasks import TaskManager, get_task_dict
from mlx_lm.evaluate import _rstrip_until
task = get_task_dict(["ifeval"], TaskManager())["ifeval"]
task.build_all_requests(limit=1, rank=0, world_size=1)
ctx, opt = task.instances[0].args
print(opt)
_rstrip_until("some completion", opt["until"])
lm-eval 0.4.12, |
michalk8
self-requested a review
August 5, 2026 13:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_rstrip_untilinmlx_lm/evaluate.pycallsmin()on the list of stop-sequence positions. When a task specifies no stop sequences the list is empty and it raises:lm-eval's
ifevaldoes exactly that —until: []inifeval.yaml, because the completion is meant to run tomax_gen_toks(1280) rather than stop at a marker. Every multilingual variant (ifeval_ca,ifeval_es, …) sets it too. So:fails outright instead of producing a score.
Fix
With no stop sequences there is nothing to truncate at, so the completion is returned whole.
min(f, default=l)does that; the non-empty case is unchanged.Tests
Added
TestRstripUntiltotests/test_evaluate.pycovering truncation at the first stop sequence, the earliest of several, no match, the empty-untilscase, and empty-untilson an empty string.