Skip to content

Fix evaluate crash on tasks with no stop sequences - #1664

Open
choipilkyu wants to merge 1 commit into
ml-explore:mainfrom
choipilkyu:fix-rstrip-until-empty
Open

Fix evaluate crash on tasks with no stop sequences#1664
choipilkyu wants to merge 1 commit into
ml-explore:mainfrom
choipilkyu:fix-rstrip-until-empty

Conversation

@choipilkyu

Copy link
Copy Markdown

Problem

_rstrip_until in mlx_lm/evaluate.py calls min() on the list of stop-sequence positions. When a task specifies no stop sequences the list is empty and it raises:

ValueError: min() iterable argument is empty

lm-eval's ifeval does exactly that — until: [] in ifeval.yaml, because the completion is meant to run to max_gen_toks (1280) rather than stop at a marker. Every multilingual variant (ifeval_ca, ifeval_es, …) sets it too. So:

mlx_lm.evaluate --model <any> --task ifeval

fails outright instead of producing a score.

Fix

With no stop sequences there is nothing to truncate at, so the completion is returned whole. min(f, default=l) does that; the non-empty case is unchanged.

Tests

Added TestRstripUntil to tests/test_evaluate.py covering truncation at the first stop sequence, the earliest of several, no match, the empty-untils case, and empty-untils on an empty string.

$ python -m pytest tests/test_evaluate.py -q
......                                                                   [100%]
6 passed

`_rstrip_until` calls `min()` on the list of stop-sequence positions, which
raises `ValueError: min() iterable argument is empty` when a task specifies
none. lm-eval's `ifeval` does exactly that -- `until: []`, because the
completion is meant to run to `max_gen_toks` -- as do all of its multilingual
variants (`ifeval_ca`, `ifeval_es`, ...). `mlx_lm.evaluate --task ifeval`
therefore fails outright rather than producing a score.

With no stop sequences there is nothing to truncate at, so the completion
should be returned whole. `min(f, default=l)` does that and leaves the
non-empty case unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@choipilkyu

Copy link
Copy Markdown
Author

Reproduction against main (e5baded), using the request lm-eval actually builds for ifeval — no model needed, since the failure is in post-processing:

from lm_eval.tasks import TaskManager, get_task_dict
from mlx_lm.evaluate import _rstrip_until

task = get_task_dict(["ifeval"], TaskManager())["ifeval"]
task.build_all_requests(limit=1, rank=0, world_size=1)
ctx, opt = task.instances[0].args
print(opt)
_rstrip_until("some completion", opt["until"])
{'until': [], 'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 1280}
Traceback (most recent call last):
  File "mlx_lm/evaluate.py", line 38, in _rstrip_until
    return s[: min(f)]
               ~~~^^^
ValueError: min() iterable argument is empty

generate_until passes opt["until"] straight through at evaluate.py:360 with no normalisation, so the empty list from the task config reaches _rstrip_until unchanged.

lm-eval 0.4.12, lm_eval/tasks/ifeval/ifeval.yaml.

@michalk8
michalk8 self-requested a review August 5, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant