Description
Running mlx_lm.generate with default sampling parameters, a model looped
into repeated text with no warning. Looking at generate.py on main,
other sampling parameters have explicit DEFAULT_* constants defined at
module level (DEFAULT_TEMP, DEFAULT_TOP_P, DEFAULT_MIN_P,
DEFAULT_TOP_K, etc.), but there is no DEFAULT_REPETITION_PENALTY.
SERVER.md also documents defaults for repetition_context_size (20),
presence_penalty (0.0), and frequency_penalty (0.0), but doesn't state
a default for repetition_penalty itself.
To reproduce
- Run
mlx_lm.generate --model <model> --prompt "<prompt>" with no
--repetition-penalty flag set.
- With some prompts/models, generation degenerates into repeated text
or loops.
Expected behavior
Either a sensible default repetition penalty applied out of the box, or
at minimum a clear note in the docs/CLI help that repetition_penalty is
off by default and users should set it explicitly for longer generations.
Question
Is the lack of a default intentional (e.g. to match upstream sampling
behavior exactly)? If so, a one-line doc/help-text note would still help
first-time users avoid this.
Environment
mlx-lm version: [will confirm]
Python version: [will confirm]
OS: macOS [version], chip [M1/M2/M3/...]
Happy to submit a small docs PR to add this note if useful.
Description
Running
mlx_lm.generatewith default sampling parameters, a model loopedinto repeated text with no warning. Looking at
generate.pyonmain,other sampling parameters have explicit
DEFAULT_*constants defined atmodule level (
DEFAULT_TEMP,DEFAULT_TOP_P,DEFAULT_MIN_P,DEFAULT_TOP_K, etc.), but there is noDEFAULT_REPETITION_PENALTY.SERVER.mdalso documents defaults forrepetition_context_size(20),presence_penalty(0.0), andfrequency_penalty(0.0), but doesn't statea default for
repetition_penaltyitself.To reproduce
mlx_lm.generate --model <model> --prompt "<prompt>"with no--repetition-penaltyflag set.or loops.
Expected behavior
Either a sensible default repetition penalty applied out of the box, or
at minimum a clear note in the docs/CLI help that repetition_penalty is
off by default and users should set it explicitly for longer generations.
Question
Is the lack of a default intentional (e.g. to match upstream sampling
behavior exactly)? If so, a one-line doc/help-text note would still help
first-time users avoid this.
Environment
mlx-lm version: [will confirm]
Python version: [will confirm]
OS: macOS [version], chip [M1/M2/M3/...]
Happy to submit a small docs PR to add this note if useful.