Skip to content

-m take precedence over model arg being passed #297

@damoonsh

Description

@damoonsh

I have extended MultiTurn and customized the env_response. Based on the given model I want customize sampling_args but the -m cmd takes precedence over model arg passed to my env. This causes me to pass my model name twice (once in -m and once in -a). Could we give precedence to model arg instead of -m inside eval.py file?

here is cmd:

uv run vf-eval spiral-bench \
  -b https://openrouter.ai/api/v1 \
  -m google/gemma-3-27b-it:free \
  -n 1 \
  -r 1 \
  -k AK \
  -a '{
    "max_turns": 3,
    "user_model": "qwen/qwen3-14b:free",
    "eval_model": "google/gemma-3-27b-it:free",
    "judge_model": "meta-llama/llama-3.3-70b-instruct:free",
    "injection_prob": 0.5,
    "custom_judge_sampling_args": {"max_tokens":1024},
    "custom_user_sampling_args": {"max_tokens":2048}
  }'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions