Tool call not working with Mistral Small 24B

[mmoody-vv]

I'm trying to use Tool Calling with stelterlab/Mistral-Small-24B-Instruct-2501-AWQ. I've started the model with --enable-auto-tool-choice --tool-call-parser mistral --chat-template /opt/vllm/app/templates/tool_chat_template_mistral_parallel.jinja. As a test, I tried running the example code here: https://docs.mistral.ai/capabilities/function_calling/

I'm getting this as a result:
ChatCompletion(id='chatcmpl', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='[{"name": "retrieve_payment_status", "arguments": {"transaction_id": "T1001"}}]', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], reasoning_content=None), stop_reason=None)], created=1738701730, model='stelterlab/Mistral-Small-24B-Instruct-2501-AWQ', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=27, prompt_tokens=266, total_tokens=293, completion_tokens_details=None, prompt_tokens_details=None), prompt_logprobs=None)

The tool call is going under message, not in tool_calls. I looked through the vLLM code, and I think what's happening is the [TOOL CALLS] tag isn't being applied for some reason, so the parser isn't parsing.

I'm looking for help here. If I'm right, is there something easy that can fix that? If I'm wrong, any ideas on what is happening? Thanks in advance for any help you can offer.

[LianxinGao]

watch this: #8515

[alew3]

@mmoody-vv I'm having a hard time to get it running with quantization, can you share the full parameter list you are using?

[mmoody-vv]

What parameter list are you referring to? To start the server or in my call to it?

[alew3]

to get the server running. Finally worked for me (but not function calling)

docker run
--runtime nvidia
-e VLLM_USE_V1=1
--gpus all
--ipc=host
-p "${MODEL_PORT}:8000"
--env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}"
-v "${HF_HOME}:/root/.cache/huggingface"
vllm/vllm-openai:latest
--model casperhansen/mistral-small-24b-instruct-2501-awq
--enforce-eager
--tool-call-parser mistral
--enable-auto-tool-choice

[alew3]

When passing in the template , I get a different error:
jinja2.exceptions.TemplateError: Only user and assistant roles are supported, with the exception of an initial optional system message!

[LianxinGao]

you need --tokenizer-mode mistral

[mmoody-vv]

Is that to me or to alew above?

[alew3]

@mmoody-vv it was for me!
@LianxinGao thanks! it worked, just the repo I used didn't have the tekken.json file needed, so I copied it from the original repo.

[alew3]

@mmoody-vv got it working with casperhansen/mistral-small-24b-instruct-2501-awq
but the repo has a missing tekken.json file that I copied from the original Mistral repo and I set HF_HUB_OFFLINE=1 so it doesn't overwrite the downloaded model.

docker run \
--runtime nvidia \
-e VLLM_USE_V1=1 \
--ipc=host \
-p "${MODEL_PORT}:8000" \
--env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}" \
--env "HF_HUB_OFFLINE=1" \
-v "${HF_HOME}:/root/.cache/huggingface" \
vllm/vllm-openai:latest \
--model  casperhansen/mistral-small-24b-instruct-2501-awq  \
--enforce-eager \
--tool-call-parser mistral \
--enable-auto-tool-choice \
--tokenizer-mode mistral 

[eschmidbauer]

i think the problem is the awq version doesn't come with the tokenizer
passing this
--tokenizer mistralai/Mistral-Small-24B-Instruct-2501 seemed to solve the issue for me