Skip to content

Commit d7cfe1f

Browse files
ochafikochafik
authored
docs: add docs/function-calling.md to lighten server/README.md's plight (ggml-org#12069)
1 parent a82c9e7 commit d7cfe1f

File tree

2 files changed

+393
-374
lines changed

2 files changed

+393
-374
lines changed

Diff for: ‎docs/function-calling.md

+390
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,390 @@
1+
# Function Calling
2+
3+
[chat.h](../common/chat.h) (https://github.com/ggml-org/llama.cpp/pull/9639) adds support for [OpenAI-style function calling](https://platform.openai.com/docs/guides/function-calling) and is used in:
4+
- `llama-server` when started w/ `--jinja` flag
5+
- `llama-cli` (WIP: https://github.com/ggml-org/llama.cpp/pull/11556)
6+
7+
## Universal support w/ Native & Generic handlers
8+
9+
Function calling is supported for all models (see https://github.com/ggml-org/llama.cpp/pull/9639):
10+
11+
- Native tool call formats supported:
12+
- Llama 3.1 / 3.3 (including builtin tools support - tool names for `wolfram_alpha`, `web_search` / `brave_search`, `code_interpreter`), Llama 3.2
13+
- Functionary v3.1 / v3.2
14+
- Hermes 2/3, Qwen 2.5
15+
- Qwen 2.5 Coder (WIP: https://github.com/ggml-org/llama.cpp/pull/12034)
16+
- Mistral Nemo
17+
- Firefunction v2
18+
- Command R7B
19+
- DeepSeek R1 (WIP / seems reluctant to call any tools?)
20+
21+
- Generic tool call is supported when the template isn't recognized by native format handlers (you'll see `Chat format: Generic` in the logs).
22+
- Use `--chat-template-file` to override the template when appropriate (see examples below)
23+
- Generic support may consume more tokens and be less efficient than a model's native format.
24+
25+
<details>
26+
<summary>Show some common templates and which format handler they use</summary>
27+
28+
| Template | Format |
29+
|----------|--------|
30+
| Almawave-Velvet-14B.jinja | Hermes 2 Pro |
31+
| AtlaAI-Selene-1-Mini-Llama-3.1-8B.jinja | Llama 3.x |
32+
| CohereForAI-aya-expanse-8b.jinja | Generic |
33+
| CohereForAI-c4ai-command-r-plus-default.jinja | Generic |
34+
| CohereForAI-c4ai-command-r-plus-rag.jinja | Generic |
35+
| CohereForAI-c4ai-command-r-plus-tool_use.jinja | Generic |
36+
| CohereForAI-c4ai-command-r7b-12-2024-default.jinja | Command R7B (extract reasoning) |
37+
| CohereForAI-c4ai-command-r7b-12-2024-rag.jinja | Command R7B (extract reasoning) |
38+
| CohereForAI-c4ai-command-r7b-12-2024-tool_use.jinja | Command R7B (extract reasoning) |
39+
| CohereForAI-c4ai-command-r7b-12-2024.jinja | Generic |
40+
| DavieLion-Llama-3.2-1B-SPIN-iter3.jinja | Generic |
41+
| Delta-Vector-Rei-12B.jinja | Mistral Nemo |
42+
| EpistemeAI-Mistral-Nemo-Instruct-12B-Philosophy-Math.jinja | Mistral Nemo |
43+
| FlofloB-83k_continued_pretraining_Qwen2.5-0.5B-Instruct_Unsloth_merged_16bit.jinja | Hermes 2 Pro |
44+
| FlofloB-test_continued_pretraining_Phi-3-mini-4k-instruct_Unsloth_merged_16bit.jinja | Generic |
45+
| HelpingAI-HAI-SER.jinja | Generic |
46+
| HuggingFaceTB-SmolLM2-1.7B-Instruct.jinja | Generic |
47+
| HuggingFaceTB-SmolLM2-135M-Instruct.jinja | Generic |
48+
| HuggingFaceTB-SmolLM2-360M-Instruct.jinja | Generic |
49+
| INSAIT-Institute-BgGPT-Gemma-2-27B-IT-v1.0.jinja | Generic |
50+
| Ihor-Text2Graph-R1-Qwen2.5-0.5b.jinja | Hermes 2 Pro |
51+
| Infinigence-Megrez-3B-Instruct.jinja | Generic |
52+
| Josephgflowers-TinyLlama_v1.1_math_code-world-test-1.jinja | Generic |
53+
| LGAI-EXAONE-EXAONE-3.5-2.4B-Instruct.jinja | Generic |
54+
| LGAI-EXAONE-EXAONE-3.5-7.8B-Instruct.jinja | Generic |
55+
| LatitudeGames-Wayfarer-12B.jinja | Generic |
56+
| Magpie-Align-Llama-3-8B-Magpie-Align-v0.1.jinja | Generic |
57+
| Magpie-Align-Llama-3.1-8B-Magpie-Align-v0.1.jinja | Generic |
58+
| MaziyarPanahi-calme-3.2-instruct-78b.jinja | Generic |
59+
| MiniMaxAI-MiniMax-Text-01.jinja | Generic |
60+
| MiniMaxAI-MiniMax-VL-01.jinja | Generic |
61+
| NaniDAO-deepseek-r1-qwen-2.5-32B-ablated.jinja | DeepSeek R1 (extract reasoning) |
62+
| NexaAIDev-Octopus-v2.jinja | Generic |
63+
| NousResearch-Hermes-2-Pro-Llama-3-8B-default.jinja | Generic |
64+
| NousResearch-Hermes-2-Pro-Llama-3-8B-tool_use.jinja | Hermes 2 Pro |
65+
| NousResearch-Hermes-2-Pro-Mistral-7B-default.jinja | Generic |
66+
| NousResearch-Hermes-2-Pro-Mistral-7B-tool_use.jinja | Hermes 2 Pro |
67+
| NousResearch-Hermes-3-Llama-3.1-70B-default.jinja | Generic |
68+
| NousResearch-Hermes-3-Llama-3.1-70B-tool_use.jinja | Hermes 2 Pro |
69+
| NovaSky-AI-Sky-T1-32B-Flash.jinja | Hermes 2 Pro |
70+
| NovaSky-AI-Sky-T1-32B-Preview.jinja | Hermes 2 Pro |
71+
| OnlyCheeini-greesychat-turbo.jinja | Generic |
72+
| Orenguteng-Llama-3.1-8B-Lexi-Uncensored-V2.jinja | Llama 3.x |
73+
| OrionStarAI-Orion-14B-Chat.jinja | Generic |
74+
| PowerInfer-SmallThinker-3B-Preview.jinja | Generic |
75+
| PrimeIntellect-INTELLECT-1-Instruct.jinja | Generic |
76+
| Qwen-QVQ-72B-Preview.jinja | Generic |
77+
| Qwen-QwQ-32B-Preview.jinja | Hermes 2 Pro |
78+
| Qwen-Qwen1.5-7B-Chat.jinja | Generic |
79+
| Qwen-Qwen2-7B-Instruct.jinja | Generic |
80+
| Qwen-Qwen2-VL-72B-Instruct.jinja | Generic |
81+
| Qwen-Qwen2-VL-7B-Instruct.jinja | Generic |
82+
| Qwen-Qwen2.5-0.5B.jinja | Hermes 2 Pro |
83+
| Qwen-Qwen2.5-1.5B-Instruct.jinja | Hermes 2 Pro |
84+
| Qwen-Qwen2.5-14B-Instruct-1M.jinja | Hermes 2 Pro |
85+
| Qwen-Qwen2.5-14B.jinja | Hermes 2 Pro |
86+
| Qwen-Qwen2.5-32B-Instruct.jinja | Hermes 2 Pro |
87+
| Qwen-Qwen2.5-32B.jinja | Hermes 2 Pro |
88+
| Qwen-Qwen2.5-3B-Instruct.jinja | Hermes 2 Pro |
89+
| Qwen-Qwen2.5-72B-Instruct.jinja | Hermes 2 Pro |
90+
| Qwen-Qwen2.5-7B-Instruct-1M.jinja | Hermes 2 Pro |
91+
| Qwen-Qwen2.5-7B-Instruct.jinja | Hermes 2 Pro |
92+
| Qwen-Qwen2.5-7B.jinja | Hermes 2 Pro |
93+
| Qwen-Qwen2.5-Coder-32B-Instruct.jinja | Hermes 2 Pro |
94+
| Qwen-Qwen2.5-Coder-7B-Instruct.jinja | Hermes 2 Pro |
95+
| Qwen-Qwen2.5-Math-1.5B.jinja | Hermes 2 Pro |
96+
| Qwen-Qwen2.5-Math-7B-Instruct.jinja | Hermes 2 Pro |
97+
| Qwen-Qwen2.5-VL-3B-Instruct.jinja | Hermes 2 Pro |
98+
| Qwen-Qwen2.5-VL-72B-Instruct.jinja | Hermes 2 Pro |
99+
| Qwen-Qwen2.5-VL-7B-Instruct.jinja | Hermes 2 Pro |
100+
| RWKV-Red-Team-ARWKV-7B-Preview-0.1.jinja | Hermes 2 Pro |
101+
| SakanaAI-TinySwallow-1.5B-Instruct.jinja | Hermes 2 Pro |
102+
| SakanaAI-TinySwallow-1.5B.jinja | Hermes 2 Pro |
103+
| Sao10K-70B-L3.3-Cirrus-x1.jinja | Llama 3.x |
104+
| SentientAGI-Dobby-Mini-Leashed-Llama-3.1-8B.jinja | Llama 3.x |
105+
| SentientAGI-Dobby-Mini-Unhinged-Llama-3.1-8B.jinja | Llama 3.x |
106+
| Steelskull-L3.3-Damascus-R1.jinja | Llama 3.x |
107+
| Steelskull-L3.3-MS-Nevoria-70b.jinja | Llama 3.x |
108+
| Steelskull-L3.3-Nevoria-R1-70b.jinja | Llama 3.x |
109+
| THUDM-glm-4-9b-chat.jinja | Generic |
110+
| THUDM-glm-edge-1.5b-chat.jinja | Generic |
111+
| Tarek07-Progenitor-V1.1-LLaMa-70B.jinja | Llama 3.x |
112+
| TheBloke-FusionNet_34Bx2_MoE-AWQ.jinja | Generic |
113+
| TinyLlama-TinyLlama-1.1B-Chat-v1.0.jinja | Generic |
114+
| UCLA-AGI-Mistral7B-PairRM-SPPO-Iter3.jinja | Generic |
115+
| ValiantLabs-Llama3.1-8B-Enigma.jinja | Llama 3.x |
116+
| abacusai-Fewshot-Metamath-OrcaVicuna-Mistral.jinja | Generic |
117+
| ai21labs-AI21-Jamba-1.5-Large.jinja | Generic |
118+
| allenai-Llama-3.1-Tulu-3-405B-SFT.jinja | Generic |
119+
| allenai-Llama-3.1-Tulu-3-405B.jinja | Generic |
120+
| allenai-Llama-3.1-Tulu-3-8B.jinja | Generic |
121+
| arcee-ai-Virtuoso-Lite.jinja | Hermes 2 Pro |
122+
| arcee-ai-Virtuoso-Medium-v2.jinja | Hermes 2 Pro |
123+
| arcee-ai-Virtuoso-Small-v2.jinja | Hermes 2 Pro |
124+
| avemio-GRAG-NEMO-12B-ORPO-HESSIAN-AI.jinja | Generic |
125+
| bespokelabs-Bespoke-Stratos-7B.jinja | Hermes 2 Pro |
126+
| bfuzzy1-acheron-m1a-llama.jinja | Generic |
127+
| bofenghuang-vigogne-2-70b-chat.jinja | Generic |
128+
| bytedance-research-UI-TARS-72B-DPO.jinja | Generic |
129+
| bytedance-research-UI-TARS-7B-DPO.jinja | Generic |
130+
| bytedance-research-UI-TARS-7B-SFT.jinja | Generic |
131+
| carsenk-phi3.5_mini_exp_825_uncensored.jinja | Generic |
132+
| cyberagent-DeepSeek-R1-Distill-Qwen-14B-Japanese.jinja | DeepSeek R1 (extract reasoning) |
133+
| cyberagent-DeepSeek-R1-Distill-Qwen-32B-Japanese.jinja | DeepSeek R1 (extract reasoning) |
134+
| databricks-dbrx-instruct.jinja | Generic |
135+
| deepseek-ai-DeepSeek-Coder-V2-Instruct.jinja | Generic |
136+
| deepseek-ai-DeepSeek-Coder-V2-Lite-Base.jinja | Generic |
137+
| deepseek-ai-DeepSeek-Coder-V2-Lite-Instruct.jinja | Generic |
138+
| deepseek-ai-DeepSeek-R1-Distill-Llama-70B.jinja | DeepSeek R1 (extract reasoning) |
139+
| deepseek-ai-DeepSeek-R1-Distill-Llama-8B.jinja | DeepSeek R1 (extract reasoning) |
140+
| deepseek-ai-DeepSeek-R1-Distill-Qwen-1.5B.jinja | DeepSeek R1 (extract reasoning) |
141+
| deepseek-ai-DeepSeek-R1-Distill-Qwen-14B.jinja | DeepSeek R1 (extract reasoning) |
142+
| deepseek-ai-DeepSeek-R1-Distill-Qwen-32B.jinja | DeepSeek R1 (extract reasoning) |
143+
| deepseek-ai-DeepSeek-R1-Distill-Qwen-7B.jinja | DeepSeek R1 (extract reasoning) |
144+
| deepseek-ai-DeepSeek-R1-Zero.jinja | DeepSeek R1 (extract reasoning) |
145+
| deepseek-ai-DeepSeek-R1.jinja | DeepSeek R1 (extract reasoning) |
146+
| deepseek-ai-DeepSeek-V2-Lite.jinja | Generic |
147+
| deepseek-ai-DeepSeek-V2.5.jinja | DeepSeek R1 (extract reasoning) |
148+
| deepseek-ai-DeepSeek-V3.jinja | DeepSeek R1 (extract reasoning) |
149+
| deepseek-ai-deepseek-coder-33b-instruct.jinja | Generic |
150+
| deepseek-ai-deepseek-coder-6.7b-instruct.jinja | Generic |
151+
| deepseek-ai-deepseek-coder-7b-instruct-v1.5.jinja | Generic |
152+
| deepseek-ai-deepseek-llm-67b-chat.jinja | Generic |
153+
| deepseek-ai-deepseek-llm-7b-chat.jinja | Generic |
154+
| dicta-il-dictalm2.0-instruct.jinja | Generic |
155+
| ehristoforu-Falcon3-8B-Franken-Basestruct.jinja | Hermes 2 Pro |
156+
| fireworks-ai-llama-3-firefunction-v2.jinja | FireFunction v2 |
157+
| godlikehhd-alpaca_data_sampled_ifd_new_5200.jinja | Hermes 2 Pro |
158+
| godlikehhd-alpaca_data_score_max_0.7_2600.jinja | Hermes 2 Pro |
159+
| google-gemma-2-27b-it.jinja | Generic |
160+
| google-gemma-2-2b-it.jinja | Generic |
161+
| google-gemma-2-2b-jpn-it.jinja | Generic |
162+
| google-gemma-7b-it.jinja | Generic |
163+
| huihui-ai-DeepSeek-R1-Distill-Llama-70B-abliterated.jinja | DeepSeek R1 (extract reasoning) |
164+
| huihui-ai-DeepSeek-R1-Distill-Llama-8B-abliterated.jinja | DeepSeek R1 (extract reasoning) |
165+
| huihui-ai-DeepSeek-R1-Distill-Qwen-14B-abliterated-v2.jinja | DeepSeek R1 (extract reasoning) |
166+
| huihui-ai-DeepSeek-R1-Distill-Qwen-32B-abliterated.jinja | DeepSeek R1 (extract reasoning) |
167+
| huihui-ai-DeepSeek-R1-Distill-Qwen-7B-abliterated-v2.jinja | DeepSeek R1 (extract reasoning) |
168+
| huihui-ai-Qwen2.5-14B-Instruct-1M-abliterated.jinja | Hermes 2 Pro |
169+
| ibm-granite-granite-3.1-8b-instruct.jinja | Generic |
170+
| indischepartij-MiniCPM-3B-OpenHermes-2.5-v2.jinja | Generic |
171+
| inflatebot-MN-12B-Mag-Mell-R1.jinja | Generic |
172+
| jinaai-ReaderLM-v2.jinja | Generic |
173+
| kms7530-chemeng_qwen-math-7b_24_1_100_1_nonmath.jinja | Hermes 2 Pro |
174+
| knifeayumu-Cydonia-v1.3-Magnum-v4-22B.jinja | Mistral Nemo |
175+
| langgptai-qwen1.5-7b-chat-sa-v0.1.jinja | Generic |
176+
| lightblue-DeepSeek-R1-Distill-Qwen-7B-Japanese.jinja | DeepSeek R1 (extract reasoning) |
177+
| mattshumer-Reflection-Llama-3.1-70B.jinja | Generic |
178+
| meetkai-functionary-medium-v3.1.jinja | Functionary v3.1 Llama 3.1 |
179+
| meetkai-functionary-medium-v3.2.jinja | Functionary v3.2 |
180+
| meta-llama-Llama-2-7b-chat-hf.jinja | Generic |
181+
| meta-llama-Llama-3.1-8B-Instruct.jinja | Llama 3.x |
182+
| meta-llama-Llama-3.2-11B-Vision-Instruct.jinja | Llama 3.x |
183+
| meta-llama-Llama-3.2-1B-Instruct.jinja | Llama 3.x |
184+
| meta-llama-Llama-3.2-3B-Instruct.jinja | Llama 3.x |
185+
| meta-llama-Llama-3.3-70B-Instruct.jinja | Llama 3.x |
186+
| meta-llama-Meta-Llama-3-8B-Instruct.jinja | Generic |
187+
| meta-llama-Meta-Llama-3.1-8B-Instruct.jinja | Llama 3.x |
188+
| microsoft-Phi-3-medium-4k-instruct.jinja | Generic |
189+
| microsoft-Phi-3-mini-4k-instruct.jinja | Generic |
190+
| microsoft-Phi-3-small-8k-instruct.jinja | Generic |
191+
| microsoft-Phi-3.5-mini-instruct.jinja | Generic |
192+
| microsoft-Phi-3.5-vision-instruct.jinja | Generic |
193+
| microsoft-phi-4.jinja | Generic |
194+
| migtissera-Tess-3-Mistral-Nemo-12B.jinja | Generic |
195+
| ministral-Ministral-3b-instruct.jinja | Generic |
196+
| mistralai-Codestral-22B-v0.1.jinja | Generic |
197+
| mistralai-Mistral-7B-Instruct-v0.1.jinja | Generic |
198+
| mistralai-Mistral-7B-Instruct-v0.2.jinja | Generic |
199+
| mistralai-Mistral-7B-Instruct-v0.3.jinja | Mistral Nemo |
200+
| mistralai-Mistral-Large-Instruct-2407.jinja | Mistral Nemo |
201+
| mistralai-Mistral-Large-Instruct-2411.jinja | Generic |
202+
| mistralai-Mistral-Nemo-Instruct-2407.jinja | Mistral Nemo |
203+
| mistralai-Mistral-Small-24B-Instruct-2501.jinja | Generic |
204+
| mistralai-Mixtral-8x7B-Instruct-v0.1.jinja | Generic |
205+
| mkurman-Qwen2.5-14B-DeepSeek-R1-1M.jinja | Hermes 2 Pro |
206+
| mlabonne-AlphaMonarch-7B.jinja | Generic |
207+
| mlx-community-Josiefied-Qwen2.5-0.5B-Instruct-abliterated-v1-float32.jinja | Hermes 2 Pro |
208+
| mlx-community-Qwen2.5-VL-7B-Instruct-8bit.jinja | Hermes 2 Pro |
209+
| mobiuslabsgmbh-DeepSeek-R1-ReDistill-Qwen-1.5B-v1.1.jinja | DeepSeek R1 (extract reasoning) |
210+
| netcat420-MFANNv0.20.jinja | Generic |
211+
| netcat420-MFANNv0.24.jinja | Generic |
212+
| netease-youdao-Confucius-o1-14B.jinja | Hermes 2 Pro |
213+
| nvidia-AceMath-7B-RM.jinja | Hermes 2 Pro |
214+
| nvidia-Eagle2-1B.jinja | Hermes 2 Pro |
215+
| nvidia-Eagle2-9B.jinja | Hermes 2 Pro |
216+
| nvidia-Llama-3.1-Nemotron-70B-Instruct-HF.jinja | Llama 3.x |
217+
| onnx-community-DeepSeek-R1-Distill-Qwen-1.5B-ONNX.jinja | DeepSeek R1 (extract reasoning) |
218+
| open-thoughts-OpenThinker-7B.jinja | Hermes 2 Pro |
219+
| openchat-openchat-3.5-0106.jinja | Generic |
220+
| pankajmathur-orca_mini_v6_8b.jinja | Generic |
221+
| princeton-nlp-Mistral-7B-Base-SFT-RDPO.jinja | Generic |
222+
| princeton-nlp-Mistral-7B-Instruct-DPO.jinja | Generic |
223+
| princeton-nlp-Mistral-7B-Instruct-RDPO.jinja | Generic |
224+
| prithivMLmods-Bellatrix-Tiny-1.5B-R1.jinja | Hermes 2 Pro |
225+
| prithivMLmods-Bellatrix-Tiny-1B-R1.jinja | Llama 3.x |
226+
| prithivMLmods-Bellatrix-Tiny-1B-v3.jinja | Generic |
227+
| prithivMLmods-Bellatrix-Tiny-3B-R1.jinja | Llama 3.x |
228+
| prithivMLmods-Blaze-14B-xElite.jinja | Generic |
229+
| prithivMLmods-Calcium-Opus-14B-Elite2-R1.jinja | Hermes 2 Pro |
230+
| prithivMLmods-Calme-Ties-78B.jinja | Generic |
231+
| prithivMLmods-Calme-Ties2-78B.jinja | Generic |
232+
| prithivMLmods-Calme-Ties3-78B.jinja | Generic |
233+
| prithivMLmods-ChemQwen2-vL.jinja | Generic |
234+
| prithivMLmods-GWQ2b.jinja | Generic |
235+
| prithivMLmods-LatexMind-2B-Codec.jinja | Generic |
236+
| prithivMLmods-Llama-3.2-6B-AlgoCode.jinja | Llama 3.x |
237+
| prithivMLmods-Megatron-Opus-14B-Exp.jinja | Hermes 2 Pro |
238+
| prithivMLmods-Megatron-Opus-14B-Stock.jinja | Hermes 2 Pro |
239+
| prithivMLmods-Megatron-Opus-7B-Exp.jinja | Hermes 2 Pro |
240+
| prithivMLmods-Omni-Reasoner-Merged.jinja | Hermes 2 Pro |
241+
| prithivMLmods-Omni-Reasoner4-Merged.jinja | Hermes 2 Pro |
242+
| prithivMLmods-Primal-Opus-14B-Optimus-v1.jinja | Hermes 2 Pro |
243+
| prithivMLmods-QwQ-Math-IO-500M.jinja | Hermes 2 Pro |
244+
| prithivMLmods-Qwen-7B-Distill-Reasoner.jinja | DeepSeek R1 (extract reasoning) |
245+
| prithivMLmods-Qwen2.5-1.5B-DeepSeek-R1-Instruct.jinja | Hermes 2 Pro |
246+
| prithivMLmods-Qwen2.5-14B-DeepSeek-R1-1M.jinja | Hermes 2 Pro |
247+
| prithivMLmods-Qwen2.5-32B-DeepSeek-R1-Instruct.jinja | Hermes 2 Pro |
248+
| prithivMLmods-Qwen2.5-7B-DeepSeek-R1-1M.jinja | Hermes 2 Pro |
249+
| prithivMLmods-Triangulum-v2-10B.jinja | Hermes 2 Pro |
250+
| qingy2024-Falcon3-2x10B-MoE-Instruct.jinja | Hermes 2 Pro |
251+
| rubenroy-Zurich-14B-GCv2-5m.jinja | Hermes 2 Pro |
252+
| rubenroy-Zurich-7B-GCv2-5m.jinja | Hermes 2 Pro |
253+
| silma-ai-SILMA-Kashif-2B-Instruct-v1.0.jinja | Generic |
254+
| simplescaling-s1-32B.jinja | Hermes 2 Pro |
255+
| sometimesanotion-Lamarck-14B-v0.7.jinja | Hermes 2 Pro |
256+
| sonthenguyen-zephyr-sft-bnb-4bit-DPO-mtbr-180steps.jinja | Generic |
257+
| sthenno-tempesthenno-icy-0130.jinja | Generic |
258+
| sumink-qwft.jinja | Hermes 2 Pro |
259+
| teknium-OpenHermes-2.5-Mistral-7B.jinja | Generic |
260+
| thirdeyeai-elevate360m.jinja | Generic |
261+
| tiiuae-Falcon3-10B-Instruct.jinja | Hermes 2 Pro |
262+
| unsloth-DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit.jinja | DeepSeek R1 (extract reasoning) |
263+
| unsloth-DeepSeek-R1-Distill-Llama-8B.jinja | DeepSeek R1 (extract reasoning) |
264+
| unsloth-DeepSeek-R1.jinja | DeepSeek R1 (extract reasoning) |
265+
| unsloth-Mistral-Small-24B-Instruct-2501-unsloth-bnb-4bit.jinja | Generic |
266+
| upstage-solar-pro-preview-instruct.jinja | Generic |
267+
| whyhow-ai-PatientSeek.jinja | Generic |
268+
| xwen-team-Xwen-72B-Chat.jinja | Hermes 2 Pro |
269+
| xwen-team-Xwen-7B-Chat.jinja | Hermes 2 Pro |
270+
271+
This table can be generated with:
272+
273+
```bash
274+
./build/bin/test-chat ../minja/build/tests/*.jinja 2>/dev/null
275+
```
276+
277+
</details>
278+
279+
# Usage - need tool-aware Jinja template
280+
281+
First, start a server with any model, but make sure it has a tools-enabled template: you can verify this by inspecting the `chat_template` or `chat_template_tool_use` properties in `http://localhost:8080/props`).
282+
283+
Here are some models known to work (w/ chat template override when needed):
284+
285+
```shell
286+
# Native support:
287+
288+
llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
289+
llama-server --jinja -fa -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K_L
290+
llama-server --jinja -fa -hf bartowski/functionary-small-v3.2-GGUF:Q4_K_M
291+
llama-server --jinja -fa -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M
292+
293+
# Native support for DeepSeek R1 works best w/ our own template (official template buggy)
294+
295+
llama-server --jinja -fa -hf bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF:Q6_K_L \
296+
--chat-template-file models/templates/llama-cpp-deepseek-r1.jinja
297+
298+
llama-server --jinja -fa -hf bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF:Q4_K_M \
299+
--chat-template-file models/templates/llama-cpp-deepseek-r1.jinja
300+
301+
# Native support requires the right template for these GGUFs:
302+
303+
llama-server --jinja -fa -hf bartowski/Hermes-2-Pro-Llama-3-8B-GGUF:Q4_K_M \
304+
--chat-template-file <( python scripts/get_chat_template.py NousResearch/Hermes-2-Pro-Llama-3-8B tool_use )
305+
306+
llama-server --jinja -fa -hf bartowski/Hermes-3-Llama-3.1-8B-GGUF:Q4_K_M \
307+
--chat-template-file <( python scripts/get_chat_template.py NousResearch/Hermes-3-Llama-3.1-8B tool_use )
308+
309+
llama-server --jinja -fa -hf bartowski/firefunction-v2-GGUF -hff firefunction-v2-IQ1_M.gguf \
310+
--chat-template-file <( python scripts/get_chat_template.py fireworks-ai/llama-3-firefunction-v2 tool_use )
311+
312+
llama-server --jinja -fa -hf bartowski/c4ai-command-r7b-12-2024-GGUF:Q6_K_L \
313+
--chat-template-file <( python scripts/get_chat_template.py CohereForAI/c4ai-command-r7b-12-2024 tool_use )
314+
315+
# Generic format support
316+
llama-server --jinja -fa -hf bartowski/phi-4-GGUF:Q4_0
317+
llama-server --jinja -fa -hf bartowski/gemma-2-2b-it-GGUF:Q8_0
318+
llama-server --jinja -fa -hf bartowski/c4ai-command-r-v01-GGUF:Q2_K
319+
```
320+
321+
> [!TIP]
322+
> If there is no official `tool_use` Jinja template, you may want to set `--chat-template chatml` to use a default that works with many models (YMMV!), or write your own (e.g. we provide a custom [llama-cpp-deepseek-r1.jinja](../models/templates/llama-cpp-deepseek-r1.jinja) for DeepSeek R1 distills)
323+
324+
Test in CLI (or with any library / software that can use OpenAI-compatible API backends):
325+
326+
```bash
327+
curl http://localhost:8080/v1/chat/completions -d '{
328+
"model": "gpt-3.5-turbo",
329+
"tools": [
330+
{
331+
"type":"function",
332+
"function":{
333+
"name":"python",
334+
"description":"Runs code in an ipython interpreter and returns the result of the execution after 60 seconds.",
335+
"parameters":{
336+
"type":"object",
337+
"properties":{
338+
"code":{
339+
"type":"string",
340+
"description":"The code to run in the ipython interpreter."
341+
}
342+
},
343+
"required":["code"]
344+
}
345+
}
346+
}
347+
],
348+
"messages": [
349+
{
350+
"role": "user",
351+
"content": "Print a hello world message with python."
352+
}
353+
]
354+
}'
355+
```
356+
357+
<details>
358+
<summary>Show output</summary>
359+
360+
```json
361+
{
362+
"choices": [
363+
{
364+
"finish_reason": "tool",
365+
"index": 0,
366+
"message": {
367+
"content": null,
368+
"tool_calls": [
369+
{
370+
"name": "python",
371+
"arguments": "{\"code\":\" \\nprint(\\\"Hello, World!\\\")\"}"
372+
}
373+
],
374+
"role": "assistant"
375+
}
376+
}
377+
],
378+
"created": 1727287211,
379+
"model": "gpt-3.5-turbo",
380+
"object": "chat.completion",
381+
"usage": {
382+
"completion_tokens": 16,
383+
"prompt_tokens": 44,
384+
"total_tokens": 60
385+
},
386+
"id": "chatcmpl-Htbgh9feMmGM0LEH2hmQvwsCxq3c6Ni8"
387+
}
388+
```
389+
390+
</details>

0 commit comments

Comments
 (0)