A visual benchmark for testing and scoring LLM models on tool-calling capabilities, designed specifically for evaluating which models work best with Hermes Agent (NousResearch's tool-calling model family).
Built on top of ToolCall-15 — enhanced with Hermes-native tool format support, latency tracking, cost estimation, and context window awareness.
Test 10 models across 15 scenarios — color-coded pass/partial/fail with live scoring.
Three test suites covering different capabilities: General tool calling, Business/Enterprise workflows, and Memory & Retrieval.
| Business / Enterprise | Memory & Retrieval |
|---|---|
 |
 |
Ranked model comparison with latency, context window, and cost metrics.
Auto-detects your machine specs and tells you in plain English which models you can run locally — with Hermes Agent compatibility ratings for each model.
Adjust the workload selector to account for what else runs on your machine:
Toggle to reveal full specs, RAM requirements, quantization info, and Hermes format compatibility.
The benchmark runs 15 fixed scenarios across 5 categories and scores each model on multiple dimensions:
- Tool Selection — Can it pick the right tool?
- Parameter Precision — Does it pass the right arguments?
- Multi-Step Chains — Can it thread data across steps?
- Restraint & Refusal — Does it know when NOT to act?
- Error Recovery — What happens when things go wrong?
Each scenario is scored as 2 (pass), 1 (partial), or 0 (fail). Final score is the average of 5 category percentages.
- Speed & Latency — Average and total response time per model across all scenarios
- Context Window — Known context sizes for common models, with env overrides
- Cost Estimation — Estimated cost per benchmark run based on known provider pricing
- Avg Turns — How many conversation turns the model needs to complete each scenario
This benchmark includes first-class support for the Hermes tool-calling format used by NousResearch models (Hermes 2 Pro, Hermes 3). Select hermes in the Tools Format dropdown to:
- Inject tool definitions into the system prompt inside
<tools></tools>XML tags - Parse
<tool_call>{"name": "...", "arguments": {...}}</tool_call>responses - Format tool results with
<tool_response></tool_response>tags - Handle multi-turn conversations in the Hermes ChatML format
| Format | When to Use |
|---|---|
default |
Models served with OpenAI-compatible tool support (Ollama with native tools, vLLM with --tool-call-parser hermes, OpenRouter) |
hermes |
Raw Hermes models via llama.cpp or vLLM without automatic tool-call conversion |
lfm |
Liquid Foundation Models |
| Provider | Key | Notes |
|---|---|---|
| OpenRouter | openrouter |
Cloud API — best for Hermes 3 405B, comparison models |
| Ollama | ollama |
Local — ollama pull hermes3:8b handles Hermes format natively |
| llama.cpp | llamacpp |
Local — use --chat-template hermes2pro flag |
| vLLM | vllm |
Local/cloud — use --tool-call-parser hermes --enable-auto-tool-choice |
| mlx_lm | mlx |
Apple Silicon local inference |
| LM Studio | lmstudio |
GUI-based local inference |
- Node.js 20 or newer
- npm
- At least one reachable OpenAI-compatible provider with a Hermes model
npm install
cp .env.example .envEdit .env with your providers and models.
Quick start with Ollama:
ollama pull hermes3:8bOLLAMA_HOST=http://localhost:11434
LLM_MODELS=ollama:hermes3:8bCompare Hermes variants via OpenRouter:
OPENROUTER_API_KEY=your-key
LLM_MODELS=openrouter:nousresearch/hermes-3-llama-3.1-8b,openrouter:nousresearch/hermes-3-llama-3.1-70b
LLM_MODELS_2=openrouter:openai/gpt-4.1,openrouter:anthropic/claude-sonnet-4Local Hermes with vLLM:
vllm serve NousResearch/Hermes-3-Llama-3.1-8B --tool-call-parser hermes --enable-auto-tool-choiceVLLM_HOST=http://localhost:8000
LLM_MODELS=vllm:NousResearch/Hermes-3-Llama-3.1-8Bnpm run devOpen http://localhost:3000.
npm run lint
npm run typecheck- The runner advances scenario-by-scenario, not model-by-model.
- Score cards show accuracy percentage, average latency, average turns, context window, and estimated cost.
- The config button opens a modal for generation parameters and Tools Format (default/hermes/lfm).
Shift+Clicka scenario header to rerun only that scenario across all models.- Clicking a cell opens the raw trace for inspection.
The core benchmark score. See METHODOLOGY.md for the full 15-scenario specification.
Measured end-to-end per scenario (includes all turns). Displayed as average latency across scenarios. Lower is better. Critical for agent responsiveness.
Pulled from a built-in lookup table of known Hermes and popular models, overridable via MODEL_CONTEXT_WINDOWS env var. Larger context = better for complex multi-step agent tasks.
Estimated from known per-million-token rates for cloud providers. Local models show no cost. Useful for comparing cloud Hermes variants against each other.
The Tools Format selector lets you test whether a model works better with:
- Native OpenAI tool calling (
default) - Hermes ChatML prompt injection (
hermes) - LFM format (
lfm)
This reveals which serving configuration produces the best results for each model.
Each model gets a Hermes Agent compatibility rating based on 5 factors:
| Factor | What It Measures |
|---|---|
| Format Support | Native Hermes ChatML (<tool_call> XML) vs OpenAI JSON adapter |
| System Prompt | Follows Hermes system prompt conventions |
| Multi-Turn Chains | Handles sequential tool-call turn structure |
| Hermes Features | Skill creation, cron scheduling, MCP, sub-agent delegation |
| Community Tested | Verified in real Hermes Agent usage |
Ratings: Excellent (Hermes 3 family — native format, all features), Good (works reliably via adapter), Basic (simple calls only), Untested.
The "My Hardware" tab auto-detects your system and answers: Can I run this locally?
- Scans CPU, RAM, GPU via system commands (macOS/Linux/Windows)
- Accounts for your workload (Light / Normal / Heavy) to estimate available RAM
- Shows which models fit comfortably, which are tight, and which need the cloud
- Recommends the best model for your specific machine
- app/ — Next.js app router entry points and styles.
- components/dashboard.tsx — Benchmark UI and live event handling.
- app/api/run/route.ts — Streams benchmark progress over SSE.
- lib/benchmark.ts — Benchmark spec, mocked tools, scoring logic, and extended metrics.
- lib/orchestrator.ts — Runs scenarios, captures traces and timing.
- lib/llm-client.ts — OpenAI-compatible client with Hermes/LFM format adapters.
- lib/models.ts — Provider config, model metadata (context window, cost).
- lib/hardware.ts — Hardware detection, model compatibility engine, Hermes Agent compatibility database.
- app/api/hardware/route.ts — Server-side system hardware scanning.
- docs/images/ — Screenshots for documentation.
- This isolates tool-use behavior under a fixed tool schema — not a general intelligence benchmark.
- Mocked tools measure orchestration quality, not live service quality.
- Cost estimates are rough approximations based on typical token counts.
- Latency includes network overhead and varies by provider infrastructure.
Based on ToolCall-15 by stevibe, released under the MIT License.
MIT License. See LICENSE.