Total: 0 models

[DuanyangYuan]

(mcpbench) /mcp-bench$ python run_benchmark.py --list-models
2025-11-03 22:14:14,803 - mcp_modules.tool_cache - INFO - Tool cache initialized at cache/tool_cache.db with TTL=0 hours
2025-11-03 22:14:14,803 - benchmark.runner - INFO - Tool cache enabled: dir=./cache, TTL=permanent
2025-11-03 22:14:14,804 - utils.local_server_config - INFO - Loaded 28 local server configurations
2025-11-03 22:14:14,804 - utils.local_server_config - INFO - Loaded 4 API keys
Available models:

Total: 0 models

Usage examples:
python run_benchmark.py --models MODEL_NAME

In the mcp-bench directory I created the .env file like this:
cat > .env << EOF
export OPENROUTER_API_KEY="your_openrouterkey_here"
EOF

and also applied the changes under “Optional: Using only OpenRouter API”, yet it still shows “Total: 0 models”.

[ZhentingWang]

Hi, Thanks for your question. Could you kindly check if source .env was run prior to python run_benchmark.py --list-models?

[DuanyangYuan]

Hi, Thanks for your question. Could you kindly check if source .env was run prior to python run_benchmark.py --list-models?

I replaced openai with deepseek in the code, and the runtime results are as follows. This is obviously incorrect. I believe this has nothing to do with the replacement of openai with deepseek. Do you know the reason?

{
  "task_completion_score": 6.463636363636364,
  "tool_selection_score": 8.094545454545456,
  "planning_effectiveness_and_efficiency_score": 5.89090909090909,
  "task_fulfillment": 5.669090909090908,
  "grounding": 7.258181818181821,
  "tool_appropriateness": 8.280000000000001,
  "parameter_accuracy": 7.90909090909091,
  "dependency_awareness": 6.290909090909091,
  "parallelism_and_efficiency": 5.490909090909091,
  "input_schema_compliance": 0.9916413859989004,
  "valid_tool_name_rate": 0.9987868056921765,
  "tool_call_success_rate": 0.8990534183023965,
  "avg_execution_time": 434.2801296147433,
  "avg_agent_execution_time": 416.09046988053757,
  "avg_evaluation_time": 122.61230173110962,
  "task_success_rate": 0.9821428571428571,
  "avg_total_rounds": 5.745454545454545,
  "avg_tool_calls_per_task": 28.0,
  "avg_output_tokens": 4089.0,
  "avg_prompt_tokens": 209414.70909090908,
  "avg_total_tokens": 213503.70909090908
}

[ZhentingWang]

Hi, what is the judge model used in your experiments? o4-mini should be used as the judge model.

[DuanyangYuan]

o4-mini should be used as the judge model.

deepseek-chat be used as the judge model.

[ZhentingWang]

Hi, Because score distributions vary across judge models, we standardize on o4-mini as the judge to make the results consistent and comparable.

[DuanyangYuan]

Hi, Because score distributions vary across judge models, we standardize on o4-mini as the judge to make the results consistent and comparable.

Although the models differ, I think it only affects how high or low the final scores are, not that the entire distribution of results is wrong. For example, task_completion_score should be between 0 and 1, but now it's already greater than 1.

[ZhentingWang]

I see, sorry for the confusion, the raw score returned by the LLM judge is scaled from 0-10, it should be normalized to 0-1 via divided by 10. We will highlight it and revise the code to automate the normalization later.

[DuanyangYuan]

I see, sorry for the confusion, the raw score returned by the LLM judge is scale from 0-10, it should be normalized to 0-1 via divided by 10. We will highlight it and revise the code to automate the normalization later.

Okay, I understand. This means that the scores such as 'task_completion_score': 6.463636363636364, 'tool_selection_score': 8.094545454545456, 'planning_effectiveness_and_efficiency_score': 5.89090909090909, 'task_fulfillment': 5.669090909090908, 'grounding': 7.258181818181821, 'tool_appropriateness': 8.280000000000001, 'parameter_accuracy': 7.90909090909091, 'dependency_awareness': 6.290909090909091, and 'parallelism_and_efficiency': 5.490909090909091 all need to be divided by 10, while the other data points like 'input_schema_compliance': 0.9916413859989004, 'valid_tool_name_rate': 0.9987868056921765, 'tool_call_success_rate': 0.8990534183023965, 'avg_execution_time': 434.2801296147433, 'avg_agent_execution_time': 416.09046988053757, 'avg_evaluation_time': 122.61230173110962, 'task_success_rate': 0.9821428571428571, 'avg_total_rounds': 5.745454545454545, 'avg_tool_calls_per_task': 28.0, 'avg_output_tokens': 4089.0, 'avg_prompt_tokens': 209414.70909090908, and 'avg_total_tokens': 213503.70909090908 are all normal.

[ZhentingWang]

yes, that's correct

[DuanyangYuan]

OK. Thank you for your reply.

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "Accenture/mcp-bench" ***@***.***&gt;; 发送时间:&nbsp;2025年11月9日(星期天) 上午10:05 ***@***.***&gt;; ***@***.******@***.***&gt;; 主题:&nbsp;Re: [Accenture/mcp-bench] Total: 0 models (Issue #23) ZhentingWang left a comment (Accenture/mcp-bench#23) yes, that's correct — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***&gt;