Results have no scores except task_success_rate

[yongyct]

Hi,

I'm trying to run this benchmark against a openai_compatible API, specifically using Mistral's console with the below code added to llm/factory.py

        if os.getenv("MISTRAL_API_KEY"):
            configs["devstral-medium-latest"] = ModelConfig(
                name="devstral-medium-latest",
                provider_type="openai_compatible",
                api_key=os.getenv("MISTRAL_API_KEY"),
                base_url="https://api.mistral.ai/v1",
                model_name="devstral-medium-latest"
            )

In this case, I omitted providing the openrouter api key, and provided the azure openai envs for LLM as judge

After running the benchmark across 2 days, i get weird results back in my benchmark_results_...json

{
  "./tasks/mcpbench_tasks_single_runner_format.json": {
    "task_completion_score": 0.0,
    "tool_selection_score": 0.0,
    "planning_effectiveness_and_efficiency_score": 0.0,
    "task_fulfillment": 0.0,
    "grounding": 0.0,
    "tool_appropriateness": 0.0,
    "parameter_accuracy": 0.0,
    "dependency_awareness": 0.0,
    "parallelism_and_efficiency": 0.0,
    "input_schema_compliance": 0.0,
    "valid_tool_name_rate": 0.0,
    "tool_call_success_rate": 0.0,
    "avg_execution_time": 0.0,
    "avg_agent_execution_time": 0.0,
    "avg_evaluation_time": 0.0,
    "task_success_rate": 1.0,
    "avg_total_rounds": 0.0,
    "avg_tool_calls_per_task": 0.0,
    "avg_output_tokens": 0.0,
    "avg_prompt_tokens": 0.0,
    "avg_total_tokens": 0.0
  },
  "./tasks/mcpbench_tasks_multi_2server_runner_format.json": {
    "task_completion_score": 0.0,
    "tool_selection_score": 0.0,
    "planning_effectiveness_and_efficiency_score": 0.0,
    "task_fulfillment": 0.0,
    "grounding": 0.0,
    "tool_appropriateness": 0.0,
    "parameter_accuracy": 0.0,
    "dependency_awareness": 0.0,
    "parallelism_and_efficiency": 0.0,
    "input_schema_compliance": 0.0,
    "valid_tool_name_rate": 0.0,
    "tool_call_success_rate": 0.0,
    "avg_execution_time": 0.0,
    "avg_agent_execution_time": 0.0,
    "avg_evaluation_time": 0.0,
    "task_success_rate": 0.9666666666666667,
    "avg_total_rounds": 0.0,
    "avg_tool_calls_per_task": 0.0,
    "avg_output_tokens": 0.0,
    "avg_prompt_tokens": 0.0,
    "avg_total_tokens": 0.0
  },
  "./tasks/mcpbench_tasks_multi_3server_runner_format.json": {
    "task_completion_score": 0.0,
    "tool_selection_score": 0.0,
    "planning_effectiveness_and_efficiency_score": 0.0,
    "task_fulfillment": 0.0,
    "grounding": 0.0,
    "tool_appropriateness": 0.0,
    "parameter_accuracy": 0.0,
    "dependency_awareness": 0.0,
    "parallelism_and_efficiency": 0.0,
    "input_schema_compliance": 0.0,
    "valid_tool_name_rate": 0.0,
    "tool_call_success_rate": 0.0,
    "avg_execution_time": 0.0,
    "avg_agent_execution_time": 0.0,
    "avg_evaluation_time": 0.0,
    "task_success_rate": 1.0,
    "avg_total_rounds": 0.0,
    "avg_tool_calls_per_task": 0.0,
    "avg_output_tokens": 0.0,
    "avg_prompt_tokens": 0.0,
    "avg_total_tokens": 0.0
  }
}

am i doing/configuring something wrong for adding other openai compatible endpoints?

[ZhentingWang]

Hi, thanks for your question. Could you share your log for running the benchmark if it is available? I can take a look and help you to analyze the cause.