Hardcoded to use OpenRouter for both bechmarks and judge

[jb510]

I told my OpenClaw to run this skill using /openai-codex/gpt-5.1-codex-mini as a test run of my thinking parameter #12. It rather surprisingly ran the tests using "openrouter/openai-codex/gpt-5.1-codex-mini" which actually failed and resulted in an empty result, but it then passed over the empty files to openrouter/opus-4.5 for judging anyway! (burning some unexpected tokens or I wouldn't have even realized what happened).

I do understand that as an internal tool you want it to be consistent, and hardcoding these things may make sense. However, if it's going to stay that way, it should be documented better in the readme.md.

My preference, and what I'm working on a PR for right now, is the ability to pass in any provider/model that OpenClaw recognizes to be benchmarked and also any provider/model that the user wants to play judge.

I'd obviously keep your public results consistent, but this would allow users to run whatever tests they wanted.

[CameronVetter]

I had a similar problem. I'm trying to test MY model on MY Openclaw instance. I don't use openrouter and I'm not interested in the performance through openrouter.

This PR makes a lot of sense, I'd love to see it merged ASAP.

If there is a concern about the comparison being fair, you could enforce a requirement that the judge model is always through openrouter/anthropic/claude-opus-4.5 to allow a score to be submitted.

[olearycrew]

OpenRouter is no longer hardcoded as the provider over all

[ScuttleBot]

Following up on Brendan's comment — I did a deep dive through the current codebase to confirm the full picture:

Current State (as of main)

Benchmark model (--model)

✅ No longer hardcoded to OpenRouter. The --model flag accepts any provider prefix your OpenClaw instance supports:

./scripts/run.sh --model anthropic/claude-sonnet-4      # Direct Anthropic
./scripts/run.sh --model openrouter/openai/gpt-4o       # Via OpenRouter
./scripts/run.sh --model my-local/llama-3               # Local provider

Judge model (--judge)

✅ Configurable. The --judge flag overrides the default:

# From lib_grading.py
DEFAULT_JUDGE_MODEL = "openrouter/anthropic/claude-opus-4.5"

To use a different judge:

./scripts/run.sh --model anthropic/claude-sonnet-4 --judge anthropic/claude-opus-4.5

Documentation

The README already documents both flags:

• Line 95: --model MODEL — "Model to test"
• Line 96: --judge MODEL — "Judge model for LLM grading (default: openrouter/anthropic/claude-opus-4.5)"

Summary

Both @jb510's and @CameronVetter's use cases are now supported:

• Test any model via any provider
• Use any judge model

The only remaining question is whether the default judge should remain openrouter/anthropic/claude-opus-4.5. Current behavior seems reasonable — most users have OpenRouter configured, and Opus 4.5 is a solid choice for judging.

I think this issue can be closed as resolved. 🦀