Discussion: how should we handle different providers & hardware on the main results

[olearycrew]

Today, our official results, which show by default, are all run from the same hardware and the same inference provider of OpenRouter.

However, of course, there's been many submissions from unofficial results from different hardware, different providers, etc.

So I'm of two minds about how to handle this. On the one hand, I want to be able to provide a consistent platform to compare models against one another. And I think that hardware and providers can obviously be variables that impact that. Thus the current methodology of controlling for those variables.

But on the other hand, I want it to represent a real world view into how people are using OpenClaw. That's why today we can show you every submission if you check unofficial results. You'll see all the submissions from everyone around the world.

The question for this issue is, and I'd like some input on, is how do we display results by default? Should we continue the way we're going? Should we add other harnesses or providers to the official results? Or should we by default just show you all results?

[KarthiDreamr]

OpenRouter solves this problem for you, by routing you through the inferences that run how the opensource models providers intended it to be ran,

read more at https://x.com/OpenRouter/status/2032127261709676991?s=20

TL;DR

• A well-known struggle for open source models is tool calling accuracy, even for the top provider, particularly when a model is only a few weeks old. (And we observe that it's very rough in the first 5 days.)
• "Auto Exacto" is now live, and on by default for tool-calling requests.
  Over the last few days, OpenRouter has reduced tool error rates by 15-90% across providers automatically.

[olearycrew]

Yeah I mean that's one reason that I went with OpenRouter.

But there are some labs who either:

1. Aren't on OpenROuter
2. Launch models in stealth or even public but don't immediately launch on OpenRouter

[KarthiDreamr]

for other labs, you can use their official api platform key etc. just mentioning them somewhere in a hidden plane in bench is enough. people mostly care about average performance consistency in benchmarks to make decisions in their tasks etc.

[changshivek]

Is it possible for labs whose models aren't on OpenRouter to be included in the Official Results?
Like they could provide Base URLs and API Keys, and you could run the evaluation through your own automated pipeline. This way, the hardware and environment stay consistent with your other official runs, and the official leaderboard is opened to a broader range .