Skip to content

feat: add OrcaRouter as a named VLMEvalKit API provider - #332

Open
nissrin2020ali-ux wants to merge 1 commit into
NVIDIA:mainfrom
nissrin2020ali-ux:feat/orcarouter-provider
Open

feat: add OrcaRouter as a named VLMEvalKit API provider#332
nissrin2020ali-ux wants to merge 1 commit into
NVIDIA:mainfrom
nissrin2020ali-ux:feat/orcarouter-provider

Conversation

@nissrin2020ali-ux

Copy link
Copy Markdown

Motivation

VLMEvalKit evaluates large vision-language models against a large registry of named API providers — Together AI, SiliconFlow, MiniMax, and others each have a first-class wrapper in vlmeval/api/ plus preset entries in vlmeval/config.py. When benchmarking with an external model gateway, users currently have no way to pick OrcaRouter without treating it as an anonymous custom --base-url. This PR adds OrcaRouter as a first-class provider in the same style as the existing Together AI integration, so Cosmos Reasoner benchmarks can run against OrcaRouter's model namespace directly.

OrcaRouter is an OpenAI-compatible AI gateway built for both models and agents. Like OpenRouter, it exposes a provider/model namespace across many models — but it also combines adaptive routing, automatic failover, zero-markup inference, observability, guardrails, and agent-tool governance behind the same endpoint. Adding orcarouter as a first-class provider means this project's users can use that stack directly, without treating OrcaRouter as an anonymous custom base URL.

It also runs gateway-level, zero-trust security for AI agents on the same endpoint — screening every prompt/response and governing every tool call on a default-deny basis, with no application code changes.

Changes

  • vlmeval/api/orcarouter.py — new OrcaRouterAPI wrapper, modeled directly on vlmeval/api/together.py: OpenAI-compatible chat completions with image_url (base64 or URL) support, ORCAROUTER_API_KEY env key resolution, retries, and a configurable api_base defaulting to https://api.orcarouter.ai/v1/chat/completions.
  • vlmeval/api/__init__.py — import and export OrcaRouterAPI.
  • vlmeval/config.py — four preset api_models entries: OrcaRouter_Gemini-3.1-Flash-Lite, OrcaRouter_Gemini-3.5-Flash, OrcaRouter_Claude-Sonnet-5, and OrcaRouter_DeepSeek-V4-Flash-Vision.
  • docs/en/Quickstart.md — add ORCAROUTER_API_KEY to the documented .env API-key list.

Verification

  • python -m py_compile and flake8 --max-line-length=99 pass on the new files (config.py is pre-commit-excluded).
  • L3 live round-trips through the new OrcaRouterAPI class against the live https://api.orcarouter.ai/v1 endpoint succeeded for all four preset models (google/gemini-3.1-flash-lite, google/gemini-3.5-flash, anthropic/claude-sonnet-5, deepseek/deepseek-v4-flash-vision-exp), each returning ret_code=0 with a valid completion.

Discord: discord.gg/YEubt8enRA · X: https://x.com/OrcaRouter

I'm an engineer on the OrcaRouter team.

Add a first-class OrcaRouter API wrapper mirroring the existing TogetherAI
integration, plus four preset models in the api_models registry and the
ORCAROUTER_API_KEY entry in the Quickstart env list.

Signed-off-by: nissrin2020ali-ux <nissrin2020ali-ux@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant