structured-output-compliance-bench

A benchmark for one narrow question: when you ask an LLM API for strict structured output (JSON against a schema), how often do you actually get it — and how much junk comes with it?

Why a dedicated benchmark

General-purpose leaderboards measure reasoning and knowledge. They do not measure the thing that breaks production pipelines: schema violations, prose wrapped around JSON, truncated objects, or field-name drift. If your application parses the response programmatically, a single wrapped code block is a crash, not a stylistic difference.

What we measure

For each candidate endpoint and each test case:

1. Schema compliance — does the response parse and validate against the target JSON Schema / Pydantic model, with zero repair?
2. Verbosity overhead — response token count beyond the schema-mandated content (prose preambles, markdown fences, repeated fields).
3. Field fidelity — required keys present, enum values in-range, nested depth respected.

Compliance is binary per request; verbosity and fidelity are reported as distributions, not single numbers.

What we deliberately do NOT measure

• End-to-end latency. Latency is dominated by model generation on the serving side and by network path; publishing headline latency numbers invites cargo-cult comparisons. We publish only the methodology for decomposing latency (gateway overhead vs generation), because the overhead component is the only part an operator actually controls.
• Open-ended quality judgments. Compliance is mechanical; this bench stays mechanical.

Headline results (2026-07 run, anonymized)

Metric	Endpoint A (gateway under test)	Endpoint B (western commercial baseline)
Schema compliance	5/5 case groups	5/5 case groups
Median verbosity overhead	baseline − 35%	baseline

Full case groups, prompts, schemas, and raw responses are under results/. Methodology is in methodology.md; the scoring rubric is in rubric.md.

Honest caveats

• n = 5 case groups × 20 requests each. Small sample; we report it as a parity signal, not a domination claim.
• Endpoint B was a widely-used commercial baseline chosen by the testing customer; identity withheld pending permission to name.
• Compliance results depend on the models served at test time and can drift as models change. Re-run before you rely.

Reproduce

python3 runner/run_bench.py --cases cases/ --out results/raw/
python3 runner/score.py --raw results/raw/ --rubric rubric.md

The runner speaks the OpenAI-compatible chat-completions interface; point --base-url and --api-key at any compatible endpoint, including your own deployment of anything.

---

Maintained by NovaRouteAI.