test(bench): assert the gate's behaviour at the boundary, not a gateway's name

lib/box_qualify_test.sh asserted "gomodel +5.4% -> FAIL" and eight sibling cases
labelled by entrant. Two problems. The gate does not know a gateway's name and
must not: it takes a drift percentage and returns a verdict, so the label was
never the thing under test. And the assertions read as claims about products when
the finding was about a contaminated EC2 box.

Every case is now described by the value under test. EVERY MEASURED NUMBER IS
KEPT, verbatim, because the numbers are the evidence for the bands: the nine
healthy floor drifts (-2.6%..+3.8%), the suspect +5.4%, the eight healthy peak
replay deltas (-13%..+13%), the suspect -86%, the two boxes wrongly terminated
for BEATING their baseline (-5.06%, -7.41%). The healthy populations are now
driven as loops over those exact values rather than as nine near-identical named
lines, and the provenance stays in the comments. Same assertions, same coverage.

lib/box_qualify.sh's header carried the same calibration table as a per-gateway
list; it is now the same figures without the identities, which is also more
honest about what it measures (one box per gateway, and which entrant sat on the
bad box is not what the gate keys on).

site/test.mjs fixtures moved to the synthetic keys the file already uses
elsewhere, and three assertions got STRICTER on the way:

  * the roster test spot-checked that one named entry was present; it now asserts
    set equality against the whole field, so no entry can be special-cased in or
    out of the roster
  * the class test spot-checked one gateway's published cls; it now checks, for
    EVERY gateway, that the published cls is that manifest's own GW_CLASS
  * the committed-sweep helper named one results/perf file and returned null when
    it was absent; it now discovers whatever is on disk

And one real drop-in blocker fixed: the star-snapshot test HARD-FAILED on any
gateway missing from gateways/stars.json, which quietly made adding a gateway a
two-step operation, drop in the directory and the site build goes red until
someone also runs fetch-stars. A star count is decoration, gen-data already
degrades a missing entry to null, so a pending refresh is now a printed note, not
a build break. In exchange the test gained a check it never had: a stars.json key
with no gateway directory is real drift and now fails.

The rest is prose. Roster comments in lib/harness.sh, lib/ingress.sh,
lib/mem_rss_test.sh, site/app.js, charts.py and README.md that listed the field
by name go stale the moment a directory is added or removed; they say what they
meant instead.

Author: MattJackson

README.md

@@ -41,14 +41,14 @@ exact build/commit measured, plus the charts below.
 - **Go** - builds the load generator (`loadgen/`).
 - **Docker** - for the container-based gateways (Bifrost, Kong, GoModel, One-API, …).
 - **Python 3 + matplotlib** - draws the charts (`pip install matplotlib`). Optional; JSON results are written either way.
-- Docker. Every gateway pulls its own pinned official image on first run (see `gateways/versions.env`); helicone and litellm-rust build from pinned source (no arm64 image exists — see their manifests).
+- Docker. Every gateway pulls its own pinned official image on first run; the pin lives in that gateway's own `gateways/<name>/gateway.sh`. A few build from pinned source because no arm64 image exists - each says so in its own manifest header.
 
 **To run the one-click cloud version** (`run-on-ec2.sh`) the *only* extra dependency is **AWS CLI v2**, configured (`aws configure` - creds + a default region). The script launches a fresh Graviton box, installs everything on it, runs the full suite, pulls the results back, and **terminates the box** - nothing to set up, nothing to clean up.
 
 ## Run it - one command, every metric
 
 Clone, then run one script. Everything is at the repo root, and **every gateway provisions itself**
-from the ref pinned in [`gateways/versions.env`](gateways/versions.env) - Docker images, pip, source,
+from the ref pinned in its own [`gateways/<name>/gateway.sh`](gateways/) - Docker images, pip, source,
 or (for a native gateway) its released image's binary. Nothing to fetch by hand for any of them.
 
 ```sh
@@ -83,8 +83,8 @@ One-API to its arm64 tag), so nothing is quietly arm64-only or x86-only.
 
 A gateway that can't be stood up (unreachable, or needs infra a single container can't provide) is
 recorded `served: false` and shown as such - never silently dropped. To pin a different build of any
-gateway, edit its line in `gateways/versions.env` (or override the env var); the exact ref is stamped
-into every result.
+gateway, edit the pin in its own `gateways/<name>/gateway.sh` (or override the env var); the exact ref
+is stamped into every result.
 
 ### How long it takes
 
@@ -234,10 +234,11 @@ functions. The runners are gateway-agnostic; there is nothing else to edit. See
 
 ## Honesty notes (the receipts)
 
-- **Source refs are config, not defaults buried in a script.** Everything is pinned in
-  [`gateways/versions.env`](gateways/versions.env) and overridable; the *actual* version/commit
-  built is written into each result's `build` field. "You used an old branch" is answerable by
-  pointing at the file and the recorded commit.
+- **Source refs are config, not defaults buried in a script.** Each gateway's ref is pinned in its
+  own `gateways/<name>/gateway.sh`, next to that manifest's disclosure of what it deviates from and
+  why, and every pin is overridable from the environment; the *actual* version/commit built is
+  written into each result's `build` field. "You used an old branch" is answerable by pointing at the
+  manifest and the recorded commit.
 - **Each gateway is launched the only way it actually serves the endpoint.** For example,
   LiteLLM-Rust's `/v1/messages` route only serves the `azure_ai` provider *and* only serves at all
   under its `python-config` reader (the lean env config returns `400`) - verified against its own

charts.py

@@ -5,8 +5,8 @@
 
 Nothing is hard-coded: every number is read from results/<suite>/<gateway>.json (written by the
 runners). Bars are colored by MEASUREMENT — a neutral highlight goes to whichever gateway measured
-best on the metric, so if busbar loses, busbar isn't highlighted. The highlight is deliberately not a
-brand color.
+best on the metric, so the operator's own entry is highlighted only when it actually wins. The
+highlight is deliberately not a brand color.
 
 Add a chart = append one `Chart(...)` to CHARTS below. Add a gateway = it shows up automatically
 once it has a result file (label/order from GATEWAYS). Run after the benchmark:
@@ -198,7 +198,7 @@ def _mpl():
 # Each gateway is fully defined by its own dir: gateways/<key>/gateway.sh declares GW_DISPLAY (label),
 # GW_LANG (color bucket), and GW_REPO (linked from the name in the report table). Add a dir → it shows
 # up in the charts/tables/run-lists; delete it → it's gone everywhere. A gateway only appears in a
-# chart once it also has a result file this run. Alphabetical by key — deliberately NOT busbar-first;
+# chart once it also has a result file this run. Alphabetical by key, so no entrant is seated first;
 # order here is only load order, every chart + table sorts by the MEASURED value.
 def _manifest_meta():
     out = {}
@@ -598,8 +598,8 @@ def _mem_annot(r):
                         if r.get("_xlate_ingress") else None,
     ),
     # Governance is intentionally NOT charted on the neutral board: the governed suite is a
-    # non-default, busbar-only launch (only busbar's manifest wires it), so a comparison would
-    # spotlight busbar and read "not tested" for the rest. Governance overhead belongs on the
+    # non-default launch wired by a single manifest, so a comparison would spotlight that one
+    # entrant and read "not tested" for the rest. Governance overhead belongs on the
     # advocacy site. The governed suite still runs and its data is kept for that use.
 ]
 
@@ -1323,7 +1323,8 @@ def us_cell(r, field):
     lines.append("Method: added latency = gateway p99 − direct-to-mock p99 at concurrency 1; RPS "
                  "ceiling = highest sustained req/s with p99 < 1 s and <0.1% errors; RSS idle = after "
                  "first 200, peak = under sustained load. Same box, same mock, same load, one gateway "
-                 "at a time. Source refs pinned in `gateways/versions.env`; the built commit is in each row.")
+                 "at a time. Each gateway's source ref is pinned in its own `gateways/<name>/gateway.sh`; "
+                 "the built commit is in each row.")
     lines.append("")
     lines.append(f"<sub>Page + charts regenerated **{RENDER_TS}** from the raw `results/*.json`.</sub>")
     return "\n".join(lines) + "\n"

lib/box_qualify.sh

@@ -11,13 +11,14 @@
 # gateway anywhere in the path. It measures the BOX, not the gateway. In the 2026-07-25 field run vs
 # the run before it, the per-box median drift of that floor was:
 #
-#     agentgateway -2.6%   apisix -2.6%   bifrost +2.7%   busbar +1.3%   gomodel +5.4%
-#     kong -2.6%   litellm-python +0.6%   litellm-rust -2.5%   one-api -1.2%   portkey +1.3%
-#     tensorzero +3.8%
+#     -2.6%  -2.6%  -2.5%  -1.2%  +0.6%  +1.3%  +1.3%  +2.7%  +3.8%   (ten healthy boxes)
+#     +5.4%                                                             (one suspect box)
 #
-# gomodel's box was contaminated — and gomodel's ABSOLUTE floor was 77-81us, INSIDE the healthy
-# population's 73-82us. A static absolute threshold could not have caught it, and did not: the run
-# published gomodel as a gateway regression when the honest finding was "we measured it on a bad box".
+# One box per gateway, one gateway per box; which entrant happened to sit on the bad box is not what
+# the gate keys on and is deliberately not recorded here. The suspect box was contaminated - and its
+# ABSOLUTE floor was 77-81us, INSIDE the healthy population's 73-82us. A static absolute threshold
+# could not have caught it, and did not: the run published a BOX fault as a gateway regression when
+# the honest finding was "we measured it on a bad box".
 # That is the single fact that shapes this whole file: the gate must be RELATIVE TO THAT BOX'S OWN
 # HISTORY, not to a global constant. The absolute envelope below is retained only as a coarse
 # catch-all for grossly bad hardware.
@@ -38,9 +39,8 @@
 #   throughput to the recorded value. It exercises the real measured path (gateway + mock + loadgen)
 #   and is FAR more discriminating than the floor. Same run pair, rps_max_proxy:
 #
-#     healthy: agentgateway -13%  tensorzero -12%  apisix -8%  bifrost -4%  busbar -0%
-#              litellm-rust +8%   portkey -3%      kong +13%              -> max |delta| 13%
-#     suspect: gomodel -86%
+#     healthy: -13%  -12%  -8%  -4%  -3%  -0%  +8%  +13%      -> max |delta| 13%
+#     suspect: -86%
 #
 #   Floor separates healthy-from-bad by 3.8% vs 5.4% (thin). Peak replay separates 13% vs 86%
 #   (enormous). Stage 2 is therefore the strong gate; stage 1 is the cheap early one.
@@ -61,10 +61,10 @@
 #   the rig happens to do is a rubber stamp, not a measurement-quality gate.
 #
 # Why the two bands differ by so much: they measure different things with different natural variance.
-# A 4% band on stage 2 would false-reject 6 of the 8 HEALTHY gateways above and put the rig into
-# permanent relaunch churn; 25% still catches the gomodel class (-86%) with an enormous margin.
+# A 4% band on stage 2 would false-reject 6 of the 8 HEALTHY boxes above and put the rig into
+# permanent relaunch churn; 25% still catches the -86% class with an enormous margin.
 #
-# Stage 1 at 4% is DELIBERATELY TIGHT and the margin is THIN: the worst healthy box (tensorzero) drifted
+# Stage 1 at 4% is DELIBERATELY TIGHT and the margin is THIN: the worst healthy box drifted
 # +3.8% against a 4.0% band — 0.2 percentage points of headroom. That is accepted on purpose. A
 # deviation larger than 4% must never be shrugged off as normal box variance; a false reject is cheap
 # (bounded box relaunch, self-healing) while a false accept publishes bad data. Err tight.
@@ -85,7 +85,7 @@ BQ_FLOOR_RATCHET_PCT="${BQ_FLOOR_RATCHET_PCT:-8.0}"
 # ABSOLUTE sanity envelope (us). Coarse catch-all for grossly bad hardware and the ONLY floor gate
 # available on a first run with no history. The healthy population sat at 73-82us on arm64
 # m7g.4xlarge; this envelope is deliberately much wider so it never false-rejects — it exists to catch
-# a box that is 2x slow, not to catch the gomodel class (which it demonstrably cannot).
+# a box that is 2x slow, not to catch the inside-the-envelope class above (which it demonstrably cannot).
 BQ_FLOOR_ABS_MIN_US="${BQ_FLOOR_ABS_MIN_US:-45}"
 BQ_FLOOR_ABS_MAX_US="${BQ_FLOOR_ABS_MAX_US:-115}"
 # JITTER / stability. UNCALIBRATED — we have no published per-sample spread from the historical runs,
@@ -277,7 +277,7 @@ def lane(key, cell_key=None, conc_key=None, src_key=None, prefix="", only_gw=Fal
 #   and a per-gateway baseline is a category error. It is also actively harmful: with one or two
 #   historical samples each, a gateway's baseline is a random draw from that population, so a gateway
 #   that happened to draw a fast box rejects perfectly normal ones while a gateway that drew a slow
-#   one accepts almost anything. Observed live on 2026-07-25: apisix carried a 73us baseline and was
+#   one accepts almost anything. Observed live on 2026-07-25: one box carried a 73us baseline and was
 #   rejected at 77us, a value squarely inside that same run's healthy spread of 70-80us, while a
 #   genuinely bad box at 89us was correctly caught. Pooling gives ~one sample per gateway per run, so
 #   the reference converges immediately and means the same thing for everyone.