fix: stamp the commit that ran, and group the board by instrument everywhere

THE STAMP NAMED THE WRONG COMMIT. run-on-ec2.sh derived BENCH_ENGINE_COMMIT from
the operator's local HEAD while the boxes fetch $BENCH_COMMIT. Those are equal
only when BENCH_COMMIT is left to default - and pinning it is exactly what you do
when re-measuring gateways onto the harness the rest of the board already used.

So the thirteen gateways re-measured pinned to 80030c2, specifically so the
board would be ONE instrument, came back stamped 7fd350e. They ran the pinned
tree; only the label was wrong. The board then concluded it was mixed and blanked
busbar-150 - the one row genuinely on the pinned engine. Measurements right,
label wrong, conclusion inverted, and nothing looked broken.

THE INSTRUMENT, NOT THE COMMIT, IN EVERY READER. C8 already grouped commits whose
built binaries are byte-identical, but gen-data compared raw shas and bench-audit
filtered on a sha prefix. Three parts of one pipeline disagreed about what "the
same engine" means, so C8 could pass a board that gen-data then marked mixed and
bench-audit refused to audit. They now all resolve through
instrument-equivalence.json. The per-row stamp still reports the exact commit:
the instrument decides comparability, the sha still identifies the tree.

80030c2 and 7fd350e are attested as one instrument on BUILT-ARTIFACT evidence -
both build otb to e97eb008... - which is the only evidence that file admits.

AND ONE AUDIT INVARIANT WAS ACCUSING THE SEARCH OF ITS OWN DESIGN.
check_no_rung_fails_below_one_already_carried tested `<= clean` while its
docstring said BELOW. The ascending sweep probes a rung once; if that probe
passes, confirmation at the SAME concurrency fails, and the engine steps down and
publishes the lower rung - the normal way a ceiling is found. apisix
anthropic>anthropic did exactly that (c=16384 passed the sweep, lost both
confirmation windows, 8192 published after holding 4 of 4) and was reported as
"impossible for the gateway alone". A failure AT the proven top is the search
working; only one strictly BELOW it has no gateway-only explanation. The 14
red-before guards in bench-audit_test still each reject their own violation.

Author: MattJackson

bench-audit.py

@@ -127,8 +127,53 @@ def snapshot_paths():
 MAX_RPS_PER_CONNECTION = 20_000
 
 
+def _instrument_map():
+    """commit sha -> instrument id, from site/instrument-equivalence.json.
+
+    THE INSTRUMENT, NOT THE COMMIT - the same grouping C8 and gen-data use. Two commits whose built
+    binaries are byte-identical are one instrument, so pinning the audit to a raw sha would drop rows
+    that were measured by the very same binary. That is how busbar-150 fell out of this audit while
+    being on the board: it was stamped 80030c2f and the rest 7fd350ed, commits that build identical
+    bytes. An entry is only honoured here if it carries the artifact evidence its own file demands.
+    """
+    p = os.path.join(HERE, "site", "instrument-equivalence.json")
+    out = {}
+    try:
+        doc = json.load(open(p))
+    except Exception:
+        return out
+    for inst in doc.get("instruments") or []:
+        commits = inst.get("commits") or []
+        ev = ((inst.get("evidence") or {}).get("otb_release_sha256")) or {}
+        hashes = list(ev.values())
+        if not inst.get("id") or not commits:
+            continue
+        # No artifact evidence, or hashes that disagree, means the entry proves nothing.
+        if len(hashes) < len(commits) or len(set(hashes)) != 1:
+            continue
+        for c in commits:
+            out[c] = inst["id"]
+    return out
+
+
+def _same_instrument(sha, engine, imap):
+    """Does `sha` belong to the instrument identified by `engine` (a sha prefix or an instrument id)?"""
+    if not engine:
+        return True
+    if sha.startswith(engine):
+        return True
+    mine = imap.get(sha)
+    if mine is None:
+        return False
+    # `engine` may be an instrument id, or any sha belonging to that instrument.
+    if mine == engine:
+        return True
+    return any(mine == inst for c, inst in imap.items() if c.startswith(engine))
+
+
 def load(engine=None, gateway=None):
-    """The newest snapshot per gateway, pinned to one engine so a board is audited as a board."""
+    """The newest snapshot per gateway, pinned to one INSTRUMENT so a board is audited as a board."""
+    imap = _instrument_map()
     by_gw = {}
     for f in snapshot_paths():
         try:
@@ -140,7 +185,7 @@ def load(engine=None, gateway=None):
         sha = ((d.get("rig") or {}).get("engine") or {}).get("commit") or ""
         if not gw or (gateway and gw != gateway):
             continue
-        if engine and not sha.startswith(engine):
+        if not _same_instrument(sha, engine, imap):
             continue
         by_gw[gw] = (f, d, sha)
     return by_gw
@@ -484,7 +529,13 @@ def stream_trace(c):
 
 def proven_clean_top(rungs):
     """The highest concurrency the UNCONTAMINATED ascending prefix carried - every rung from the
-    first up to and including it passed, before anything in this cell had failed."""
+    first up to and including it passed, before anything in this cell had failed.
+
+    This is the ASCENDING PREFIX and nothing else: it deliberately stops at the first failed window,
+    so the concurrency it returns is one the cell reached before anything in it had gone wrong. What
+    counts as a violation relative to that top is decided by the caller - see the note there on why a
+    failure AT the top is the search working and only a failure BELOW it is a finding.
+    """
     top = 0
     for r in rungs:
         if r.get("passed") is not True:
@@ -563,7 +614,17 @@ def check_no_rung_fails_below_one_already_carried(name, c):
     clean = proven_clean_top(rungs)
     if clean <= 0:
         return
-    below = [r.get("conc") for r in rungs if r.get("passed") is not True and (r.get("conc") or 0) <= clean]
+    # BELOW, NOT AT. This read `<= clean`, which flags the search's own terminating condition: the
+    # ascending sweep probes a rung once, that probe passes, confirmation at the SAME concurrency then
+    # fails, and the engine steps down and publishes the lower rung. apisix anthropic>anthropic did
+    # exactly that - c=16384 passed the sweep, lost both confirmation windows, and 8192 was published
+    # after holding 4 of 4 - and it was reported as "impossible for the gateway alone".
+    #
+    # A failure AT the top is how a ceiling is found. A failure strictly BELOW a concurrency the cell
+    # already carried is the thing that has no gateway-only explanation, which is what the docstring
+    # above has always said ("cannot fail BELOW one the same cell has already carried") and what the
+    # 2026-07-31 six-cell case actually looked like. The comparison now matches the claim.
+    below = [r.get("conc") for r in rungs if r.get("passed") is not True and (r.get("conc") or 0) < clean]
     if not below:
         return
     absences = c.get("absences") or {}
@@ -1430,7 +1491,7 @@ def main():
         sha = ((d.get("rig") or {}).get("engine") or {}).get("commit") or ""
         if not gw or (args.gateway and gw != args.gateway) or gw in snaps:
             continue
-        if not sha.startswith(engine):
+        if not _same_instrument(sha, engine, _instrument_map()):
             skipped[gw] = sha[:7] or "no engine stamp"
 
     violations = collections.defaultdict(list)

run-on-ec2.sh

@@ -325,7 +325,21 @@ ARCH="${ARCH:-arm64}"
 #
 # Everything else still counts, including UNTRACKED files: a new gateways/<name>/ directory is
 # untracked and absolutely does change what runs.
-BENCH_ENGINE_COMMIT="$(git -C "$HERE" rev-parse HEAD 2>/dev/null || echo '')"
+# THE STAMP MUST NAME THE COMMIT THE BOXES ACTUALLY RAN, which is $BENCH_COMMIT - the tree they fetch
+# by `git ls-tree` - and NOT this machine's HEAD.
+#
+# These are the same value only when BENCH_COMMIT is left to default. Override it to pin a run to an
+# older engine (the whole point of pinning: re-measure lagging gateways on the harness the rest of the
+# board already used) and the two diverge silently: the boxes build the pinned tree and the artifact is
+# stamped with whatever the operator's checkout happened to be sitting on.
+#
+# That is not cosmetic. The engine stamp is C8's entire input. On 2026-08-03 thirteen gateways were
+# re-measured pinned to 80030c2f specifically so the board would be ONE instrument; they were stamped
+# 7fd350ed - a local commit that touched no engine file - and the board concluded it was mixed and
+# blanked the one row that was genuinely on the pinned engine. The measurements were right and the
+# label was wrong, which is the worst shape of this bug: nothing looks broken, and the conclusion is
+# inverted. A stamp that can name a commit the run never used is not provenance.
+BENCH_ENGINE_COMMIT="${BENCH_COMMIT:-$(git -C "$HERE" rev-parse HEAD 2>/dev/null || echo '')}"
 if [ -n "$(git -C "$HERE" status --porcelain -- . ':(exclude)results' 2>/dev/null)" ]; then BENCH_ENGINE_DIRTY=1; else BENCH_ENGINE_DIRTY=0; fi
 export BENCH_ENGINE_COMMIT BENCH_ENGINE_DIRTY
 

site/gen-data.mjs

@@ -26,6 +26,7 @@ import { join, dirname } from "node:path";
 import { fileURLToPath } from "node:url";
 import { createHash } from "node:crypto";
 import { snapshotCellCoords, isStrictSubset, layerScopedMatrix } from "./snapshots.mjs";
+import { instrumentOf } from "./check-consistency.mjs";
 import { sealMetric, sealFrontier, makeSource, SWEEP, UNGATED_LAT_FIELDS, UNGATED_COST_FIELDS, DEFAULT_BOUND_MS, frontierAt, UNGATED_STREAM_FIELDS, isMetricField, zeroNoteFor } from "./seal.mjs";
 
 const HERE = dirname(fileURLToPath(import.meta.url));
@@ -1129,15 +1130,40 @@ if (existsSync(redirects) && OUT !== HERE) copyFileSync(redirects, join(OUT, "_r
 //
 // The board's version is the engine of the most recently measured row, which is the one a re-run
 // moves forward. A row whose engine differs from it is marked, and the site renders that in red.
-const engineOf = (g) => (g && g.rig && g.rig.engine && g.rig.engine.commit) || null;
+// THE INSTRUMENT, NOT THE COMMIT - the same resolution C8 uses, from the same file.
+//
+// C8 groups the board by instrument: commits whose BUILT BINARIES are byte-identical are one
+// instrument, attested in site/instrument-equivalence.json. This function compared raw shas, so
+// gen-data and check-consistency disagreed about what "the same engine" means - C8 would pass a board
+// that gen-data then marked as mixed and (under OTB_SINGLE_ENGINE) blanked. Two commits that provably
+// build the same binary cannot be a real difference in one place and not the other.
+//
+// Falls back to the raw sha for any commit the file does not attest, which is the safe default: an
+// unlisted commit is its own instrument, so silence never merges anything.
+const equivalence = (() => {
+  const f = join(HERE, "instrument-equivalence.json");
+  if (!existsSync(f)) return new Map();
+  return instrumentOf(readFileSync(f, "utf8"));
+})();
+const engineOf = (g) => {
+  const sha = (g && g.rig && g.rig.engine && g.rig.engine.commit) || null;
+  return sha == null ? null : (equivalence.get(sha) || sha);
+};
+// The raw commit, for the per-row stamp a reader sees. The instrument decides COMPARABILITY; the sha
+// is still what identifies the exact tree, and collapsing it here would hide which commit ran.
+const engineShaOf = (g) => (g && g.rig && g.rig.engine && g.rig.engine.commit) || null;
 const newestRow = gateways
   .filter((g) => displayedMeasuredMs(g) > 0)
   .sort((a, b) => displayedMeasuredMs(b) - displayedMeasuredMs(a))[0];
 const boardEngine = engineOf(newestRow);
 for (const g of gateways) {
-  const sha = engineOf(g);
+  // `sha` is what ran; `inst` is what it is comparable TO. A row is current when its INSTRUMENT
+  // matches the board's, so two commits proven to build the same binary both read as current while
+  // the row still reports the exact commit that produced it.
+  const sha = engineShaOf(g);
+  const inst = engineOf(g);
   g.engine = sha
-    ? { sha, short: sha.slice(0, 7), current: boardEngine == null || sha === boardEngine }
+    ? { sha, short: sha.slice(0, 7), current: boardEngine == null || inst === boardEngine }
     // A row with no engine stamp predates the stamp entirely; saying so is better than implying it
     // matches, and better than omitting the field so the render site has to guess.
     : { sha: null, short: null, current: false };

site/instrument-equivalence.json

@@ -40,6 +40,22 @@
         },
         "verified_at": "2026-08-02T19:00:00Z"
       }
+    },
+    {
+      "id": "otb-e97eb008",
+      "commits": [
+        "80030c2fe13e9312f7cddd5db4c5908a1e54d94e",
+        "7fd350ed5ecc7cd60953a33d78afc15fd8ec74d7"
+      ],
+      "reason": "busbar 1.5.0 measured at 80030c2f. The other thirteen were then re-measured PINNED to that same commit (BENCH_COMMIT=80030c2f) precisely so the board would be one instrument - and they came back stamped 7fd350ed, because run-on-ec2.sh derived the stamp from the operator's local HEAD instead of the commit the boxes fetch. They ran the pinned tree; only the label was wrong. 7fd350ed is 80030c2f plus one board commit (drop the 1.4.1 declaration, move a star record, add a snapshot) and touches no engine file. The stamping bug itself is fixed in the same change that adds this entry, so the two can no longer diverge; this entry exists because the artifacts on disk were already stamped by the broken path and re-running fourteen gateways to correct a label would change no measurement.",
+      "evidence": {
+        "method": "cargo build --release --bin otb at each commit in a separate detached git worktree, same host and toolchain, hashed with shasum -a 256. `git diff 80030c2f 7fd350ed -- engine/` is also empty, but the binaries are the admissible evidence and the diff is only what made the entry worth checking.",
+        "otb_release_sha256": {
+          "80030c2fe13e9312f7cddd5db4c5908a1e54d94e": "e97eb00893a1c6c4c4960c27920313b92ad1b00e6ca0e2a0b056760808ea0a60",
+          "7fd350ed5ecc7cd60953a33d78afc15fd8ec74d7": "e97eb00893a1c6c4c4960c27920313b92ad1b00e6ca0e2a0b056760808ea0a60"
+        },
+        "verified_at": "2026-08-03T18:10:00Z"
+      }
     }
   ]
 }