Skip to content

Commit 9cbcc08

Browse files
committed
audit: whole-tree codeaudit + sharded-path fixes (engine/mock/site/python/bash/CI) — no ENGINE_PIN bump
1 parent b2d71cd commit 9cbcc08

52 files changed

Lines changed: 3416 additions & 532 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/bench-rig.yml‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,12 @@ jobs:
2727
steps:
2828
- uses: actions/checkout@v4
2929

30-
- name: Install Rust
31-
run: |
32-
curl -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal
33-
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
30+
# Install Rust via the pinned action rather than piping sh.rustup.rs into a shell. This job holds
31+
# contents:write + GH_TOKEN and publishes the otb/mock binaries every bench box downloads as the
32+
# frozen instrument; the build-commit self-check below runs AFTER the toolchain install and cannot
33+
# catch a compromised toolchain that still compiles the correct source. dtolnay/rust-toolchain is
34+
# the same install path bench-tests.yml already trusts.
35+
- uses: dtolnay/rust-toolchain@stable
3436

3537
- name: Build mock (Rust, native ${{ matrix.arch }})
3638
run: |

‎.github/workflows/bench-tests.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,10 @@ on:
1616
- "gateways/**"
1717
- "lib/**"
1818
- "run-on-ec2.sh"
19+
# history/append.py is the append-only, git-tracked, tamper-evident record; without this path an
20+
# edit that regressed its nesting/dedup (audit P1-11, and the top-level-vs-matrix nesting bug) fired
21+
# NO workflow and ran NO test - the same paths-as-decoration defect the rig-delivery job documents.
22+
- "history/**"
1923
# The audit gate's own subjects. Without these, an edit that broke bench-audit.py (the exact
2024
# wrong-key-path class its header describes) or a results push that violated every invariant
2125
# never triggered the job that exists to catch it - the same paths-as-decoration defect the
@@ -62,6 +66,10 @@ on:
6266
- "gateways/**"
6367
- "lib/**"
6468
- "run-on-ec2.sh"
69+
# history/append.py is the append-only, git-tracked, tamper-evident record; without this path an
70+
# edit that regressed its nesting/dedup (audit P1-11, and the top-level-vs-matrix nesting bug) fired
71+
# NO workflow and ran NO test - the same paths-as-decoration defect the rig-delivery job documents.
72+
- "history/**"
6573
# The audit gate's own subjects. Without these, an edit that broke bench-audit.py (the exact
6674
# wrong-key-path class its header describes) or a results push that violated every invariant
6775
# never triggered the job that exists to catch it - the same paths-as-decoration defect the
@@ -188,6 +196,26 @@ jobs:
188196
- uses: actions/checkout@v4
189197
- name: The fetch lands files with the mode the commit recorded
190198
run: bash lib/fetch_modes_test.sh
199+
# These lib/ guards existed on disk but ran NOWHERE - lib/** was a path trigger while only
200+
# fetch_modes_test.sh was invoked, so a regression in the promote guard (RULES.md's "never
201+
# overwrite a better snapshot with a worse one"), the concurrent-publish lock, or the
202+
# box-reachability poll shipped green. Run them all here.
203+
- name: The promote guard never lets a boot/build failure overwrite a served result
204+
run: python3 lib/promote_guard_test.py
205+
- name: Concurrent-publish lock (unique per-publish token, owner-only release, flock path)
206+
run: bash lib/publish_lock_test.sh
207+
- name: Box-reachability poll retries a transient ssh failure before declaring a box gone
208+
run: bash lib/box_reachable_test.sh
209+
# The provenance chain RULES.md calls the project's core: which instrument each run used.
210+
- name: Rig provenance (fetch fallback order, origin labels, results-scoped engine dirty flag)
211+
run: bash lib/rig_test.sh
212+
# The shared-key lifetime filter behind the 2026-08-21 key-stranding incident.
213+
- name: Shared-key teardown filter excludes only this run's own boxes
214+
run: bash lib/teardown_filter_test.sh
215+
# The append-only history collector: nesting (top-level vs data['matrix']), the dedup-by-
216+
# (suite, measured_at) contract, and the dropped-row / garbage-line error paths.
217+
- name: Append-only history preserves the nested matrix measurements (and dedups)
218+
run: python3 history/append_test.py
191219

192220
mock-shape:
193221
# The mock must never bottleneck and must serve every dialect identically; its per-dialect

‎.github/workflows/pages.yml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,13 +25,32 @@ concurrency:
2525
jobs:
2626
build:
2727
runs-on: ubuntu-latest
28+
# PARITY WITH cf-pages.yml, because this workflow is STILL LIVE (pages: write / id-token: write) and
29+
# fires on the same push paths. Without OTB_SINGLE_ENGINE=1 gen-data.mjs would NOT suppress rows a
30+
# newer engine has not re-measured (its SINGLE_ENGINE gate only activates when this is literally "1"),
31+
# and without the consistency gates below this job would upload a mixed-instrument or internally
32+
# inconsistent board straight to the still-live GitHub Pages URL - the exact state cf-pages.yml's
33+
# policy exists to prevent. This workflow can be deleted outright once the Cloudflare nameserver
34+
# cutover is complete (see cf-pages.yml's header); until then it must not ship a board cf-pages would
35+
# refuse.
36+
env:
37+
OTB_SINGLE_ENGINE: "1"
2838
steps:
2939
- uses: actions/checkout@v4
3040
- uses: actions/setup-node@v4
3141
with:
3242
node-version: 22
3343
- name: Generate site data
3444
run: node site/gen-data.mjs
45+
# Gate the bundle BEFORE upload, exactly as cf-pages.yml does: structural invariants on the file
46+
# being deployed, then the per-surface coverage siblings, then the table==drawer==compare==charts
47+
# consistency guard. A red here must block the artifact upload, not ship past it.
48+
- name: Structural invariants C1-C5 on the bundle being deployed
49+
run: node site/check-consistency.mjs site/data.json
50+
- name: Site coverage siblings (URL codec fixed points, one envelope one story)
51+
run: node site/test-coverage.mjs
52+
- name: Consistency guard (table == drawer == compare == charts)
53+
run: node site/test.mjs
3554
- uses: actions/upload-pages-artifact@v3
3655
with:
3756
path: site

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,10 @@ __pycache__/
3636
# orchestrator ships back onto the box. The DURABLE record is rig.box_qualify inside each snapshot
3737
# (results/snapshots/), which is committed — these are the working files that produced it.
3838
results/box-qualify/
39+
# write_snapshot's per-gateway advisory lockfile (flock rendezvous point held across the promote
40+
# guard's read->decide->rename critical section). Working-directory plumbing, never a result, so it
41+
# must not be committed alongside the snapshot JSON.
42+
results/**/.snapshot-lock-*
3943
target/
4044

4145
# Rendered gateway config. The TEMPLATE (*.tmpl) is the source and is tracked; what the harness

‎audit-every-metric.py‎

Lines changed: 1 addition & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -70,19 +70,6 @@
7070
# equals the difference of its own published legs - the statement that is actually true of them.
7171

7272

73-
# The gateway's pinned core count, and how far the two samplers may disagree before it is a defect.
74-
#
75-
# NOT TUNING KNOBS. `run-on-ec2.sh` pins the gateway to CORES=0-3, so four is the divisor the
76-
# utilisation fraction is taken over - if that split changes, this must change with it or the check
77-
# silently compares against the wrong denominator. The tolerance is deliberately LOOSE (3x) because
78-
# the honest direction of disagreement is one-sided: the utilisation window spans a little more wall
79-
# time than the load window, so implied-from-CPU runs BELOW measured. Only the impossible direction
80-
# is flagged - a gateway appearing to burn more CPU than its cores accumulated, which means it is not
81-
# confined to them and the comparable-basis claim is false.
82-
COST_PINNED_CORES = 4
83-
COST_UTIL_TOLERANCE = 3.0
84-
85-
8673
def num(v):
8774
if isinstance(v, dict):
8875
v = v.get("value")
@@ -195,7 +182,7 @@ def main():
195182
# 0.3 MiB (0.6%) on agentgateway's 48 MiB tree and 11.5 MiB (1.33%) on bifrost's 876 MiB
196183
# one. Board-wide: 32 of 92 cells overshoot, median 0.22%, and the only cell above 1% is
197184
# the largest tree on the board. That is the artefact's signature, not a defect's.
198-
if hwm is not None and peak is not None and peak > hwm:
185+
if hwm is not None and hwm != 0 and peak is not None and peak > hwm:
199186
warnings.append(
200187
f"{at}: sampled peak ({peak:.1f} MiB) exceeds kernel HWM ({hwm:.1f} MiB) by "
201188
f"{(peak - hwm):.1f} MiB ({(peak / hwm - 1) * 100:.2f}%) - transient-worker artefact"

‎bench-audit.py‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -380,9 +380,8 @@ def check_sweep_carries_its_latency(name, c):
380380
"plateaued", "load_s",
381381
# Absent BECAUSE the measurement succeeded: these describe HOW a window failed to settle, so a
382382
# window that DID settle has no shape to publish and says exactly that. They are listed here
383-
# for the reason every other field is - a null must carry a reason - and NOT held to
384-
# check_no_bare_absence's "a served cell publishes a number", which is why that check reads
385-
# SHAPE_FIELDS below and lets a reasoned absence stand.
383+
# for the reason every other field is - a null must carry a reason in the absences map, which
384+
# check_no_bare_absence enforces uniformly for them just like every other field.
386385
"shape", "idle_shape",
387386
# The idle window's own verdict and its fitted slope. The memory group measured both on every
388387
# served cell and CellMemory had nowhere to put them, so they reached the artifact as neither
@@ -393,9 +392,6 @@ def check_sweep_carries_its_latency(name, c):
393392
],
394393
}
395394

396-
# The fields whose absence is a RESULT rather than a gap. See the note in ABSENCE_CARRYING_FIELDS.
397-
SHAPE_FIELDS = {"memory": {"shape", "idle_shape"}}
398-
399395
# The absence reasons that legitimately excuse a CAPACITY metric from being a number. Everything
400396
# else must be a number - a gateway that failed the gate at every rung is a measured 0, not a hole.
401397
CAPACITY_ABSENCE_OK = {"untestable", "rig_limited", "search_exhausted", "harness_error", "not_served"}

‎bench-audit_test.py‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -306,6 +306,27 @@ def main():
306306
if list(audit.check_stream_capacity_is_a_number("t", explained)):
307307
failures.append("check_stream_capacity_is_a_number rejected an absence that explains itself")
308308

309+
# ── check_search_spent_its_budget: the budget-EXHAUSTED accept side ────────────────────────────
310+
#
311+
# The REJECTS entry above proves the GAVE-UP branch (a gap the search declined to probe). Its own
312+
# accept short-circuit `if walked_down >= MAX_CEILING_STEPDOWNS: return` had no positive fixture: a
313+
# search that spent its FULL step-down budget ran OUT, it did not give up, and must be accepted even
314+
# though it stopped at a rung above the one it carried. This is the plano anthropic>anthropic shape
315+
# from the 2026-08-01 run - the bisection proved c=224, confirmation failed, and the step-down
316+
# walked 176 -> 152 -> 140 -> 134, exactly MAX_CEILING_STEPDOWNS rungs. Delete or invert that
317+
# short-circuit and this block fires; the REJECTS-side gap fixture would still pass, so only this
318+
# catches it.
319+
with isolated(failures, "check_search_spent_its_budget: budget-exhausted accept side"):
320+
spent = [rung(32), rung(64), rung(224, passed=False), rung(176, passed=False),
321+
rung(152, passed=False), rung(140, passed=False), rung(134, passed=False)]
322+
assert len({r["conc"] for r in spent if r["conc"] > 64 and r["conc"] < 224}) == audit.MAX_CEILING_STEPDOWNS
323+
budget_spent = trace_cell(spent, sustained=None,
324+
absence_detail="the bisection proved c=224 but no stepped-down rung "
325+
"held within MAX_CEILING_STEPDOWNS attempts")
326+
if list(audit.check_search_spent_its_budget("t", budget_spent)):
327+
failures.append("check_search_spent_its_budget flagged a search that spent its FULL "
328+
"step-down budget - a run-out is not a give-up")
329+
309330
# ── the omitted-field check, both ways (ledger TOOL-04) ───────────────────────────────────────
310331
#
311332
# ACCEPT: a cell that carries every declared field passes, whether the field holds a number or an

‎bench-dashboard.py‎

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -140,14 +140,12 @@ def cell_fraction_done(phase_name):
140140
IP_RE = re.compile(r"ip=([0-9.]+)")
141141
TS_RE = re.compile(r"^\[(\d\d):(\d\d):(\d\d)\]")
142142

143-
# A cell whose verdict is one of these was skipped or refused, and cost almost nothing. Anything else
144-
# was measured, and is what the remaining time is actually made of.
145-
CHEAP = ("not_configurable", "untestable", "unprobed_auth")
146-
# A CELL IS SERVED ONLY IF THE ENGINE SAID SO. This used to count anything whose verdict was not one
147-
# of CHEAP, which is fail-open: a new line shaped like `[cell N/M] <id>: <anything>` silently became a
148-
# served cell. That happened the day a per-cell cost breakdown was added under the same prefix - every
149-
# measured cell counted twice, gateways showed "10/8 served", and the ETA hit zero while they were
150-
# still running. Listing what DOES count means an unrecognised line is ignored rather than believed.
143+
# A CELL IS SERVED ONLY IF THE ENGINE SAID SO. This used to count anything whose verdict was not a
144+
# cheap skip/refusal ("not_configurable", "untestable", "unprobed_auth"), which is fail-open: a new
145+
# line shaped like `[cell N/M] <id>: <anything>` silently became a served cell. That happened the day a
146+
# per-cell cost breakdown was added under the same prefix - every measured cell counted twice, gateways
147+
# showed "10/8 served", and the ETA hit zero while they were still running. Listing what DOES count
148+
# means an unrecognised line is ignored rather than believed.
151149
MEASURED = ("served", "failed")
152150

153151

‎cell-verdict.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,8 @@
1313
cell-verdict.py --baseline <snap.json> --new <partial.json> --state <state.json> [--count-only]
1414
1515
A cell counts as "measured" once it has a frontier RPS at the 10ms p99 bound; latency (added p50/p99,
16-
c=1 p99, cpu us/req) rides along as corroboration. Absences print their own reason rather than a zero.
16+
c=1 p99, cpu us/req) rides along as corroboration. An absent corroboration figure prints an em-dash
17+
(—) rather than a zero, so a missing measurement is never shown as a real value.
1718
"""
1819
import argparse
1920
import json
@@ -52,6 +53,8 @@ def frontier_rps(cell, bound_us=BOUND_US):
5253

5354

5455
def perf_num(cell, field):
56+
if not cell:
57+
return None
5558
return ((cell.get("perf") or {}) or {}).get(field)
5659

5760

‎compare-busbar-versions.py‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
#!/usr/bin/env python3
2-
"""Side-by-side: busbar 1.4.1 (published board) vs busbar 1.5.0 (private evaluation).
2+
"""Side-by-side: busbar 1.4.1 (published board) vs the private-eval busbar build.
33
4-
Reads the 1.4.1 numbers from its committed snapshot and the 1.5.0 numbers from whatever the
4+
The private-eval build's display name is read from its own gateways/busbar-151/definition.json (so the
5+
label never drifts from the version actually measured), NOT hardcoded here.
6+
7+
Reads the 1.4.1 numbers from its committed snapshot and the private-eval numbers from whatever the
58
evaluation box has written so far - the live `busbar-151.json`, pulled partial, or final snapshot -
69
so a cell can be compared the moment it lands rather than at the end of an 11-hour grid.
710
@@ -88,6 +91,16 @@ def delta(a, b):
8891
]
8992

9093
HERE = os.path.dirname(os.path.abspath(__file__))
94+
95+
96+
def gateway_display(gw, fallback):
97+
"""The gateway's own declared display name, so a version label never drifts from the data it
98+
prints. The private-eval snapshots come from gateways/<gw>/definition.json's build."""
99+
d = load(os.path.join(HERE, "gateways", gw, "definition.json")) or {}
100+
return d.get("display") or fallback
101+
102+
103+
NEW_DISPLAY = gateway_display("busbar-151", "Busbar 1.5.1")
91104
pairs = sys.argv[1:] or ["openai>openai"]
92105

93106
old_doc = load(newest(os.path.join(HERE, "results/snapshots/result_busbar_2026-08-02*.json"))
@@ -98,17 +111,17 @@ def delta(a, b):
98111
new_doc = load(new_path)
99112

100113
print(f"busbar 1.4.1 (published board) <- {os.path.basename(newest(os.path.join(HERE,'results/snapshots/result_busbar_2026-08-02*.json')) or '?')}")
101-
print(f"busbar 1.5.0 (private eval) <- {os.path.basename(new_path) if new_path else 'not written yet'}")
114+
print(f"{NEW_DISPLAY} (private eval) <- {os.path.basename(new_path) if new_path else 'not written yet'}")
102115
print()
103116

104117
for pair in pairs:
105118
ing, _, eg = pair.partition(">")
106119
a, b = cell_of(old_doc, ing, eg), cell_of(new_doc, ing, eg)
107120
print(f"=== {ing} > {eg} ===")
108121
if not b:
109-
print(" 1.5.0 has not written this cell yet\n")
122+
print(f" {NEW_DISPLAY} has not written this cell yet\n")
110123
continue
111-
print(f" {'metric':<20} {'1.4.1':<28} {'1.5.0':<28}")
124+
print(f" {'metric':<20} {'1.4.1':<28} {NEW_DISPLAY:<28}")
112125
for label, block, field, unit, higher_better in ROWS:
113126
av, ar = num(a, block, field)
114127
bv, br = num(b, block, field)

0 commit comments

Comments
 (0)