Skip to content

Commit 4dea8c4

Browse files
committed
fix(box-qualify): pool the floor baseline across the field, it measures the rig not the gateway
direct_c1_p99_us is the load generator hitting the mock with NO gateway in the path. It describes the rig and the instance type, so keeping a per-gateway baseline was a category error, and an actively harmful one: with one or two historical samples each, a gateway's baseline was a random draw from the box population, so a gateway that happened to draw a fast box rejected perfectly normal ones while a gateway that drew a slow box accepted almost anything. Observed live: apisix carried a 73us baseline and was rejected at 77us, a value squarely inside that same run's healthy spread of 70-80us, while a genuinely bad box at 89us was correctly caught on a 77us baseline. Same gate, same band, opposite outcomes, decided by which box each gateway happened to draw months ago. The floor lane now pools every gateway's snapshots for the arch (window 15, roughly one full field run's population) and the median describes the population rather than one lucky draw. The peak lane stays per gateway, because throughput genuinely is a property of the gateway under test. Verified against the live snapshots: every gateway now resolves to a 77us pooled floor, apisix's 77us reads +0.00% and passes, and the 89us box still reads +15.58% and fails. A new gateway with no history of its own now inherits the field's floor and is protected from a bad box on its first run, while still seeding its own peak.
1 parent 814f910 commit 4dea8c4

2 files changed

Lines changed: 50 additions & 16 deletions

File tree

‎lib/box_qualify.sh‎

Lines changed: 38 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,12 @@ BQ_PEAK_RATCHET_PCT="${BQ_PEAK_RATCHET_PCT:-40}"
111111
# single previous run. Comparing against only the immediately-previous run means one contaminated run
112112
# silently becomes the next run's reference and the gate normalizes the contamination.
113113
BQ_BASELINE_WINDOW="${BQ_BASELINE_WINDOW:-5}"
114+
# The FLOOR window is separate and wider, because the floor baseline is POOLED across every gateway
115+
# on the arch rather than kept per gateway (direct_c1_p99_us is measured with NO gateway in the path,
116+
# so it describes the rig, not the gateway). Pooling yields roughly one sample per gateway per run,
117+
# so a window of 15 spans about one full field run's box population and the median describes the
118+
# population rather than one lucky or unlucky draw.
119+
BQ_FLOOR_WINDOW="${BQ_FLOOR_WINDOW:-15}"
114120
# How far back the anti-ratchet reference reaches (records, not runs of this window).
115121
BQ_BASELINE_RETAIN="${BQ_BASELINE_RETAIN:-12}"
116122
# Seed the baseline from PRE-FEATURE snapshots (which carry no qualification block) by reading their
@@ -177,11 +183,11 @@ bq_json_str(){ if [ -z "${1:-}" ]; then printf 'null'; else printf '"%s"' "$(pri
177183
# block — see bq_stage1_verdict / bq_stage2_verdict).
178184
# ═════════════════════════════════════════════════════════════════════════════════════════════════
179185
bq_read_baselines(){ # gateway arch snapshots_dir
180-
python3 - "$1" "$2" "$3" "$BQ_BASELINE_WINDOW" "$BQ_BASELINE_RETAIN" "$BQ_ALLOW_LEGACY_BASELINE" <<'PY'
186+
python3 - "$1" "$2" "$3" "$BQ_BASELINE_WINDOW" "$BQ_BASELINE_RETAIN" "$BQ_ALLOW_LEGACY_BASELINE" "$BQ_FLOOR_WINDOW" <<'PY'
181187
import glob, json, os, statistics, sys
182188
183-
gw, arch, snapdir, win, retain, allow_legacy = sys.argv[1:7]
184-
win, retain, allow_legacy = int(win), int(retain), allow_legacy == "1"
189+
gw, arch, snapdir, win, retain, allow_legacy, floor_win = sys.argv[1:8]
190+
win, retain, allow_legacy, floor_win = int(win), int(retain), allow_legacy == "1", int(floor_win)
185191
186192
def med(vs):
187193
return statistics.median(vs) if vs else None
@@ -195,21 +201,24 @@ def num(x):
195201
return x if isinstance(x, (int, float)) and not isinstance(x, bool) else None
196202
197203
recs = []
198-
for path in sorted(glob.glob(os.path.join(snapdir, "result_%s_*.json" % gw))):
204+
# EVERY gateway's snapshots are read, not just this one's. The FLOOR is pooled across the whole
205+
# field (see the lane() calls below); only the PEAK is filtered back down to this gateway.
206+
for path in sorted(glob.glob(os.path.join(snapdir, "result_*.json"))):
199207
try:
200208
with open(path) as fh:
201209
j = json.load(fh)
202210
except Exception:
203211
continue # a corrupt snapshot is not a baseline, and never fatal here
204212
if arch and j.get("arch") and j.get("arch") != arch:
205-
continue # a baseline is per gateway AND per arch
213+
continue # a baseline is always per arch
206214
m = j.get("matrix") or {}
207215
bq = ((j.get("rig") or {}).get("box_qualify")) or {}
208216
qualified = (bq.get("verdict") == "pass")
209217
legacy = not bq # predates the gate entirely
210218
if not qualified and not (legacy and allow_legacy):
211219
continue # a FAILED qualification must never seed the baseline
212-
rec = {"measured_at": j.get("measured_at") or "", "legacy": legacy}
220+
rec = {"measured_at": j.get("measured_at") or "", "legacy": legacy,
221+
"gw": j.get("gateway") or ""}
213222
# ── floor: this run's own stage-1 median when it has one; else the published direct_c1_p99_us,
214223
# which is the SAME measurement (loadgen -> mock, c1, no gateway) taken by the 6x6 itself.
215224
f = num(((bq.get("stage1") or {}).get("floor_p99_us_median")))
@@ -248,18 +257,33 @@ for path in sorted(glob.glob(os.path.join(snapdir, "result_%s_*.json" % gw))):
248257
recs.sort(key=lambda r: r["measured_at"])
249258
out = {}
250259
251-
def lane(key, cell_key=None, conc_key=None, src_key=None, prefix=""):
252-
vals = [(r[key], r) for r in recs if r.get(key)]
253-
keep = vals[-retain:]
254-
window = [v for v, _ in keep[-win:]]
255-
out[prefix + "N"] = str(len(window))
256-
out[prefix + "VAL"] = fmt(med(window)) if window else ""
260+
def lane(key, cell_key=None, conc_key=None, src_key=None, prefix="", only_gw=False, window=None):
261+
src = [r for r in recs if not only_gw or r.get("gw") == gw]
262+
vals = [(r[key], r) for r in src if r.get(key)]
263+
w = window or win
264+
# Retain must never starve the window, or the median would silently be taken over fewer samples
265+
# than asked for (the pooled floor window is wider than the default retain).
266+
keep = vals[-max(retain, w):]
267+
sample = [v for v, _ in keep[-w:]]
268+
out[prefix + "N"] = str(len(sample))
269+
out[prefix + "VAL"] = fmt(med(sample)) if sample else ""
257270
out[prefix + "OLDEST"] = fmt(keep[0][0]) if keep else ""
258271
out[prefix + "SRC"] = keep[-1][1].get(src_key, "") if keep else ""
259272
return keep
260273
261-
lane("floor", src_key="floor_src", prefix="FLOOR_")
262-
peak_keep = lane("peak", src_key="peak_src", prefix="PEAK_")
274+
# FLOOR: POOLED ACROSS EVERY GATEWAY on this arch, deliberately.
275+
# direct_c1_p99_us is the load generator hitting the mock with NO gateway in the path. It measures
276+
# the RIG and the INSTANCE TYPE, not the gateway, so every gateway is sampling the same population
277+
# and a per-gateway baseline is a category error. It is also actively harmful: with one or two
278+
# historical samples each, a gateway's baseline is a random draw from that population, so a gateway
279+
# that happened to draw a fast box rejects perfectly normal ones while a gateway that drew a slow
280+
# one accepts almost anything. Observed live on 2026-07-25: apisix carried a 73us baseline and was
281+
# rejected at 77us, a value squarely inside that same run's healthy spread of 70-80us, while a
282+
# genuinely bad box at 89us was correctly caught. Pooling gives ~one sample per gateway per run, so
283+
# the reference converges immediately and means the same thing for everyone.
284+
# PEAK: per gateway, because throughput genuinely IS a property of the gateway under test.
285+
lane("floor", src_key="floor_src", prefix="FLOOR_", window=floor_win)
286+
peak_keep = lane("peak", src_key="peak_src", prefix="PEAK_", only_gw=True)
263287
# The cell + concurrency to REPLAY come from the MOST RECENT qualified record, not the median: they
264288
# are identifiers, not measurements, and replaying the cell that produced the newest baseline is what
265289
# keeps the comparison like-for-like.

‎lib/box_qualify_test.sh‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -233,11 +233,21 @@ mksnap x 2026-07-25T02-00-00Z arm64 120 pass 1400 208
233233
bq_load_baselines gw arm64 "$TMPD"
234234
check "one wild PASSED run cannot own the floor baseline" "77" "$BQ_BL_FLOOR_US"
235235
check "one wild PASSED run cannot own the peak baseline" "9900" "$BQ_BL_PEAK_RPS"
236-
# A gateway with no snapshots at all: every field empty -> the caller seeds instead of blocking.
236+
# A GATEWAY WITH NO SNAPSHOTS AT ALL. The two lanes deliberately answer differently, and this is the
237+
# "drop a folder in and it works" property:
238+
# FLOOR is POOLED across every gateway on the arch, because direct_c1_p99_us is measured with NO
239+
# gateway in the path: it describes the rig and the instance type, not the gateway. So a
240+
# brand new gateway inherits the field's floor history and is protected from a bad box on
241+
# its very FIRST run, with no per-gateway history of its own.
242+
# PEAK is per gateway, because throughput genuinely is a property of the gateway. A new gateway
243+
# has none, so stage 2 seeds instead of blocking.
237244
bq_load_baselines nosuchgw arm64 "$TMPD"
238-
check "unknown gateway -> no floor baseline" "" "$BQ_BL_FLOOR_US"
245+
check "unknown gateway -> INHERITS the pooled floor" "77" "$BQ_BL_FLOOR_US"
239246
check "unknown gateway -> no peak baseline" "" "$BQ_BL_PEAK_RPS"
240247
check "unknown gateway -> no replay cell" "" "$BQ_BL_PEAK_CELL"
248+
# ...and the pooled floor must still be arch-scoped: a different arch shares nothing.
249+
bq_load_baselines nosuchgw x86_64 "$TMPD"
250+
check "unknown gateway, other arch -> no floor" "" "$BQ_BL_FLOOR_US"
241251

242252
echo "box_qualify_test: PROVENANCE — the run's raw evidence is always recorded, pass or fail"
243253
bq_stage1_verdict 81 79 83 36 20000 76.85 76

0 commit comments

Comments
 (0)