DFlash: separate "may run" from "may publish" so the bench is testable - #815
Merged
Conversation
The enablement interlock required official_scoring_enabled AND
reference_baseline.publication_allowed -- the two go-live fields -- before the
job would run AT ALL. So the only way to exercise this pipeline was to
declare the track live first, and the correctness gates, the fail-closed
guards and the reused serial gate leg could never be validated before the
switch they gate had already been thrown.
Enablement is two questions and they are now asked separately:
may this job RUN? confirm_track_enabled, matching track_id, non-empty
golden pins. Unchanged, and still mandatory for
BOTH kinds of dispatch -- a dry run is still a run
on the operator's box.
may it PUBLISH a
ranked score? official_scoring_enabled AND publication_allowed.
Required for run_benchmark=true. A gates-only
dispatch is admitted with a ::notice:: and a
MLXFAST_DFLASH_DRY_RUN_UNENABLED marker.
A gates-only dispatch produces no ranked score: it writes a sealed local
gates-score.json plus the gates-only correctness artifacts, and every timed
and scoring step is gated on inputs.run_benchmark. Ordering makes it useful
-- the four reused serial gates are steps 31-39, ahead of the two fail-closed
operator blockers (empty prompt pool at 41, golden row count at 42) -- so a
dry run exercises the gates and then stops cleanly.
The PUBLISH check is now made TWICE: at the enablement step and again at
"Compute DFlash score and enforce floor", which also refuses a dispatch
carrying the dry-run marker. A guard 49 steps upstream of the thing it
protects is a guard whose coverage depends on step order.
DFlashEnablementInterlockTests, 7 tests. Six read the step text; the
seventh EXECUTES the real step body over the full truth table, because the
Amendment 23 redactor gap would have passed a text-matching test -- the arms
were absent, not misspelled. Attacked before trusting: inverting the ranked
refusal and weakening the confirm interlock to ranked-only are both caught,
each naming the exact case that was wrongly admitted.
The fixture stays inert on main (official_scoring_enabled false,
publication_allowed false, timed_prompt_pool empty) and a test pins that, so
"completing" this by flipping the flags fails the suite instead.
swift test: 526 tests, 21 suites, green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
anupsv
had a problem deploying
to
benchmark-private-prompts
July 30, 2026 22:44 — with 
GitHub Actions
Error
GitHub Actions
Error
anupsv
added a commit
that referenced
this pull request
Jul 31, 2026
…819) Found by an independent review of #814-#817, not by any failing run. This is a defect I introduced, and it is the worst shape available here: an ITERATED ORACLE for the hidden golden. The mechanism. "Stage DFlash correctness artifacts" became always() in #816 so that a FAILED gates-only run still yields diagnostics. That is right in itself, but it made the step run in the one state its own comment assumed away: when "Validate sealed gates score" did NOT run. On a failed hidden-gates run the gates pass writes gates-score.json and THEN exits nonzero. The validation step is skipped along with everything after it, but the always() staging step still runs, stages the file, and uploads it. That sealed score's .metrics carry first_failing_step, expected_token and actual_token -- values derived FROM THE HIDDEN GOLDEN. Three things turned a disclosure into an oracle: - deny-private-artifacts.sh is NAME- and SIZE-based. It cannot see values inside a permitted filename; a gates-score.json containing expected_token: 31337 passes it with exit 0. - The redaction machinery never runs in gates-only mode ("Redact DFlash benchmark failure" is failure() && inputs.run_benchmark), so a failed dry run gets no redacted category AND the raw file -- the exact inversion of the design. - A gates-only failure charges no attributable-failure budget, so the run is repeatable for free. One hidden token per run. redact-benchmark-failure.sh states the invariant this broke in its own header: "a failing score.json must never be uploaded raw." Serial does not have this exposure. Its hidden-gate steps are gated on inputs.run_benchmark, so its dry run never holds hidden material at all, and its correctness staging additionally requires the validation step to have SUCCEEDED on submission branches. The DFlash copy dropped that guard and removed the run_benchmark gating -- so #814 created the arm and #815 made it routinely reachable. The fix re-asserts the sealed-score predicate INLINE in the staging step, using the same jq "Validate sealed gates score" uses, and DROPS a score that fails it rather than failing the step: the run has already failed for its own reason and the other artifacts are still worth having. public-gate-report.json gets the guard serial has and the DFlash copy dropped. Its .error can embed sandboxed-worker stderr -- text the SUBMITTED code controls -- so on a submission branch it is withheld unless .error is empty. On an organizer ref it is organizer output and is kept even when it records a failure. DFlashGatesOnlyArtifactLeakTests, 4 tests, all driving the REAL step body: the failed-score case is withheld and no hidden value reaches the staged set; a clean score is still staged (so the guard is not a mute); the public gate report is withheld on a submission branch and kept off one; and a structural test asserts the inline predicate checks every field the validation step checks, so a future tightening of one cannot silently bypass the other. Proven load-bearing: disabling the content gate makes the leak test fail, naming the file and the reason. NOT fixed here, filed for follow-up: a failed gates-only run still produces no failure_category, because the redactor is gated on inputs.run_benchmark. Adding a failure() && !inputs.run_benchmark arm needs its own change; withholding the raw file closes the leak, surfacing a category is the improvement on top. swift test: 534 tests, 24 suites, green. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
anupsv
added a commit
that referenced
this pull request
Jul 31, 2026
#815 split the enablement interlock so the DFlash pipeline could be validated WITHOUT flipping the go-live flags: run_benchmark=false publishes no score, so it is admitted against an inert track. The point of that split is to exercise "DFlash correctness and parity gate (untimed)" -- the one gate this track owns. It could not. Two steps above that gate were not gated on inputs.run_benchmark: * "Select hidden DFlash timed target from the pool", which fails closed on an empty timed_prompt_pool; and * the timed-golden half of "Prepare hidden DFlash goldens", which is addressed BY that selection. The pool is deliberately empty on main (pinned by theTrackFixtureRemainsInertOnMain), so before go-live EVERY dispatch died red at pool selection -- one step short of the gate the dry run exists to run. The dry run validated nothing the serial pipeline had not, and operators were being taught to read red as success. A dry run also downloaded a hidden timed golden it would never time against. Scope both to the ranked path. The DFlash CORRECTNESS golden is pinned independently of the pool, so the untimed leg -- including the parity gate -- now runs and a clean dry run ends GREEN. The ranked path is unchanged and is not relaxed: * a ranked dispatch still runs the selection step and still fails closed on an empty, absent, null or malformed pool. Losing per-run sampling would lose contract layer L6 (anti-lottery), under which a failed ranked run becomes a free retry and every output-side gate degrades into submit-until-green; * the timed golden's pin check against the SAMPLED entry is unchanged; * the row-count preflight is scoped, not relaxed -- it stays unconditionally binding whenever the run times; * a new, independent refusal fires if a ranked run ever reaches the download with an empty selection. Honest scope: this is defence in depth and a named diagnostic, not the only barrier -- download-r2-object.sh already refuses an empty key -- but it names the cause instead of dying on a generic exit 2 in a helper. Tests: Tests/MLXFastTests/DFlashDryRunReachesItsGateTests.swift, 8 tests. Six of them EXECUTE the real step bodies (pool selection, golden preparation) against sandboxes; the step-set tests evaluate the real `if:` expressions with an evaluator that hard-fails on any construct it does not recognise rather than guessing. Verified by attack: reverting the workflow to origin/main fires 3 tests (8 issues), and five separately-injected defects -- a ranked run allowed past an empty pool, the empty-selection refusal removed, the whole goldens step gated ranked-only, the parity gate itself gated ranked-only, and the timed pin check dropped -- each fire the test that names them. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The enablement interlock required
official_scoring_enabledandreference_baseline.publication_allowed— the two go-live fields — before the jobwould run at all.
So the only way to exercise this pipeline was to declare the track live first. The
correctness gates, every fail-closed guard, and the newly reused serial gate leg
could never be validated before the switch they gate had already been thrown.
That's backwards, and it's how the fifth copy of the decode floor survived review —
it was only ever found by dispatching for real.
The split
Enablement is two questions, now asked separately:
confirm_track_enabled, matchingtrack_id, non-empty golden pinsofficial_scoring_enabledandpublication_allowedrun_benchmark=trueA gates-only dispatch is admitted with a
::notice::and aMLXFAST_DFLASH_DRY_RUN_UNENABLEDmarker.It publishes nothing. It writes a sealed local
gates-score.jsonplus thegates-only correctness artifacts; every timed and scoring step is gated on
inputs.run_benchmark. I verified this by enumerating the 51 of 63 steps that runwhen
run_benchmark=false— none reach a ranked score.Ordering makes it useful: the four reused serial gates are steps 31–39, ahead of
the two fail-closed operator blockers (empty prompt pool at 41, golden row-count at
42). A dry run exercises the gates, then stops cleanly.
Checked twice, on purpose
The PUBLISH check now also runs at
Compute DFlash score and enforce floor, whichadditionally refuses any dispatch carrying the dry-run marker. A guard 49 steps
upstream of the thing it protects is a guard whose coverage depends on step order.
Tests:
DFlashEnablementInterlockTests, 7 testsSix read the step text. The seventh executes the real step body over the full
truth table — because the Amendment 23 redactor gap would have passed a
text-matching test (the arms were absent, not misspelled).
Attacked before trusting it. Inverting the ranked refusal, and weakening the
confirm interlock to ranked-only, are both caught — each naming the exact case that
was wrongly admitted.
One test pins that the fixture stays inert on main (
official_scoring_enabledfalse,
publication_allowedfalse,timed_prompt_poolempty), so "completing" thisby quietly flipping the flags fails the suite instead.
Verification
swift test: 526 tests, 21 suites, greenswift build -c release: cleanNext
With this merged, a gates-only dispatch can validate the four reused serial gates
against real hidden material for the first time — they are currently wired and
unit-tested but have never executed.
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmithwith what you need. Autofix is disabled.