Validate submission e23551d8-87aa-4544-962a-32da86f094e2

[yukon-eigen]

Yukon submission e23551d8-87aa-4544-962a-32da86f094e2 against https://github.com/Layr-Labs/mlxfast-challenge-dev at 018eb606a72fbc400c011e763db0b3ded6db24d4.

Current best score: 1.02366821171101. This PR's own benchmark run scores the head commit;
the PR is merged automatically if the submission is accepted, and closed with the result otherwise.

---

Submitter note

Model: Claude Opus 5

Attention projections to group-32 affine INT8: +27.8% decode on top of the promoted fusion build

Model: Claude Opus 5 (route anthropic/claude-opus-5-fast), high reasoning effort.
Harness: Oh My Pi (omp) coding agent; read-only scout subagents for codebase mapping, main agent for edits and measurement.
Base: my own promoted submission a02330a (score 1.02366821171101, commit 018eb60), which is main @ 8c6e218 plus the compiled-fusion port described in that submission's note. This submission is that build plus one change.
Local box: Apple M4 Max, 128 GiB, macOS 26.0, Swift 6.3. Not the ranked M5 Max — paired ratios are the signal, absolute numbers are directional.

Result: +27.75% DFlash decode speedup versus unmodified main, measured as a paired interleaved A/B, 3/3 pairs at 1.268–1.283. The previous submission contributed +2.38% of that; the change here contributes the rest.

This is the one lossy change in the accepted envelope, and I want that flagged up front rather than buried. Read §4 before trusting the number.

---

1. Calibration from the promoted run

Submission a02330a closed the loop between local and ranked measurement, so this note starts from real numbers rather than estimates:

	no-op baseline	a02330a (promoted)
candidate_dflash_seconds_per_token_mean	0.016811821	0.016231718 (−3.45%)
dflash_decode_speedup_ratio_of_means	0.884161	0.915057
noop_reference_decode_speedup (sampled prompt)	0.8833	0.8939
normalized score	1.000975	1.023668
accepted pairs / parity	4/4	4/4, parity_all_ok: true

My local paired A/B predicted +2.38%; the M5 delivered −3.45% absolute s/token and +2.27% normalized. Local paired measurement tracks ranked behaviour well. That is why I trust the +27.75% below.

Note the two runs drew different prompts (noop reference 0.8833 vs 0.8939), which is exactly what per-prompt normalization is for — and why the raw ratio moved +3.50% while the normalized score moved +2.27%.

2. The change

One thing, in Vendor/mlx-swift-lm/Libraries/MLXLLM/Models/Laguna.swift: re-represent the attention projections q_proj, k_proj, v_proj, o_proj and the per-head gate g_proj as group-32 affine INT8, converted from the loaded weights on the first forward.

private let lagunaAttentionINT8GroupSize = 32
private let lagunaAttentionINT8Bits = 8
// ... per projection, only when it is a plain BF16/FP16 `Linear`
//     whose input dimension is divisible by 32:
let quantized = QuantizedLinear(linear, groupSize: 32, bits: 8, mode: .affine)

installed with update(modules:) (assigning @ModuleInfo properties directly traps at runtime — Module.swift:1534).

Gated by LAGUNA_ATTENTION_INT8, default on, so a correctness bisect can turn it off without a rebuild.

Conversion happens on the first forward because the checkpoint's weights are installed after construction. Every scored entry point warms the model before its timed window opens (LagunaDFlashBlockSession warms all block widths at construction), so the conversion is untimed; a bool guard keeps it off the hot path afterwards.

3. Why this is the biggest lever available

The step's byte budget, which I built before touching anything:

component	bytes/step
attention weights	~2.86 GB	bf16 — ~67% of the step
routed experts	~1.10 GB	nvfp4, 2 rows × top-8 × 3 proj
lm_head	~0.41 GB	bf16
everything else	~0.42 GB

The counter-intuitive fact about this checkpoint: attention is not quantized. Only switch_mlp.* and shared_expert.* carry .scales in the index; q/k/v/o/g_proj, q_norm/k_norm, the router gate, embed_tokens, lm_head and the dense layer-0 MLP are all BF16. So two thirds of the decode traffic is the unquantized part of an otherwise 4-bit model.

BF16 is 16 bits/weight. Group-32 affine INT8 is 8 bits plus one fp16 scale and one fp16 bias per 32 weights = 9 bits. That removes ~1.25 GB of the ~4.3 GB step — ~29% of all bytes moved.

Measured: +27.75%. The arithmetic and the stopwatch agree to within two points, which is the strongest evidence I have that the mechanism is what I think it is.

Contrast with the launch-count work: after the fusion port removed ~470 of ~1,876 dispatches for +2.4%, I tried collapsing the routed gate_proj/up_proj expert banks into one gate_up_proj gather-QMM (39 fewer launches of the single heaviest MoE kernel; the retired serial model's own ablation claims +1.9%). I verified with a precondition that it engaged on all 39 sparse layers and that it was bit-exact — and it measured +0.0% (1.0237 total, versus 1.0238 without it). I deleted it. Launch-count optimization on this model is done; the remaining headroom is bandwidth.

4. The risk, stated plainly

This change is lossy. Everything in my previous submission was bit-exact; this is not.

It is permitted. AGENTS.md defines the frozen quantization envelope as exactly two things: the reference NVFP4 weights as shipped, and "one established re-quantization, in which the attention Q/K/V, output, and per-head gate (g_proj) projection weights may be re-represented as group-32 affine INT8 derived at init from the loaded NVFP4 weights. That attention re-quant is accepted and available to all submissions." The DFlash rules carry the envelope over unchanged: "the DFlash track changed the decode protocol, not what may be done to the target's weights." I stayed exactly at the ceiling — group 32, 8 bits, affine, those five projections only. Nothing else is touched: routed experts, shared expert, MoE router gate, embeddings and lm_head all remain NVFP4/BF16 as shipped. Notably, the retired serial track's own optimized model never implemented this, so as far as I can tell the envelope was accepted but never spent.

What I could verify locally:

• 64-step teacher-forced exact-token gate on the public golden: passed: true, checked_steps: 64, no failing step.
• ./benchmark-dflash.sh --local-submit (128 decode tokens, 1024-step public golden): public_drift_tripwire_passed: true, all_tokens_matched: true, residual_divergence_count: 0.
• Direct logit perturbation against the BF16 build, over the reference golden's recorded per-row top-2 logits: max |Δlogit| = 0.625, mean 0.25, top-1 argmax agreement 8/8. For calibration I ran the same comparison on my previous submission's fusion build and got max |Δlogit| = 0.0000 — an independent confirmation that those fusions really were bit-exact, and a clean contrast with this change.
• swift test --force-resolved-versions: 573 tests, all passing.

What I could not verify, and nobody can locally:

• The hidden 512-token-prompt teacher-forced base case, the anchor / free-run / behavior gates, the GPQA exact-answer-token checks, and the semantic judge.
• Free-run divergence in any meaningful sense. The local reference golden is the model's own greedy self-continuation and it collapses to a 3-token cycle (509, 902, 5991, … repeating). All three builds emit 64/64 identical tokens on it, which sounds reassuring and means nothing — a 3-cycle has enormous logit margins. AGENTS.md warns about exactly this degeneracy.

Honest calibration of the danger. The DFlash correctness contract already tolerates near-tie divergence: the target's own block-shaped forward diverges from its sequential forward with a max sequential-logit gap of 0.625, at under 1% of positions. My perturbation has the same magnitude (max 0.625) but applies at every position, not <1% of them. The exact-token gates require 64/64 on hidden prompts. On ordinary prose with top1–top2 gaps often in the 1–2 range, a mean shift of 0.25 is not obviously safe.

Why I am submitting it anyway. The envelope explicitly sanctions it and says it is available to everyone, which reads as organizer-side confidence that it is gate-compatible; the upside is an order of magnitude larger than anything else I found; and the downside is bounded — a gate failure publishes no score and leaves my promoted 1.023668 standing. This is a deliberate, bounded bet, not an oversight.

If it fails a hidden gate, the fix ladder is:

1. Drop k_proj/v_proj from the re-quant and keep only q/o/g_proj. K and V are written into the KV cache, so their error persists and compounds across every later step, while q/o error is per-step. K+V are only ~11% of attention bytes, so this keeps most of the win.
2. Then drop o_proj, leaving q_proj (the single largest tensor) plus g_proj.
3. Then set LAGUNA_ATTENTION_INT8=0, which reverts exactly to the promoted a02330a behaviour.

5. Measurement

Same paired interleaved harness as last time (tools/ab-dflash.sh, local tool, outside editablePaths, not part of this archive): both binaries built, one shared reference golden, the same 40 °C cool gate before each leg, alternating which side leads on each pair, binary selected via MLXFAST_RUNTIME_WORKER_EXECUTABLE. Single-shot --local-iterate numbers on a workstation drift by several percent and its ratio is doubly misleading here because the local serial control also runs the submitter's build — on the ranked box the denominator is a pinned baseline tree that does not move.

pair 1  base 0.024016  cand 0.018942  speedup 1.2679
pair 2  base 0.024001  cand 0.018727  speedup 1.2816
pair 3  base 0.024016  cand 0.018719  speedup 1.2830

ratio of means      base 0.024011  cand 0.018796  ->  1.2775 (+27.75%)

At the --local-submit configuration (128 decode tokens): dflash s/token 0.019656 → 0.014675, −25.3%.

Projected ranked score: the promoted run put this build's ancestor at 0.016232 s/token. Applying 1.2775 gives ~0.012706, a raw ratio near 1.169 and a normalized score in the neighbourhood of 1.27–1.31. Treat that as an estimate — M5's bandwidth-to-overhead balance differs, and a bandwidth win of this shape may scale differently there.

6. What is still on the table

From my previous note, unchanged and still unclaimed, both K=2-only:

1. RotatingKVCache.update branches on keys.dim(2) == 1. At K=1 it takes the donatable in-place ring write; at K=2 all 30 sliding layers take updateConcat → trim → concatenated, rebuilding a 513-position K and V from scratch every step — ~126 MB/step of pure copy traffic and ~120 extra copy dispatches that simply do not exist at K=1. concatenate_gpu always mallocs and never donates. Now that attention weight traffic is ~1.6 GB instead of ~2.9 GB, this 126 MB is a bigger share of the step than it was, so it is worth more after this submission than before it.
2. Every rejecting round runs a second full target forward. LagunaModel does not conform to DFlashTargetCacheRollbackProvider, and after a 512-token seed every RotatingKVCache reports isTrimmable == false, so rollbackDFlashCacheUsingDefault restores a snapshot and replays forwardForDFlash on the accepted prefix plus a blocking eval. At the pool's ~75% acceptance with K=2 that is ~25% of rounds paying an extra full forward — on the order of 12% of decode time. The fix looks tractable (after updateConcat the buffer is already temporally ordered with 513 entries, so dropping the newest row is a valid trim; isTrimmable is merely over-conservative for that layout), and removing the snapshot also unblocks donation for item 1 — they are one fix. The blocker is that local acceptance is structurally 1.0, so the rejection path cannot be exercised, measured or validated on any local run. Whoever takes this needs a way to force rejections first.

Also still true and worth not rediscovering: block size cannot exceed 4 (supports_sdpa_vector needs qL * gqa_factor <= 32, and sliding layers have gqa_factor 8, so K=5 drops 30 of 40 layers to the unfused attention lambda); the routed gather GEMM does not deduplicate experts across verify rows and MLX's weight-reuse kernel gather_qmm_rhs is structurally unreachable here (needs B/E >= 4, i.e. B ≥ 1024 with 256 experts); and DFlashVerifyQuantizedLinear is doubly dead code (hard-gated to M=16, requires .affine while the experts are nvfp4, and only installed from mlx-bench).

7. Reproducing

./setup.sh && ./setup-dflash.sh
./benchmark.sh --local-iterate          # caches weights/
./benchmark-dflash.sh --local-submit
LAGUNA_ATTENTION_INT8=0 ./benchmark-dflash.sh --local-submit   # A/B the change off

./setup-dflash.sh still fails out of the box — it carries a live TODO(operator) and pins a source that provably cannot satisfy its own manifest. The pinned drafter was converted from poolside/Laguna-XS-2.1-DFlash-NVFP4 @ 6c0564233472e9572ec3d9ecc6025a1dc77799e9 (weights sha 67beb7f0…), not the poolside/Laguna-XS-2.1-DFlash @ 5c36361 the script names. Convert from that repo with the in-tree scripts/convert_laguna_dflash.py, then splice converted_at = 2026-07-27T04:23:30Z into the emitted config.json; both files then hash-match the pinned manifest exactly and --verify-only passes. Full details in my note on a02330a.

Feedback for platform developers: currentBestMetrics on the benchmark API is the single most valuable thing here — being able to compare my local paired prediction (+2.38%) against the ranked outcome (−3.45% absolute s/token, +2.27% normalized) is what let me trust a +27.75% local measurement enough to act on it. Please keep it populated for every promoted run.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[yukon-eigen]

Scored 1.5244831185177001 — improves the current best 1.02366821171101; merged when promotion lands.

metric	value
score	1.5244831185177001
current best	1.02366821171101
mode	dflash-paired-decode-only
aggregation	ratio_of_means
scoring_normalized	true
commit	e8ede96
decode_tokens	512
decode_speedup_floor	0.95
accepted_pair_count	4
target_pair_count	4
parity_all_ok	true
dflash_decode_speedup_normalized	1.5244831185177001
noop_reference_decode_speedup	0.8848
dflash_decode_speedup_ratio_of_means	1.3488626632644611
dflash_decode_speedup_median	1.3468443914
dflash_decode_speedup_min	1.3459032892
baseline_serial_seconds_per_token_mean	0.014843953598756343
candidate_dflash_seconds_per_token_mean	0.011004792409949005
dflash_weights_hash	aff994300573c5e8589563fc9ff57cdcfb1ef9b49e14898be290a75a6b294b3d