bug(kv): quantized-V flash-decode diverges — *_sym regress at every ctx, now slower at 64k than the CPU path they replaced

[Pushkinist]

Found by the post-0.3.0 Bonsai-8B KV re-validation (main d8d7463, release-perf, M5 Max, nax=288, temp=0, n=3 / n=1@64k, same harness as docs/models/bonsai/8B/rMLX.md).

Symptom: the *_sym family regressed at EVERY context

codec	4k (0.2.5 → now)	64k (0.2.5 → now)
iso3_sym	133.2 → 18.9 (−86%)	5.9 → 2.4 (−59%)
iso4_sym	131.4 → 17.6 (−87%)	5.7 → 1.8 (−68%)
rotor3_sym	136.2 → 18.0 (−87%)	5.8 → 2.3 (−60%)
rotor4_sym	132.7 → 16.9 (−87%)	7.3 → 1.7 (−77%)

The symv flash-decode kernel (#219/#220, always-on after the sole-store work #279/#281/#285) provably dispatches on GPU (iso_flash_decode_symv_sdpa/rotor_flash_decode_symv_sdpa ×182 per run). It was built to fix the 64k CPU-dequant collapse — instead it is now 2.4–4.3× SLOWER than the CPU path it replaced at 64k. A GPU kernel losing to CPU dequant is a defect, not a tradeoff.

Root cause is the quantized-V path, not flash-decode generally

Marginal decode cost, fit itl = a + b·kv_seq over 4k→64k (ms per 1k KV tokens):

family	marginal	vs none
none (bf16 K+V)	0.32	1×
K-only k_iso3/4, k_rotor3/4 (quant K, bf16 V)	2.32 / 3.34 / 2.94 / 3.01	7–10×
*_sym (quant K and V)	5.92 / 8.12 / 6.17 / 8.61	18–27×

Quantizing V roughly doubles the marginal cost on top of quantizing K. The K-only ratio to bf16 is flat (~5→7-9× across 4k→64k); the _sym ratio diverges (7.1× @4k → 15.5× @64k), so no crossover is reachable at any context. Consistent with the pre-existing "4-bit-V codecs crater decode" finding for k8v4/rot_k_tq4v on this model.

And the memory win doesn't materialize

At 64k: none 10536 MB, iso3_sym 10640, k_iso3 10588, rotor3_sym 12928, k_rotor3 16255. The sole-store work shrank _sym a lot (30608 → 10640, −65%) but it still lands at or above bf16. So _sym costs the same memory as none and decodes 15× slower — strictly dominated on both axes, at every context.

Proposed fix

1. Stop routing *_sym decode through the symv kernel by default — it is strictly dominated today. Either restore a bf16-V decode path or gate the kernel off until it is competitive.
2. Attack the V-side dequant in the flash-decode step — that is where the extra ~3–5 ms/1k lives. The K-side path (K-only codecs) is already ~2.5× cheaper and stable, so the kernel scaffolding is not the problem.
3. Underlying efficiency ceiling is the feat(kv): fused quantized decode SDPA — make KV codecs live at decode (read quant store directly) #45 flash-decode-over-quant class.

Contrast — the K-only work is a genuine win (do not revert)

Same push rescued k_iso*/k_rotor* from CPU-bound unusability: 4k 0.7–5.8 → 21–26 TPS, and they now run the full range (baseline capped them at 8–16k). --rotor-qjl default off alone is ~30–34×.

Raw cells + kernel-dispatch proofs captured in the sweep checkpoint; real runs.db untouched (scratch RMLX_HOME).

🤖 Generated with Claude Code

[Pushkinist]

Correction: root cause re-scoped after falsification

My original "quantized-V dequant is the culprit" framing was wrong, and the P2-C chunk refuted it. Recording the correction rather than silently editing.

What refuted it: two codecs that quantize V and are NOT expensive —

• rot_k_tq4v (rotor-K + tq4-V, both quantized): marginal 1.13 ms/1k, ~parity with its 0.2.5 baseline.
• iso3/rotor3 (full K+V quant): 0.79 / 0.47 ms/1k.

If reading V from a quant store were inherently expensive, none of those could sit near bf16.

The actual empirical split — whether the codec dispatches a flash-decode-over-quant kernel:

dispatches kernel?	codecs (marginal ms per 1k KV tokens, 8k→64k fit)
no	none 0.32 · rotor3 0.47 · rotor4 0.52 · iso3 0.79 · rot_k_tq4v 1.13 · iso4 1.36 · k8v4 2.18
yes — K-only (iso_flash_decode_sdpa/rotor_flash_decode_sdpa)	k_iso3 2.21 · k_rotor3 2.75 · k_rotor4 3.11 · k_iso4 3.37
yes — symv K+V (*_flash_decode_symv_sdpa)	iso3_sym 6.08 · rotor3_sym 6.20 · iso4_sym 8.48 · rotor4_sym 9.02

Every kernel-dispatching codec is ≥2.21; every non-dispatching one is ≤2.18. Clean separation. Within the kernel family the symv (K+V) variants are ~2.5× worse than K-only — so V does add cost inside the kernel, but it is not the discriminator across codecs.

This does not change the reported harm or the proposed fix. *_sym still regressed at every context and is still slower at 64k than the CPU path it replaced, because it moved onto these kernels. What changes is the diagnosis: the target is the flash-decode-over-quant kernel path's marginal cost (7–28× bf16), i.e. the #45 class — not a V-specific dequant bug. k8v4's 2.18 is the separate, pre-existing tq4-V cost already documented in the 0.2.5 doc, and it is at parity (not a new regression).

Also newly measured, separate from this issue: iso3/iso4 regressed 25–33% at 8k–32k vs baseline (iso3 8k 115.3→86.1) while improving at 64k (10.8→17.5, old CPU collapse gone); rotor3/4 regressed ~10–30% across 8k–64k. Will file separately once the matrix is complete.

🤖 Generated with Claude Code