docs(bench): Bonsai-8B full KV re-validation at d8d7463 (post-0.3.0 .metal work)

[Pushkinist]

Full 25-codec × 5-context re-measurement of docs/models/bonsai/8B/rMLX.md against the 0.2.5 baseline, same harness/protocol (M5 Max, release-perf, nax=288, temp=0, n=3 / n=1@64k). 125/125 cells, CSV-audited — zero missing, zero failed, zero incoherent.

What the .metal work delivered

K-only family rescued. k_iso3/4, k_rotor3/4 went from CPU-bound-unusable (baseline capped them at 8–16k; 0.4–5.8 TPS) to GPU-functional across the full 4k–64k range, with iso_flash_decode_sdpa/rotor_flash_decode_sdpa dispatch confirmed at both 4k and 64k. The --rotor-qjl default flip on→off is the entire delta for k_rotor (~30–34×; explicit on reproduces the old 0.7 TPS exactly). KV memory dropped to ~1.00× none (from 1.64×).

What regressed

*_sym (#292) — regressed at every context; now slower at 64k (1.7–2.4 TPS) than the CPU path it replaced (5.7–7.3). The kernel provably dispatches on GPU, so this is marginal cost, not a dispatch failure: 6.25–9.42 ms per 1k KV tokens vs K-only 2.25–3.49 and none 0.33. Scoped precisely — it is the V-side dequant inside the symv kernel, not quantized-V in general (rot_k_tq4v quantizes V on the generic path at a cheap 1.16 ms/1k).

11 kernel-less codecs (#293) — iso3/4, rotor3/4, planar, planar3, planar_k, k8vturbo3, k8vturbo3tcq, tsym3, tsym4 down 6–34% at 16k–64k, while none and k8v8 hold exact parity. Hypothesis given but explicitly flagged as not bisected, with a concrete bisect recommendation rather than a guess.

The honest headline

After all of it, none (bf16) is still the fastest codec at every context and the smallest KV at every context — every quant codec lands at or above 10536 MB @64k. Its lead over the mlx-lm champion is +21…+31%. The kernels made broken codecs functional; none of them is yet a reason to choose quantized KV on this model.

New in the doc

§2.2 marginal-cost table — fits ITL = a + b·kv_seq per codec, grouped by whether a flash-decode kernel dispatches. That slope, not a single-context ratio, is what predicts whether a codec can ever beat bf16 at long context.

Also records two harness findings: #291 (rmlx baseline --prompt-tokens tokenizes the raw JSON envelope) and the prefill-only CPU V-encode in iso3/4/rotor3/4 (pre-existing, decode is full-speed GPU).

Docs-only. All runs used a scratch RMLX_HOME; the real runs.db was never written.

🤖 Generated with Claude Code