Finding
Post-0.3.0 Bonsai-8B re-validation at 4k (main d8d7463, nax=288). The K-only codecs are no longer CPU-bound — the Metal flash-decode kernel provably dispatches every decode step — but they still sit far below the none ceiling because the growing-prefix dequant restages host-side on every step.
| codec |
0.3.0-era now |
0.2.5 baseline |
Δ |
kernel dispatch proof |
| k_iso3 |
25.3 |
5.8 |
+336% |
iso_flash_decode_sdpa: dispatch ×260 |
| k_iso4 |
25.2 |
2.1 |
+1100% |
iso_flash_decode_sdpa: dispatch ×260 |
| k_rotor3 |
22.9 |
0.7 |
+3173% |
rotor_flash_decode_sdpa: dispatch ×260 |
| k_rotor4 |
23.6 |
0.7 |
+3271% |
rotor_flash_decode_sdpa: dispatch ×260 |
Reference cell: none = 133.3 TPS at the same 4k prompt. QJL ablation isolates the kernel's contribution: k_rotor3 22.9 (--rotor-qjl off, kernel) vs 0.7 (--rotor-qjl on, forced CPU) = 32.7×.
Root cause (source-documented)
crates/rmlx-kv-quant/src/quant.rs cpu_hot_path_reason() documents the residual host-side restage: the kernel dispatches, but the dequant of the growing prefix is staged through the host each step, so decode cost is bounded by that transfer rather than by the kernel. The result is a hybrid state — not the old "no kernel at all" single-digit TPS, but not fully GPU-resident either.
Why it matters
Hard rule 10 ("every KV codec decodes on-GPU; a CPU dequant fallback is a bug, not a valid mode") and the standing directive to minimize CPU work in the KV path. These four codecs are the last ones with a CPU hot path on the decode critical path. Closing it should lift them from ~23–25 TPS toward the plain-codec range.
Fix direction
Keep the prefix resident on-device across decode steps (ring/packed-store already exists for the sym families) so the kernel reads the quant store directly with no per-step host restage — the same treatment #231/#278/#279 gave the rotor/iso ring-only tail.
Evidence CSV: .rmlx/bench/bonsai_kvsweep_d8d7463.csv (gitignored, local).
🤖 Generated with Claude Code
Finding
Post-0.3.0 Bonsai-8B re-validation at 4k (main
d8d7463, nax=288). The K-only codecs are no longer CPU-bound — the Metal flash-decode kernel provably dispatches every decode step — but they still sit far below thenoneceiling because the growing-prefix dequant restages host-side on every step.iso_flash_decode_sdpa: dispatch×260iso_flash_decode_sdpa: dispatch×260rotor_flash_decode_sdpa: dispatch×260rotor_flash_decode_sdpa: dispatch×260Reference cell:
none= 133.3 TPS at the same 4k prompt. QJL ablation isolates the kernel's contribution: k_rotor3 22.9 (--rotor-qjl off, kernel) vs 0.7 (--rotor-qjl on, forced CPU) = 32.7×.Root cause (source-documented)
crates/rmlx-kv-quant/src/quant.rscpu_hot_path_reason()documents the residual host-side restage: the kernel dispatches, but the dequant of the growing prefix is staged through the host each step, so decode cost is bounded by that transfer rather than by the kernel. The result is a hybrid state — not the old "no kernel at all" single-digit TPS, but not fully GPU-resident either.Why it matters
Hard rule 10 ("every KV codec decodes on-GPU; a CPU dequant fallback is a bug, not a valid mode") and the standing directive to minimize CPU work in the KV path. These four codecs are the last ones with a CPU hot path on the decode critical path. Closing it should lift them from ~23–25 TPS toward the plain-codec range.
Fix direction
Keep the prefix resident on-device across decode steps (ring/packed-store already exists for the sym families) so the kernel reads the quant store directly with no per-step host restage — the same treatment #231/#278/#279 gave the rotor/iso ring-only tail.
Evidence CSV:
.rmlx/bench/bonsai_kvsweep_d8d7463.csv(gitignored, local).🤖 Generated with Claude Code