Speculative decoding for TTS codec prediction

[ymote]

Context

Qwen3-TTS spends ~40% of generation time in the code predictor (5 layers × 15 sequential codebook predictions per frame). The talker (28 layers) is the other ~60%.

MiniCPM-SALA already has speculative decoding infrastructure in mlx-rs-core/src/speculative.rs showing 3.12x speedup with 52.7% acceptance rate.

Opportunity

• Draft a smaller/faster codec predictor (e.g., 2 layers instead of 5)
• Verify with the full predictor
• Or: predict multiple codebooks in parallel (non-autoregressive)

Either approach could yield 20-40% total TTS speedup — far more than any kernel optimization.

Why kernel optimization hit a wall

Benchmarked on M3 Max with 8-bit Qwen3-TTS-1.7B:

• Flash attention kernel: -10% (slower than MLX SDPA for Q=1)
• Fused QK-norm+RoPE: 0% (MLX lazy eval already batches)
• Batched QKV: 0% (MLX already batches internally)
• MLX's quantized matmul is fused and M3-tuned — ~46 fps is the ceiling for this architecture