perf(deepseek-v4): raise MXFP4 large-block threshold to 16K - #2562
Conversation
|
I reviewed and benchmarked #2559, #2562, and #2563 together since they merge cleanly as a set, so I am posting the same summary on all three. Setup: M3 Ultra 512 GB, DeepSeek-V4-Flash-0731 oQ4e (4-bit top level with mxfp4 experts, so all three changes are active), MTP off, single request, cold prefill with cached_tokens=0 verified on every run, one run per point, current main vs main plus all three PRs, using the built-in throughput benchmark (tg128). With the prompt cache disabled, which gives 2048-token prefill steps matching the #2558 methodology:
tg was unchanged everywhere, which is expected for prefill-only changes. This independently reproduces the #2558 ablation on a different checkpoint, so the numbers hold up well. One finding on my side rather than yours: with the default server config the paged SSD cache aligns the rotating block size to 512, which clamps prefill to 512-token chunks, and in that regime these gains mostly disappear (the MoE route count stays below the MXFP4 crossover and the ratio-128 win shrinks with the chunk). Raising the DeepSeek V4 block alignment to 2048 with the cache still enabled recovers the full numbers above (549.5 / 558.7 / 524.2 / 461.6 pp tok/s). I will handle that follow-up separately so default-config users actually see the improvement. Merging all three. |
Summary
OMLX_DEEPSEEK_MXFP4_LARGE_BLOCK_MIN_ROUTES=8192No kernel math changes in this PR.
Performance evidence
The controlled M3 Ultra ablation in #2558 measured median prompt throughput increasing from 479.08 to 487.91 tok/s, a 1.84% improvement. Median prefill time decreased from 35.94 to 35.29 seconds.
This measurement is scoped to:
DeepSeek-V4-Flash-0731, revision7872f01b1d1fe23eabc4c98b48bffcef5a386062d2575b1df5a4012966839f75fdb200e7e0b20743a1465f4b5ee68dbd173c138dd65718bd06957439b66e99069e91f50364bf81f1cached_tokens=0READYresponseThe linked evidence package preserves the raw investigation artifacts without adding them to this PR.
This is not an unconditional cross-chip performance claim.
Cross-chip behavior
OMLX_DEEPSEEK_MXFP4_LARGE_BLOCK_MIN_ROUTES=8192to restore the previous MXFP4 crossovermx.gather_qmmfallback by default; its existing controls are unchangedValidation
uv run pytest tests/test_glm_moe_dsa_patch.py tests/test_deepseek_v4_patch.py -q -rs— 114 passed, 4 skipped because the optional native extension was unavailable in the clean PR worktreeuv run ruff check --select E,F,W ...passedgit diff --checkpassedRefs #2558.