Skip to content

perf(deepseek-v4): raise MXFP4 large-block threshold to 16K - #2562

Merged
jundot merged 1 commit into
jundot:mainfrom
DiscoStew6082:disco/deepseek-v4-mxfp4-threshold
Aug 10, 2026
Merged

perf(deepseek-v4): raise MXFP4 large-block threshold to 16K#2562
jundot merged 1 commit into
jundot:mainfrom
DiscoStew6082:disco/deepseek-v4-mxfp4-threshold

Conversation

@DiscoStew6082

Copy link
Copy Markdown
Contributor

Summary

  • keep MXFP4 BM16/variant 1 blocks through 16,383 routes and switch to BM32/variant 2 at 16,384 routes
  • preserve the existing 8,192-route crossover for affine 2/3-bit and mixed native block plans
  • allow unmeasured pre-NAX systems to restore the previous MXFP4 crossover with OMLX_DEEPSEEK_MXFP4_LARGE_BLOCK_MIN_ROUTES=8192
  • add focused boundary coverage for the default, override, and unchanged affine behavior

No kernel math changes in this PR.

Performance evidence

The controlled M3 Ultra ablation in #2558 measured median prompt throughput increasing from 479.08 to 487.91 tok/s, a 1.84% improvement. Median prefill time decreased from 35.94 to 35.29 seconds.

This measurement is scoped to:

  • Mac Studio with M3 Ultra and 512 GB unified memory
  • DeepSeek-V4-Flash-0731, revision 7872f01b1d1fe23eabc4c98b48bffcef5a386062
  • upstream source d2575b1df5a4012966839f75fdb200e7e0b20743
  • fixed 17,219-token prompt, SHA-256 a1465f4b5ee68dbd173c138dd65718bd06957439b66e99069e91f50364bf81f1
  • 2,048-token prefill steps, MTP enabled with three draft tokens, cache disabled, and cached_tokens=0
  • one warmup followed by five measured cold runs, all producing the expected READY response

The linked evidence package preserves the raw investigation artifacts without adding them to this PR.

This is not an unconditional cross-chip performance claim.

Cross-chip behavior

  • pre-NAX systems can set OMLX_DEEPSEEK_MXFP4_LARGE_BLOCK_MIN_ROUTES=8192 to restore the previous MXFP4 crossover
  • M5/NAX prefill continues to use the existing stock mx.gather_qmm fallback by default; its existing controls are unchanged
  • affine and mixed native block plans retain their existing 8,192-route crossover

Validation

  • uv run pytest tests/test_glm_moe_dsa_patch.py tests/test_deepseek_v4_patch.py -q -rs — 114 passed, 4 skipped because the optional native extension was unavailable in the clean PR worktree
  • focused default and override threshold coverage — 6 passed
  • direct production override probe passed
  • uv run ruff check --select E,F,W ... passed
  • Python compilation and git diff --check passed
  • independent read-only code review completed with no findings

Refs #2558.

@jundot

jundot commented Aug 10, 2026

Copy link
Copy Markdown
Owner

I reviewed and benchmarked #2559, #2562, and #2563 together since they merge cleanly as a set, so I am posting the same summary on all three. Setup: M3 Ultra 512 GB, DeepSeek-V4-Flash-0731 oQ4e (4-bit top level with mxfp4 experts, so all three changes are active), MTP off, single request, cold prefill with cached_tokens=0 verified on every run, one run per point, current main vs main plus all three PRs, using the built-in throughput benchmark (tg128).

With the prompt cache disabled, which gives 2048-token prefill steps matching the #2558 methodology:

context main pp tok/s +3 PRs pp tok/s delta
4,096 464.1 546.3 +17.7%
16,384 487.8 559.6 +14.7%
65,536 463.4 525.1 +13.3%
131,072 416.1 460.8 +10.7%

tg was unchanged everywhere, which is expected for prefill-only changes. This independently reproduces the #2558 ablation on a different checkpoint, so the numbers hold up well.

One finding on my side rather than yours: with the default server config the paged SSD cache aligns the rotating block size to 512, which clamps prefill to 512-token chunks, and in that regime these gains mostly disappear (the MoE route count stays below the MXFP4 crossover and the ratio-128 win shrinks with the chunk). Raising the DeepSeek V4 block alignment to 2048 with the cache still enabled recovers the full numbers above (549.5 / 558.7 / 524.2 / 461.6 pp tok/s). I will handle that follow-up separately so default-config users actually see the improvement. Merging all three.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants