Skip to content

Commit fcb4d8e

Browse files
Re-add vllm-project/vllm#58671's ROCm paged MXFP4 sparse indexer
Stacked on #3555. Re-adds the --attention-config flag enabling #58671's ROCm paged MXFP4 sparse-logits indexer and the --block-size 128 workaround its active fp4 indexer needs (vLLM's block-size auto-resolution otherwise picks 64 across this model's 4+ attention backends). Draft until #58671 merges upstream and lands in a nightly image, so #3555 can run now and this PR can re-sweep immediately once it's available. Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 3e3ad54 commit fcb4d8e

3 files changed

Lines changed: 22 additions & 5 deletions

File tree

‎inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,9 @@ base:
77
model:
88
path: hf:deepseek-ai/DeepSeek-V4.1-Flash
99
# TODO: repin once vllm-project/vllm#58655 (mHC fused Triton seams),
10-
# #53492 (sparse MLA Gluon kernel) and #58208 (DSA candidate
11-
# block-selection topk) are merged upstream and released in a nightly
12-
# image.
10+
# #58671 (paged MXFP4 sparse indexer), #53492 (sparse MLA Gluon kernel)
11+
# and #58208 (DSA candidate block-selection topk) are merged upstream and
12+
# released in a nightly image.
1313
container: vllm/vllm-openai-rocm:nightly-rocm100-TBD
1414
precision: fp4
1515
resources:
@@ -44,6 +44,16 @@ base:
4444
# aiter_triton_mxfp4_bf16 would force the W4A16 Triton kernel.
4545
moe-backend: aiter
4646
gpu-memory-utilization: 0.9
47+
# Enables vllm-project/vllm#58671's ROCm paged MXFP4 sparse-logits
48+
# indexer kernel instead of the dense fp8 indexer path.
49+
attention-config: '{"indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
50+
# With 4+ attention backends active (SparseMLA, SparseSWA, the MXFP4
51+
# indexer, Compressor), vLLM's block-size auto-resolution picks the
52+
# smallest size every backend merely supports (64), not the 128 the
53+
# indexer and SparseMLA backends actually prefer for this fp4 indexer
54+
# path. Force 128 explicitly; see vllm-project/vllm's
55+
# Platform._preferred_block_size_for_backends.
56+
block-size: 128
4757
# Five-token DSpark with probabilistic drafting. Throughput runs replace
4858
# block rejection with the golden acceptance length. Adaptive
4959
# verification trims requests on device, which the ROCm DeepSeek-V4

‎inferencex-e2e/configs/amd-master.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1386,8 +1386,8 @@ dsv4-fp4-mi355x-sglang-agentic-mtp:
13861386
# Two GPUs per server doubles the servers per node and is the layout that
13871387
# decides whether DSv4.1-Flash is throughput- or interactivity-bound here.
13881388
dsv41flash-fp4-mi355x-vllm-agentic-dspark:
1389-
# TODO: repin once vllm-project/vllm#58655, #53492 and #58208 are merged
1390-
# upstream and released in a nightly image (see agentic.yaml).
1389+
# TODO: repin once vllm-project/vllm#58655, #58671, #53492 and #58208 are
1390+
# merged upstream and released in a nightly image (see agentic.yaml).
13911391
image: vllm/vllm-openai-rocm:nightly-rocm100-TBD
13921392
model: deepseek-ai/DeepSeek-V4.1-Flash
13931393
model-prefix: dsv41flash

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9010,3 +9010,10 @@
90109010
- "Force VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=True for vllm-project/vllm#53492's Gluon sparse-MLA kernel. Unpin image to TBD pending #58208 (faster DSA candidate topk selection), #58655 (fused mHC Triton kernel) and #53492 (sparse MLA Gluon kernel) landing in a nightly release: better topk/mHC/MLA kernels with better compute/communication stream overlap. Does not include vllm-project/vllm#58671's MXFP4 sparse-logits indexer: it isn't merging in time, and reverting a dependency it exposed (#58208's DSA candidate-block topk, vllm-project/vllm#59125) showed the fp8 dense indexer path #58671 would otherwise replace costs roughly 15% interactivity and 10% tail e2e latency at TP2 c16, with no measurable throughput/GPU change."
90119011
- "设置 VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=True 以启用 vllm-project/vllm#53492 的 Gluon sparse-MLA kernel。将镜像取消固定为 TBD,待 #58208(更快的 DSA 候选块 topk 选择)、#58655(融合 mHC Triton kernel)与 #53492(sparse MLA Gluon kernel)合并至上游并随 nightly 镜像发布:更优的 topk/mHC/MLA kernel,以及更好的计算/通信流重叠模式。不包含 vllm-project/vllm#58671 的 MXFP4 稀疏 logits indexer:该 PR 未能及时合并;回滚它所暴露的一个依赖问题(#58208 的 DSA 候选块 topk,vllm-project/vllm#59125)后测得,#58671 本应替换的 fp8 稠密 indexer 路径在 TP2 c16 下约有 15% 的 interactivity 与 10% 的尾部 e2e 延迟损失,而 throughput/GPU 无明显变化。"
90129012
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3555
9013+
9014+
- config-keys:
9015+
- dsv41flash-fp4-mi355x-vllm-agentic-dspark
9016+
description:
9017+
- "Stacked on #3555. Re-add --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' and --block-size 128 to enable vllm-project/vllm#58671's ROCm paged MXFP4 sparse-logits indexer once it merges upstream, replacing the dense fp8 indexer path. A live A/B test (TP2 c16, matched 900s window, #58208 reverted via #59125 so the dense fallback doesn't crash) measured #58671 giving +14.7/+14.9% p50/p90 interactivity and -9.5/-10.8% p50/p90 e2e latency over the dense path, with throughput/GPU unchanged. Draft until #58671 merges and lands in a nightly image; kept separate from #3555 so that PR can run now and this one can re-sweep immediately once #58671 is available."
9018+
- "基于 #3555 之上。重新添加 --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' 与 --block-size 128,以便在 vllm-project/vllm#58671 合并上游后启用其 ROCm 分页 MXFP4 稀疏 logits indexer,取代稠密 fp8 indexer 路径。一次真实的 A/B 测试(TP2、并发 16,匹配 900 秒窗口,并通过 #59125 回滚 #58208 以避免稠密回退路径崩溃)测得 #58671 相较稠密路径带来 interactivity p50/p90 提升 14.7%/14.9%,e2e 延迟 p50/p90 降低 9.5%/10.8%,而 throughput/GPU 无变化。本 PR 保持 draft 状态,直到 #58671 合并并随 nightly 镜像发布;与 #3555 分开提交,以便该 PR 可先行运行,本 PR 待 #58671 可用后立即重新扫描。"
9019+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3571

0 commit comments

Comments
 (0)