Skip to content

Commit 402aba1

Browse files
Repin DSv4.1-Flash MI355X vLLM AgentX to nightly-rocm100-ac9126e5 (includes #58671)
Enables vllm-project/vllm#58671's ROCm paged MXFP4 sparse-logits indexer via --attention-config and --block-size 128, replacing the dense fp8 indexer path. A live A/B test (TP2 c16, matched 900s window) measured #58671 giving +14.7/+14.9% p50/p90 interactivity and -9.5/-10.8% p50/p90 e2e latency over the dense path, with throughput/GPU unchanged. Also drops c128 from both TP2 and TP4 conc-lists for this recipe. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 1c5e6b9 commit 402aba1

3 files changed

Lines changed: 32 additions & 41 deletions

File tree

‎inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml‎

Lines changed: 16 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,10 @@ base:
66
name: dsv41flash-fp4-mi355x-vllm-agentic
77
model:
88
path: hf:deepseek-ai/DeepSeek-V4.1-Flash
9-
container: vllm/vllm-openai-rocm:nightly-rocm100-7f1a5398e9610d96c473931a26c0e12bbe0d0423
9+
# Repinned to a nightly build that includes vllm-project/vllm#58671
10+
# (paged MXFP4 sparse indexer), #58655 (mHC fused Triton seams) and
11+
# #53492 (sparse MLA Gluon kernel).
12+
container: vllm/vllm-openai-rocm:nightly-rocm100-ac9126e58aa7bbab1856ba6593ba4d5003fea516
1013
precision: fp4
1114
resources:
1215
gpu_type: mi355x
@@ -40,6 +43,12 @@ base:
4043
# aiter_triton_mxfp4_bf16 would force the W4A16 Triton kernel.
4144
moe-backend: aiter
4245
gpu-memory-utilization: 0.9
46+
# Enables vllm-project/vllm#58671's ROCm paged MXFP4 sparse-logits
47+
# indexer kernel instead of the dense fp8 indexer path.
48+
attention-config: '{"indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
49+
# Auto-resolution picks 64 (min across backends); force 128, which
50+
# the indexer and SparseMLA actually prefer.
51+
block-size: 128
4352
# Five-token DSpark with probabilistic drafting. Throughput runs replace
4453
# block rejection with the golden acceptance length. Adaptive
4554
# verification trims requests on device, which the ROCm DeepSeek-V4
@@ -54,6 +63,7 @@ base:
5463
VLLM_ROCM_USE_AITER: '1'
5564
VLLM_ROCM_USE_AITER_MOE: '1'
5665
AITER_TRITON_LOG_LEVEL: ERROR
66+
VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA: 'True'
5767
# DeepseekV41ForCausalLM is not torch-compiled upstream; breakable
5868
# graphs keep FULL_AND_PIECEWISE capture working.
5969
VLLM_USE_BREAKABLE_CUDAGRAPH: '1'
@@ -70,11 +80,11 @@ base:
7080
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126
7181
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:'
7282

73-
# One variant per point. Engram tables move to pinned host memory at TP2 and
74-
# at c128, where resident tables leave too little KV. Graph capture covers
75-
# 2x CONC (at least 128 sequences) x (1 + 5 drafts), up to 2048 tokens. The
76-
# prefill chunk shrinks at TP2 from c64 and at c128 to buy KV room, and where
77-
# it drops below 1024 x 6 slots the scheduler is capped at the captured shape.
83+
# One variant per point (c128 excluded for both TP2 and TP4). Engram tables
84+
# move to pinned host memory at TP2, where resident tables leave too little
85+
# KV. Graph capture covers 2x CONC x (1 + 5 drafts). The prefill chunk shrinks
86+
# at TP2 from c64 to buy KV room, and where it drops below 1024 x 6 slots the
87+
# scheduler is capped at the captured shape.
7888
override_tp4_c1:
7989
roles:
8090
agg:
@@ -166,19 +176,6 @@ override_tp4_c64:
166176
env:
167177
CONC: '64'
168178

169-
override_tp4_c128:
170-
roles:
171-
agg:
172-
gpus: 4
173-
args:
174-
tensor-parallel-size: 4
175-
engram-config: '{"cpu_offload":true}'
176-
max-cudagraph-capture-size: 2048
177-
max-num-batched-tokens: 8192
178-
benchmark:
179-
env:
180-
CONC: '128'
181-
182179
override_tp2_c1:
183180
roles:
184181
agg:
@@ -269,17 +266,3 @@ override_tp2_c64:
269266
benchmark:
270267
env:
271268
CONC: '64'
272-
273-
override_tp2_c128:
274-
roles:
275-
agg:
276-
gpus: 2
277-
args:
278-
tensor-parallel-size: 2
279-
engram-config: '{"cpu_offload":true}'
280-
max-cudagraph-capture-size: 2048
281-
max-num-batched-tokens: 4096
282-
max-num-seqs: 256
283-
benchmark:
284-
env:
285-
CONC: '128'

‎inferencex-e2e/configs/amd-master.yaml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1457,8 +1457,9 @@ dsv4-fp4-mi355x-sglang-agentic-mtp:
14571457
# Two GPUs per server doubles the servers per node and is the layout that
14581458
# decides whether DSv4.1-Flash is throughput- or interactivity-bound here.
14591459
dsv41flash-fp4-mi355x-vllm-agentic-dspark:
1460-
# ROCm 10.0 nightly channel, shared with kimik3-fp4-mi355x-vllm-agentic-mtp.
1461-
image: vllm/vllm-openai-rocm:nightly-rocm100-29468dde8b515031dc6d4d9d06bf0a2fa0442098
1460+
# Repinned to a nightly build that includes vllm-project/vllm#58671,
1461+
# #58655 and #53492 (see agentic.yaml).
1462+
image: vllm/vllm-openai-rocm:nightly-rocm100-ac9126e58aa7bbab1856ba6593ba4d5003fea516
14621463
model: deepseek-ai/DeepSeek-V4.1-Flash
14631464
model-prefix: dsv41flash
14641465
runner: cluster:mi355x-amds
@@ -1470,12 +1471,12 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark:
14701471
- dram-utilization: 0.60
14711472
search-space:
14721473
# The recipe only changes the memory split where a point ran short of KV:
1473-
# TP=4 offloads the engram tables at c128, and the prefill chunk shrinks at
1474-
# TP=2 c64, TP=4 c128 and TP=2 c128. Every point is measured here rather
1475-
# than combined from the earlier run: the image move leaves concurrency
1476-
# 1-32 measured only on the superseded nightly-eed1f3d0 pin.
1477-
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
1478-
- { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
1474+
# the prefill chunk shrinks at TP=2 c64. c128 is excluded for both TP=2
1475+
# and TP=4. Every point is measured here rather than combined from the
1476+
# earlier run: the image move leaves concurrency 1-32 measured only on
1477+
# the superseded nightly-eed1f3d0 pin.
1478+
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
1479+
- { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
14791480

14801481
# The golden acceptance curve remains the provisional GLM-5.2-derived curve
14811482
# documented in infx/golden_al_distribution/glm5.3_mtp.yaml.

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9168,3 +9168,10 @@
91689168
- "Relevant ATOM changes in the range: V4 decode reuses the sparse prefill ASM (ROCm/ATOM#2271), greedy sampler picks via aiter.topk_select (ROCm/ATOM#2244), and FP8 block scales declared scale_fmt ue8m0 are now stored as E8M0 on gfx950 by default (ROCm/ATOM#2419; previously FP32 unless ATOM_FP8_BLOCKSCALE_USE_E8M0_SCALE=1). The upstream DeepSeek-V4 recipes are unchanged across the range, and every recipe flag and choice (all2all-backend rccl, dp-load-balance least_tokens, moe-backend standard) remains valid."
91699169
- "No data-type or precision change to the DeepSeek-V4-Pro-0813 DSpark draft: no online quantization is configured, so it keeps its checkpoint precision; the E8M0 scale storage represents the checkpoint's power-of-two block scales exactly. kv-cache-dtype and index-cache-dtype touch cache storage only."
91709170
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3605
9171+
9172+
- config-keys:
9173+
- dsv41flash-fp4-mi355x-vllm-agentic-dspark
9174+
description:
9175+
- "Repin to vllm/vllm-openai-rocm:nightly-rocm100-ac9126e58aa7bbab1856ba6593ba4d5003fea516, which includes vllm-project/vllm#58671, #58655 and #53492. Re-add --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' and --block-size 128 to enable #58671's ROCm paged MXFP4 sparse-logits indexer, replacing the dense fp8 indexer path. A live A/B test (TP2 c16, matched 900s window, #58208 reverted via #59125 so the dense fallback doesn't crash) measured #58671 giving +14.7/+14.9% p50/p90 interactivity and -9.5/-10.8% p50/p90 e2e latency over the dense path, with throughput/GPU unchanged. Drops c128 from both TP2 and TP4 conc-lists for this recipe."
9176+
- "固定到 vllm/vllm-openai-rocm:nightly-rocm100-ac9126e58aa7bbab1856ba6593ba4d5003fea516,该镜像包含 vllm-project/vllm#58671、#58655 与 #53492。重新添加 --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' 与 --block-size 128,以启用 #58671 的 ROCm 分页 MXFP4 稀疏 logits indexer,取代稠密 fp8 indexer 路径。一次真实的 A/B 测试(TP2、并发 16,匹配 900 秒窗口,并通过 #59125 回滚 #58208 以避免稠密回退路径崩溃)测得 #58671 相较稠密路径带来 interactivity p50/p90 提升 14.7%/14.9%,e2e 延迟 p50/p90 降低 9.5%/10.8%,而 throughput/GPU 无变化。此配方的 TP2 与 TP4 conc-list 均移除 c128。"
9177+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3571

0 commit comments

Comments
 (0)