Skip to content

Commit b07aca1

Browse files
Drop c128 from DSv4.1-Flash MI355X vLLM AgentX recipe (TP2 and TP4)
Removes the override_tp2_c128/override_tp4_c128 variants and the corresponding conc-list entries in amd-master.yaml. Co-authored-by: Claude Sonnet 5 (Cursor) Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 76fe3ab commit b07aca1

3 files changed

Lines changed: 13 additions & 40 deletions

File tree

‎inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml‎

Lines changed: 5 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -82,11 +82,11 @@ base:
8282
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126
8383
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:'
8484

85-
# One variant per point. Engram tables move to pinned host memory at TP2 and
86-
# at c128, where resident tables leave too little KV. Graph capture covers
87-
# 2x CONC (at least 128 sequences) x (1 + 5 drafts), up to 2048 tokens. The
88-
# prefill chunk shrinks at TP2 from c64 and at c128 to buy KV room, and where
89-
# it drops below 1024 x 6 slots the scheduler is capped at the captured shape.
85+
# One variant per point (c128 excluded for both TP2 and TP4). Engram tables
86+
# move to pinned host memory at TP2, where resident tables leave too little
87+
# KV. Graph capture covers 2x CONC x (1 + 5 drafts). The prefill chunk shrinks
88+
# at TP2 from c64 to buy KV room, and where it drops below 1024 x 6 slots the
89+
# scheduler is capped at the captured shape.
9090
override_tp4_c1:
9191
roles:
9292
agg:
@@ -178,19 +178,6 @@ override_tp4_c64:
178178
env:
179179
CONC: '64'
180180

181-
override_tp4_c128:
182-
roles:
183-
agg:
184-
gpus: 4
185-
args:
186-
tensor-parallel-size: 4
187-
engram-config: '{"cpu_offload":true}'
188-
max-cudagraph-capture-size: 2048
189-
max-num-batched-tokens: 8192
190-
benchmark:
191-
env:
192-
CONC: '128'
193-
194181
override_tp2_c1:
195182
roles:
196183
agg:
@@ -281,17 +268,3 @@ override_tp2_c64:
281268
benchmark:
282269
env:
283270
CONC: '64'
284-
285-
override_tp2_c128:
286-
roles:
287-
agg:
288-
gpus: 2
289-
args:
290-
tensor-parallel-size: 2
291-
engram-config: '{"cpu_offload":true}'
292-
max-cudagraph-capture-size: 2048
293-
max-num-batched-tokens: 4096
294-
max-num-seqs: 256
295-
benchmark:
296-
env:
297-
CONC: '128'

‎inferencex-e2e/configs/amd-master.yaml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1406,12 +1406,12 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark:
14061406
- dram-utilization: 0.60
14071407
search-space:
14081408
# The recipe only changes the memory split where a point ran short of KV:
1409-
# TP=4 offloads the engram tables at c128, and the prefill chunk shrinks at
1410-
# TP=2 c64, TP=4 c128 and TP=2 c128. Every point is measured here rather
1411-
# than combined from the earlier run: the image move leaves concurrency
1412-
# 1-32 measured only on the superseded nightly-eed1f3d0 pin.
1413-
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
1414-
- { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64, 128], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
1409+
# the prefill chunk shrinks at TP=2 c64. c128 is excluded for both TP=2
1410+
# and TP=4. Every point is measured here rather than combined from the
1411+
# earlier run: the image move leaves concurrency 1-32 measured only on
1412+
# the superseded nightly-eed1f3d0 pin.
1413+
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
1414+
- { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
14151415

14161416
# Speculative decoding on an agentic scenario must run with simulated
14171417
# synthetic acceptance at the committed golden AL for this model, thinking mode

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9131,6 +9131,6 @@
91319131
- config-keys:
91329132
- dsv41flash-fp4-mi355x-vllm-agentic-dspark
91339133
description:
9134-
- "Stacked on #3555. Re-add --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' and --block-size 128 to enable vllm-project/vllm#58671's ROCm paged MXFP4 sparse-logits indexer once it merges upstream, replacing the dense fp8 indexer path. A live A/B test (TP2 c16, matched 900s window, #58208 reverted via #59125 so the dense fallback doesn't crash) measured #58671 giving +14.7/+14.9% p50/p90 interactivity and -9.5/-10.8% p50/p90 e2e latency over the dense path, with throughput/GPU unchanged. Draft until #58671 merges and lands in a nightly image; kept separate from #3555 so that PR can run now and this one can re-sweep immediately once #58671 is available."
9135-
- "基于 #3555 之上。重新添加 --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' 与 --block-size 128,以便在 vllm-project/vllm#58671 合并上游后启用其 ROCm 分页 MXFP4 稀疏 logits indexer,取代稠密 fp8 indexer 路径。一次真实的 A/B 测试(TP2、并发 16,匹配 900 秒窗口,并通过 #59125 回滚 #58208 以避免稠密回退路径崩溃)测得 #58671 相较稠密路径带来 interactivity p50/p90 提升 14.7%/14.9%,e2e 延迟 p50/p90 降低 9.5%/10.8%,而 throughput/GPU 无变化。本 PR 保持 draft 状态,直到 #58671 合并并随 nightly 镜像发布;与 #3555 分开提交,以便该 PR 可先行运行,本 PR 待 #58671 可用后立即重新扫描。"
9134+
- "Stacked on #3555. Re-add --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' and --block-size 128 to enable vllm-project/vllm#58671's ROCm paged MXFP4 sparse-logits indexer once it merges upstream, replacing the dense fp8 indexer path. A live A/B test (TP2 c16, matched 900s window, #58208 reverted via #59125 so the dense fallback doesn't crash) measured #58671 giving +14.7/+14.9% p50/p90 interactivity and -9.5/-10.8% p50/p90 e2e latency over the dense path, with throughput/GPU unchanged. Draft until #58671 merges and lands in a nightly image; kept separate from #3555 so that PR can run now and this one can re-sweep immediately once #58671 is available. Drops c128 from both TP2 and TP4 conc-lists for this recipe."
9135+
- "基于 #3555 之上。重新添加 --attention-config '{\"indexer_kv_dtype\":\"mxfp4\",\"indexer_sparse_logits\":true}' 与 --block-size 128,以便在 vllm-project/vllm#58671 合并上游后启用其 ROCm 分页 MXFP4 稀疏 logits indexer,取代稠密 fp8 indexer 路径。一次真实的 A/B 测试(TP2、并发 16,匹配 900 秒窗口,并通过 #59125 回滚 #58208 以避免稠密回退路径崩溃)测得 #58671 相较稠密路径带来 interactivity p50/p90 提升 14.7%/14.9%,e2e 延迟 p50/p90 降低 9.5%/10.8%,而 throughput/GPU 无变化。本 PR 保持 draft 状态,直到 #58671 合并并随 nightly 镜像发布;与 #3555 分开提交,以便该 PR 可先行运行,本 PR 待 #58671 可用后立即重新扫描。此配方的 TP2 与 TP4 conc-list 均移除 c128。"
91369136
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3571

0 commit comments

Comments
 (0)