Repository navigation
[AMD][MI355X] Kimi-K3 vLLM AgentX nightly 81198e97 with dcp8 c1-c48 sweep / [AMD][MI355X] Kimi-K3 vLLM AgentX 更新镜像至 nightly 81198e97 并改为 dcp8 c1-c48 扫描 - #3833
Conversation
…d c44-c50 Move the mid band to dcp8 (c10/c12/c14/c16) and the throughput band to c44/c48/c50 with a 4-token DSpark draft, dropping the unused c8 and c70 variants. CUDA graphs capture every even size up to 2x CONC x (1 + drafts). Add stream-interval 10 and VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm to the base recipe, and FP8 per-channel online quantization of linears to c1. 将中段改为 dcp8（c10/c12/c14/c16），吞吐段改为 c44/c48/c50 并使用 4 token DSpark 草稿，移除不再使用的 c8 与 c70 变体。CUDA graph 捕获不超过 2x CONC x (1 + drafts) 的所有偶数尺寸。基础配方新增 stream-interval 10 与 VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm，c1 新增线性层 FP8 per-channel 在线量化。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd dcp8 c48 Comment out the full band lists in amd-master.yaml and run one point per band: c1 (dcp1, GPU-resident), c10 and c48 (dcp8, DRAM offload). 在 amd-master.yaml 中注释掉完整的分段列表，每段只运行一个点：c1（dcp1， GPU 常驻）、c10 与 c48（dcp8，DRAM offload）。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add override_tp8_c1_dcp8, the c1 variant with decode-context-parallel-size 8 over a2a, and narrow the amd-master.yaml sweep to that single c1 dcp8 point. 新增 override_tp8_c1_dcp8：在 c1 变体基础上使用 a2a 的 decode-context-parallel-size 8，并将 amd-master.yaml 扫描缩小为该 c1 dcp8 单点。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-enable the dcp8 DRAM-offload c48 point and disable c1 dcp8 to reproduce the c48 NCCL watchdog hang. 仅运行 Kimi-K3 MI355X vLLM dcp8 c48：重新启用 dcp8 DRAM 卸载 c48 配置点， 禁用 c1 dcp8，以复现 c48 的 NCCL watchdog 挂起问题。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1-c48 Move kimik3-fp4-mi355x-vllm-agentic-mtp and its recipe to the nightly-rocm100-81198e97 image and sweep dcp1 c1, dcp8 c1 (GPU-resident) and dcp8 DRAM-offload c4, c10-c16, c44, c48. Add override_tp8_c4_dcp8_simple and raise the c14/c16 DSpark draft from 3 to 4 tokens, with graph capture sized to 2 x CONC x (1 + drafts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
seungrokj
added
full-sweep-enabled
Set VLLM_USE_BREAKABLE_CUDAGRAPH to 0 in override_tp8_c1, override_tp8_c1_dcp8, override_tp8_c4 and override_tp8_c4_dcp8_simple. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline findings, I also checked whether the throughput band's switch to even-only cudagraph_capture_sizes plus the new max_cudagraph_capture_size drops coverage for long-context prefill shapes in override_tp8_c44_dcp8_simple — the capture list still spans up to 2 * CONC * (1 + num_speculative_tokens) as described in the PR, so this isn't a gap.
Extended reasoning...
The diff re-partitions concurrency bands, speculative-decoding configs, and cudagraph sizing for the Kimi-K3 MI355X vLLM AgentX recipe and its master-config entry; it touches no auth/crypto/injection surface, but three issues were confirmed (an exp-name selection ambiguity from the new dcp8 c1 variant, a missing perf-changelog.yaml entry despite performance-affecting changes, and a stale band-boundary comment), which are enough to withhold approval. I additionally verified the throughput-band cudagraph capture-size concern raised as a candidate and found it does not hold given the stated sizing formula.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
inferencex-e2e/configs/amd-master.yaml— This change bumps the vLLM image pin, re-partitions concurrency bands to dcp8, and rewrites cudagraph/speculative-config across 9 recipe variants, but adds no inferencex-e2e/perf-changelog.yaml entry. AGENTS.md's "Non-negotiable benchmark invariants" (around line 125) requires an append-only perf-changelog.yaml entry for every change that can affect benchmark performance or any recipe addition/modification; this PR changes the image, concurrency points, DSpark draft lengths and cudagraph capture sizes without one, so downstream consumers of the changelog lose the record of why these numbers moved. Fix: add an append-only perf-changelog.yaml entry documenting the image bump and the band re-partition before merging. [also at: inferencex-e2e/perf-changelog.yaml:1 - This PR changes the image pin, speculative-decoding drafts, cudagraph capture sizes and concurrency sweep for kimik3-fp4-mi355x-vllm-agentic-mtp but adds no perf-changelog.yaml entry.]Why this was flagged
The diff changes the pinned image (36768d1...->81198e97... at amd-master.yaml:729 and agentic.yaml:14), the search-space conc-lists (amd-master.yaml:736-739), and per-variant speculative-config/cudagraph_capture_sizes throughout agentic.yaml. AGENTS.md:125 states 'Every change that can affect benchmark performance and every recipe addition or modification requires a new inferencex-e2e/perf-changelog.yaml entry.' No file named perf-changelog.yaml appears in
git diff --statfor this change (only the two listed files are touched). On base, every such change carries a changelog entry; here the record of this performance-relevant change is absent, breaking the invariant the repo relies on to track benchmark number provenance.Verification: normal. The diff touches only two files (
git diff --stat:agentic.yamlandconfigs/amd-master.yaml) and does not modifyinferencex-e2e/perf-changelog.yaml, which exists in the repo as a tracked append-only file.
| - { tp: 8, ep: 1, dcp-size: 1, kv-offloading: none, conc-list: [1], spec-decoding: mtp, srt-recipe: benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml } | ||
| - { tp: 8, ep: 1, dcp-size: 8, kv-offloading: none, conc-list: [1], spec-decoding: mtp, srt-recipe: benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml } |
There was a problem hiding this comment.
🔴 Adding the new dcp8 c1 row makes --exp-names selection for kimik3-fp4-mi355x-vllm-agentic-mtp ambiguous where it worked on the base branch. _agentic_entries in inferencex-e2e/infx/matrix/generate.py builds exp-name as {model}_tp{tp}_conc{conc}_{kv_offload_suffix}_spec-{spec_decoding} (lines ~1176-1192) without dcp-size, so the new row at amd-master.yaml:737 (dcp-size 8, kv-offloading none, conc-list [1]) and the existing row at amd-master.yaml:736 (dcp-size 1, same conc/offload/spec) both produce kimik3_tp8_conc1_kvnone_spec-mtp. Fix: include dcp-size (or another topology-distinguishing field) in the agentic exp-name format so two rows that differ only by dcp-size are no longer the same string.
Why this was flagged
Trigger: python -m infx.matrix generate test-config --config-keys kimik3-fp4-mi355x-vllm-agentic-mtp --exp-names kimik3_tp8_conc1_kvnone_spec-mtp, the standard way to select one generated point for a qualification/rerun sweep. _agentic_entries (generate.py ~1176-1192) computes exp_name from model_code, tp, conc, kv_offload suffix and spec_decoding only; dcp_size is not part of it. filter_exp_names (generate.py:1552-1570) then finds the name matches both the dcp-size:1 row (amd-master.yaml:736) and the new dcp-size:8 row (amd-master.yaml:737) and raises "Experiment name(s) matched multiple rows", which is new: on the base branch only one row had conc=1/kv-offloading=none, so the same name resolved unambiguously. Any existing script, doc, or CI step referencing this exp-name for a single point now fails. Result storage (RESULT_FILENAME_BASE in .github/workflows/benchmark-tmpl.yml:288) still disambiguates via a separate dcp${DCP_SIZE} token, so this is a name-based selection regression, not a data clobber.
Verification: This PR introduces a genuine exp-name collision that makes --exp-names selection of the conc1/kvnone point ambiguous where it was unique on the base branch. generate.py:1188-1192 builds the agentic exp-name from model_code, tp, conc, kv-offload suffix, and spec_decoding only; dcp_size is never encoded.
| # mid (8-14) dcp 1, DSpark draft, SimpleCPUOffload DRAM tier. | ||
| # throughput (44-70) dcp 8 shards the KV read across all 8 GPUs, DRAM | ||
| # offload, mtp with synthetic acceptance (no draft). | ||
| # mid (10-16) dcp 8, DSpark draft, SimpleCPUOffload DRAM tier. |
There was a problem hiding this comment.
🟡 (optional) The rewritten band-boundary comment for kimik3-fp4-mi355x-vllm-agentic-mtp is now inaccurate, which can mislead anyone reading the config to pick wrong concurrency points (nit). It still says 'interactive (1, 4)' though conc 4 was dropped from the interactive band, and 'mid (10-16)' though the actual mid conc-list is [4, 10, 12, 14, 16] (amd-master.yaml:736) — conc 4 moved into mid, not interactive. Fix: update the comment to 'interactive (1)' and 'mid (4, 10-16)' to match the search-space rows, consistent with the companion agentic.yaml:4-6 comment which was updated correctly.
Why this was flagged
amd-master.yaml:719 reads '# interactive (1, 4) GPU-resident, dcp 1, DSpark draft.' and amd-master.yaml:720 reads '# mid (10-16) dcp 8, DSpark draft, SimpleCPUOffload DRAM tier.' The actual search-space at amd-master.yaml:734-736 has interactive conc-list [1] (dcp1) and [1] (dcp8), with conc 4 now in the dram-offload mid band's conc-list [4, 10, 12, 14, 16]. The sibling recipe file's own header comment (agentic.yaml:4-6) was updated correctly to 'interactive dcp1/dcp8 c1 ... mid dcp8 c4-c16', showing the master-config comment was simply missed during the rewrite.
Verification: nit. Documentation-only inaccuracy, nothing breaks at runtime. amd-master.yaml:718 reads # interactive (1, 4) GPU-resident, dcp 1, DSpark draft. but the interactive search-space rows (736-737) now carry conc-list [1] only (one dcp1, one dcp8) — conc 4 is removed and the new dcp8 c1 row is undocumented. amd-master.yaml:719 reads # mid (10-16) dcp 8... but the mid dram band at line 738 is conc-list: [4, 10, 12, 14, 16], so conc 4 actually sits in mid, and "10-16" omits it.
Switch c10/c12/c14/c16 from FULL to FULL_AND_PIECEWISE so all overrides request the same cudagraph mode, refresh the recipe comment, and add the perf-changelog entry for #3833. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=37809542821 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=37809542821 |
Does this PR change any dtypes in the DSpark draft model? No.I checked the vLLM source at the image commit The only dtype change in this PR is the new Why that setting doesn't reach the draft:
|
|
/reuse-sweep-run 37809542821 |
Correct the kimik3-fp4-mi355x-vllm-agentic-mtp perf-changelog entry: - Every point requests FULL_AND_PIECEWISE (the mid band was FULL) with breakable CUDA graphs off; without a torch.compile submodule vLLM overrides it to FULL_DECODE_ONLY, as the c1 and c14 server logs show. - Golden (synthetic) acceptance applies at every draft point in agentic runs, not only the throughput band; note the c14 draft length 3 -> 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1-c48 - Update the image from vllm/vllm-openai-rocm:nightly-rocm100-36768d1bfd39094681cdbc8cb37d4b31c0729c89 to vllm/vllm-openai-rocm:nightly-rocm100-81198e97ba7eee2a22540caaa756b7fdddcb4d93. - Sweep dcp1 concurrency 1 and dcp8 concurrency 1 GPU-resident, and dcp8 concurrency 4, 10, 12, 14, 16, 44 and 48 with the SimpleCPUOffloadConnector DRAM tier (224.875 GB/rank). This replaces dcp1 concurrency 1/4, dcp1 DRAM-offload concurrency 8-14 and dcp8 concurrency 44/48/70. The dcp8 points use the a2a DCP backend. - DSpark draft length per point: 7 at concurrency 1, 5 at 4, 4 at 10, 3 at 12, and 4 at 14, 16, 44 and 48 (concurrency 14 was 3). The throughput band (44, 48) previously had no speculative-config and now runs the 4-token DSpark draft. As at every draft point, agentic runs replace block rejection with the golden acceptance length for that draft length. - Admission is 2x concurrency. CUDA graphs capture every even size up to 2x concurrency x (1 + draft tokens), replacing the previous every-size lists (up to 8192 in the throughput band). - Every point requests cudagraph_mode FULL_AND_PIECEWISE (the mid band was FULL) with breakable CUDA graphs disabled (VLLM_USE_BREAKABLE_CUDAGRAPH=0, previously 1 at concurrency 1 and 4). The AMD Kimi-K3 model has no torch.compile submodule, so vLLM overrides the mode to FULL_DECODE_ONLY at load, as the concurrency 1 and 14 server logs show. - Add stream-interval 10 and VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm, which routes DCP multi-token verify through the AITER ASM round-robin decode, to every point. - Concurrency 1 (dcp1 and dcp8) adds quantization-config linear fp8_per_channel: vLLM quantizes the target's linear layers to FP8 W8A8 at load, except kv_b_proj, in_proj_qkvgfab, f_b_proj, fused_qkv_a_proj, self_attn.g_proj and layer 0 mlp.down_proj, which stay BF16. The routed experts stay MXFP4. - No change to the Inferact/Kimi-K3-DSpark draft's precision: its model config takes the speculative config's own quantization (unset) and never receives quantization-config, and get_draft_quant_config() returns None for K3DSparkModel, so its weights stay BF16 and its KV cache stays fp8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
lgtm @seungrokj feel free to merge it after the vllm recipes merge
Kimi-K3 accuracy evals on MI355X (c=1 / c=2 / c=4)Image Server config. The server ran tp8, dcp1, fp8 KV, no KV offload and no
|
|
@functionstackx for the record, merged this PR before the recipe as the recipe is not necessary to reproduce this perf. I commented to the recipe vllm-project/recipes#1093 (comment) and also here: https://github.com/vllm-project/vllm/pull/59965/changes#diff-b7538eee1327212c7cfbe3f7874600dfcd41b5fb5662695cd5f396294610ddf9 |
2 participants
Summary
kimik3-fp4-mi355x-vllm-agentic-mtp(configs/amd-master.yaml) and its recipe (benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml) fromvllm/vllm-openai-rocm:nightly-rocm100-36768d1bfd39094681cdbc8cb37d4b31c0729c89tovllm/vllm-openai-rocm:nightly-rocm100-81198e97ba7eee2a22540caaa756b7fdddcb4d93. Follows [AMD][MI355X] Kimi-K3 vLLM AgentX FP8 per-channel linear quantization-config / [AMD][MI355X] Kimi-K3 vLLM AgentX 新增 FP8 per-channel linear quantization-config #3627.override_tp8_c1_dcp8(new): dcp8 a2a, GPU-resident, DSpark k=7, FP8 linear quantization, graphs 2-16.override_tp8_c4_dcp8_simple(new): as c4, plus dcp8 a2a and the DRAM tier (224.875 GB/rank); DSpark k=5, graphs 2-48.override_tp8_c10/c12/c14/c16_dcp8_simple: the former dcp1 mid band moved to dcp8.override_tp8_c44/c48/c50_dcp8_simple: DSpark k=4 (previously no draft); c70 is renamed c50.override_tp8_c14_dcp8_simpleandoverride_tp8_c16_dcp8_simpleuse DSpark k=4 (was 3).max-num-seqsis 2 x CONC.cudagraph_modeFULL_AND_PIECEWISE (c10-c14 were FULL) withVLLM_USE_BREAKABLE_CUDAGRAPH=0(c1/c4 were 1). The AMD K3 model has no torch.compile submodule, so vLLM overrides the mode to FULL_DECODE_ONLY at load (seen in the c1 and c14 server logs).stream-interval: 10andVLLM_ROCM_AITER_MLA_DCP_VERIFY=asm(DCP multi-token verify through the AITER ASM round-robin decode) to every point.quantization-configlinearfp8_per_channel: the target's linear layers are quantized to FP8 W8A8 at load, except kv_b_proj, in_proj_qkvgfab, f_b_proj, fused_qkv_a_proj, self_attn.g_proj and layer-0 mlp.down_proj (BF16); routed experts stay MXFP4. The Inferact/Kimi-K3-DSpark draft's precision is unchanged (BF16 weights, fp8 KV).override_tp8_c1override_tp8_c1_dcp8override_tp8_c4_dcp8_simpleoverride_tp8_c10_dcp8_simpleoverride_tp8_c12_dcp8_simpleoverride_tp8_c14_dcp8_simpleoverride_tp8_c16_dcp8_simpleoverride_tp8_c44_dcp8_simpleoverride_tp8_c48_dcp8_simpleoverride_tp8_c4(dcp1) andoverride_tp8_c50_dcp8_simplestay in the recipe but are not swept.Validation
infx.srt_slurm.single_node.validate_recipe(image, CONC, KV_OFFLOADING, DCP_SIZE, TOTAL_CPU_DRAM_GB 1799).max_cudagraph_capture_sizeequals 2 x CONC x (1 + num_speculative_tokens), and its capture list covers every even size up to it.perf-changelog.yamlhas akimik3-fp4-mi355x-vllm-agentic-mtpentry for this PR, appended at the end of the file.中文
概要
kimik3-fp4-mi355x-vllm-agentic-mtp（configs/amd-master.yaml）及其 recipe 的镜像由nightly-rocm100-36768d1…更新为nightly-rocm100-81198e97…。承接 [AMD][MI355X] Kimi-K3 vLLM AgentX FP8 per-channel linear quantization-config / [AMD][MI355X] Kimi-K3 vLLM AgentX 新增 FP8 per-channel linear quantization-config #3627。override_tp8_c4_dcp8_simple（沿用 c4 配置，加 dcp8 a2a 与 DRAM 卸载，DSpark k=5）；c14/c16 的 DSpark 草稿长度由 3 改为 4。max-num-seqs为 2 x CONC。stream-interval: 10与VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm。quantization-configlinearfp8_per_channel，目标模型线性层加载时量化为 FP8 W8A8（部分层保持 BF16），routed experts 仍为 MXFP4；DSpark 草稿模型精度不变。验证
validate_recipe各自恰好匹配一个 recipe 变体。max_cudagraph_capture_size均等于 2 x CONC x (1 + num_speculative_tokens)。perf-changelog.yaml末尾新增kimik3-fp4-mi355x-vllm-agentic-mtp条目。AI model disclosure
Prepared with Claude Opus 5.5 (
claude-opus-5-5) via Claude Code, the sole contributing model (recipe variants, PR text, validation).🤖 Generated with Claude Code