[AMD][MI355X] Kimi-K3 vLLM AgentX nightly fe4df3df with no-draft c92/c96 / [AMD][MI355X] Kimi-K3 vLLM AgentX 更新镜像至 nightly fe4df3df 并新增无草稿 c92/c96

[seungrokj]

Summary

• Move kimik3-fp4-mi355x-vllm-agentic-mtp (configs/amd-master.yaml) and its recipe (benchmarks/single_node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml) from vllm/vllm-openai-rocm:nightly-rocm100-81198e97ba7eee2a22540caaa756b7fdddcb4d93 to vllm/vllm-openai-rocm:nightly-rocm100-fe4df3df0eb703862c686b256e9fe44fa94c8945. Follows [AMD][MI355X] Kimi-K3 vLLM AgentX nightly 81198e97 with dcp8 c1-c48 sweep / [AMD][MI355X] Kimi-K3 vLLM AgentX 更新镜像至 nightly 81198e97 并改为 dcp8 c1-c48 扫描 #3833.
• Concurrency sweep:
  • dcp1, no KV offload: c1.
  • dcp8, no KV offload: c1.
  • dcp8, SimpleCPUOffload DRAM tier (224.875 GB/rank): c4, c10, c14, c16 (c12 dropped).
  • dcp8, SimpleCPUOffload DRAM tier: c48 (c44 dropped), plus c92 and c96 (new).
• override_tp8_c1 and override_tp8_c1_dcp8: max-num-seqs 2 → 1.
• New variants override_tp8_c92_dcp8_simple and override_tp8_c96_dcp8_simple:
  • No speculative-config (no DSpark draft); the points keep the mtp matrix label through SPEC_DECODING: mtp.
  • dcp8 a2a with the SimpleCPUOffloadConnector kv-transfer-config (224.875 GB/rank from the 1799 GB budget).
  • max-num-batched-tokens 24576 and no-async-scheduling, following the c96 baseline run in ajith-sirra-amd/InferenceMAX_Rocm_Team run 38262669991.
  • max-num-seqs 2 x CONC (184, 192); FULL_AND_PIECEWISE graphs at every even size up to 2 x CONC.
• The new image brings the Kimi-K3 ROCm changes [ROCm][Perf] Kimi-K3 Store only the current rank's shards in latent MoE up-proj  vllm-project/vllm#59591, #59069, #60717, #60446, #59523 and #59880.
• Draft data types: no change. K3DSparkModel (vllm/models/kimi_k3/nvidia/dspark_mla.py) and get_draft_quant_config() (vllm/model_executor/models/utils.py) have no commits between 81198e97 and fe4df3df, so the Inferact/Kimi-K3-DSpark draft keeps BF16 weights and an fp8 KV cache where it runs. c92 and c96 load no draft.

Variant	CONC	dcp	KV offload	DSpark k	max-num-seqs	max-num-batched-tokens
override_tp8_c1	1	1	none	7	1	16384
override_tp8_c1_dcp8	1	8	none	7	1	16384
override_tp8_c4_dcp8_simple	4	8	dram	5	8	8192
override_tp8_c10_dcp8_simple	10	8	dram	4	20	8192
override_tp8_c14_dcp8_simple	14	8	dram	4	28	8192
override_tp8_c16_dcp8_simple	16	8	dram	4	32	8192
override_tp8_c48_dcp8_simple	48	8	dram	4	96	8192
override_tp8_c92_dcp8_simple	92	8	dram	none	184	24576
override_tp8_c96_dcp8_simple	96	8	dram	none	192	24576

override_tp8_c4, override_tp8_c12_dcp8_simple, override_tp8_c44_dcp8_simple and override_tp8_c50_dcp8_simple stay in the recipe but are not swept.

Validation

• Each of the 9 matrix points matches exactly one recipe variant under infx.srt_slurm.single_node.validate_recipe (image fe4df3df, CONC, KV_OFFLOADING, DCP_SIZE, TOTAL_CPU_DRAM_GB 1799).
• perf-changelog.yaml has one kimik3-fp4-mi355x-vllm-agentic-mtp block for this PR, appended at the end of the file.
• e2e run 38336004618 (dcp1 c1, max-num-seqs 1, previous image 81198e97, no evals): 105.6 output tok/s, median TTFT 0.83 s, median TPOT 5.3 ms; 214 of 226 requests succeeded, 12 returned empty responses. Not yet run on fe4df3df or at c92/c96.

中文

概要

• 将 kimik3-fp4-mi355x-vllm-agentic-mtp（configs/amd-master.yaml）及其 recipe 的镜像由 nightly-rocm100-81198e97… 更新为 nightly-rocm100-fe4df3df…。承接 [AMD][MI355X] Kimi-K3 vLLM AgentX nightly 81198e97 with dcp8 c1-c48 sweep / [AMD][MI355X] Kimi-K3 vLLM AgentX 更新镜像至 nightly 81198e97 并改为 dcp8 c1-c48 扫描 #3833。
• 并发扫描：dcp1 无 KV 卸载 c1；dcp8 无 KV 卸载 c1；dcp8 + SimpleCPUOffload DRAM 卸载（每 rank 224.875 GB）c4、c10、c14、c16（去掉 c12）；dcp8 + DRAM 卸载 c48（去掉 c44），以及新增 c92 与 c96。
• override_tp8_c1 与 override_tp8_c1_dcp8：max-num-seqs 由 2 改为 1。
• 新增 override_tp8_c92_dcp8_simple 与 override_tp8_c96_dcp8_simple：不设 speculative-config（无 DSpark 草稿模型），通过 SPEC_DECODING: mtp 保留 mtp 标签；dcp8 a2a 加 SimpleCPUOffloadConnector DRAM 卸载；max-num-batched-tokens 24576 与 no-async-scheduling 参照 ajith-sirra-amd/InferenceMAX_Rocm_Team run 38262669991 的 c96 基线；max-num-seqs 为 2 x CONC（184、192），FULL_AND_PIECEWISE CUDA graph 覆盖至 2 x CONC 的所有偶数。
• 新镜像包含 Kimi-K3 ROCm 相关改动 [ROCm][Perf] Kimi-K3 Store only the current rank's shards in latent MoE up-proj  vllm-project/vllm#59591、#59069、#60717、#60446、#59523 与 #59880。
• 草稿模型数据类型：不变。 两个镜像之间 K3DSparkModel 与 get_draft_quant_config() 均无改动，Inferact/Kimi-K3-DSpark 草稿保持 BF16 权重与 fp8 KV cache；c92 与 c96 不加载草稿模型。

验证

• 9 个 matrix 点经 validate_recipe 各自恰好匹配一个 recipe 变体。
• 本 PR 在 perf-changelog.yaml 末尾新增一个 kimik3-fp4-mi355x-vllm-agentic-mtp 条目。
• e2e run 38336004618（dcp1 c1、max-num-seqs 1、旧镜像 81198e97、无 eval）：输出 105.6 tok/s，TTFT 中位数 0.83 s，TPOT 中位数 5.3 ms；226 个请求中 214 个成功，12 个返回空响应。尚未在 fe4df3df 或 c92/c96 上运行。

AI model disclosure

Prepared with Claude Opus 5.5 (claude-opus-5-5) via Claude Code, the sole contributing model (recipe variants, changelog, PR text, validation).

🤖 Generated with Claude Code

[claude]

Looks good, straightforward config change. Reviewed the image-tag bump (mirrored identically between the srt-slurm recipe and amd-master.yaml), the max-num-seqs 2→1 change for the two GPU-resident c1 bands, the two new no-draft high-concurrency override blocks (c92/c96) with their max-num-seqs/CUDA-graph-size math and SimpleCPUOffloadConnector config consistent with the existing dram-tier variants, and the single append-only perf-changelog.yaml block added at the tail matching this PR.

Extended reasoning...

Three-file diff: a vLLM-ROCm nightly image bump mirrored in both the recipe and master config, a max-num-seqs tightening on two existing bands, two new mechanically-patterned no-draft override blocks following the file's established style, and matching conc-list/perf-changelog updates. No security-sensitive surface (auth, crypto, permissions, injection) is touched. The new blocks' numeric fields (2x-concurrency max-num-seqs, CUDA-graph capture sizes, DRAM-tier byte budget) are internally consistent with each other and with sibling overrides, and the changelog entry is a single append-only block at the physical tail as required by repo convention.

This review covers commit 58ac1ff, which is no longer the latest commit on this pull request; later commits are not covered by it.

[functionstackx]

why go from 16 conc to 48 conc?

not an hard rule but generally can u fill in the gaps plz

[functionstackx]

big stretch of emptyness...

[functionstackx]

looking at the other skus here, they dont have big stretch of emptyness verus with this mi355 vllm u go from global conc 16 (conc 2/gpu) straight to conc 48 (conc 6 / gpu)