Repository navigation
[AMD][MI355X] DSv4.1 Flash: enable the vLLM mono decode kernels / [AMD][MI355X] DSv4.1 Flash：启用 vLLM mono decode 内核（镜像待定） - #3690
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
填写 perf-changelog 中 #3690 的 PR 链接。 Co-authored-by: Cursor <cursoragent@cursor.com>
Fangzhou-Ai
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Native DeepSeek-V4.1 a4w4 support from vllm-project/vllm#58819. | ||
| VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4: '1' |
There was a problem hiding this comment.
🔴 Setting VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1 (agentic.yaml:66) drops MoE activation precision from a8w4 to a4w4, and the recipe's own changelog says this reaches the "shared DSpark MoE layers" used by the draft (perf-changelog.yaml:9283). The old comment at line 42 confirms the prior baseline used CK a8w4; this submission-added env var lowers activations to 4-bit for those same layers when DSpark verifies, which CONTRIBUTING.md's Draft-model precision rule forbids: "Online or offline quantization of draft ... activations ... below the shipped precision, whether through a flag, environment variable" (CONTRIBUTING.md:77-80), and the baseline for comparison is the same image with no draft-related settings from the submission (CONTRIBUTING.md:40-42). …
Why this was flagged
…Matching the #3571 GSM8K baseline does not exempt this per CONTRIBUTING.md's own text, since draft-precision changes shift accuracy loss to acceptance rate, which GSM8K does not measure. Fix: either confine the a4w4 opt-in so DSpark's shared MoE path stays at the shipped baseline precision (a8w4) or obtain an explicit CODEOWNER draft-precision waiver per CONTRIBUTING.md before enabling it process-wide.
Trigger: the role env sets VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1 (agentic.yaml:66) on a DSpark recipe where speculative-config uses method dspark (agentic.yaml:56) and DSpark verification reuses the target's MoE weights (changelog says 'shared DSpark MoE layers', perf-changelog.yaml:9283). Before this diff, moe-backend: aiter resolved to the CK a8w4 kernel (old comment, now replaced at agentic.yaml:42); after this diff the same layers run a4w4, i.e. activations drop from 8-bit to 4-bit for both target and draft tokens.
Verification: normal; acknowledged in diff. This is a genuine Draft-model precision policy concern that the base branch does not have.
Decisive facts from the diff itself:
- agentic.yaml:66 adds
VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4: '1'to the roleenvof a DSpark (speculative-decoding) recipe (speculative-config: '{"method":"dspark",...}', agentic.yaml:56). - The old comment at line 42 ("The plain name lets the CK a8w4 MoE kernel win vLLM's priority list") establishes the pre-change out-of-the-box default was CK a8w4 (8-bit activations).
There was a problem hiding this comment.
@chunfangamd can we get a waiver on that? Or I can simply remove the a4w4 flag here.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38028752236 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38028752236 |
|
Closing this PR as a4w4 doesn't show significant perf gain. and a lower precision of draft model requires extra waiver. |
1 similar comment
|
Closing this PR as a4w4 doesn't show significant perf gain. and a lower precision of draft model requires extra waiver. |
Fangzhou-Ai
removed
the
full-sweep-enabled
There was a problem hiding this comment.
Code review found no new issues
No new issues were found in this update; 1 finding from earlier reviews is still open above.
This review covers commit 144ab45, which is no longer the latest commit on this pull request; later commits are not covered by it.
perf-changelog.yaml is rebuilt as origin/main byte for byte, then the #3690 and #3735 entries are appended at the tail. This keeps the GLM-5.2 #3616 entry's config-keys header, which the #3690 merge commit had folded into the preceding pr-link scalar. perf-changelog.yaml 按字节重建为 origin/main 的内容，再在末尾追加 #3690 与 #3735 两条记录。这样可保留 GLM-5.2 #3616 条目的 config-keys 头，该头 曾被 #3690 的合并提交并入前一条的 pr-link 标量。 Co-authored-by: Cursor <cursoragent@cursor.com>
Replace --max-num-batched-tokens 16384 with 8192 on the thirteen TP2/TP4 points that still carried it; TP2 c64 was already 8192. The chunk is now uniform across the arm, matching the B300 arm and the B200 TP4 arm from concurrency 8 up. A prefill chunk scheduled into the same engine step as a decode batch stalls every decoding sequence for the length of that chunk, so the stall scales with the chunk. On the two TP4 ep1 arms, the p90-minus-p50 ITL spread is 10.81 ms at c64 on MI355X at 16384 against 1.74 ms on B200 at 8192. Re-append the #3690 changelog entry at the tail. Its merge commit had folded the config-keys header of the GLM-5.2 #3616 entry into the preceding pr-link scalar, which re-keyed #3616 and left #3690's own entry unparseable. 将仍为 16384 的十三个 TP2/TP4 数据点的 --max-num-batched-tokens 改为 8192； TP2 c64 原本已是 8192。该分块现在在整个臂上统一，与 B300 臂以及并发 8 及以上 的 B200 TP4 臂一致。 与 decode 批次调度进同一个引擎步的 prefill 分块，会让所有正在解码的序列停顿 该分块的时长，因此停顿时间随分块大小增长。在两个 TP4 ep1 臂上，c64 的 ITL p90 减 p50 差值在 16384 的 MI355X 上为 10.81 ms，而在 8192 的 B200 上为 1.74 ms。 同时在文件末尾重新追加 #3690 的 changelog 记录。其合并提交曾把 GLM-5.2 #3616 条目的 config-keys 头并入前一条的 pr-link 标量，导致 #3616 被错误归键， 且 #3690 自身的记录无法被解析。 Co-authored-by: Cursor <cursoragent@cursor.com>
Merges the updated #3690 branch, so this branch again differs from it by exactly one thing. Narrows that difference: --max-num-batched-tokens 8192 now applies only to override_tp4_c32, override_tp4_c64 and override_tp2_c32 (override_tp2_c64 was already 8192); concurrency 1-16 returns to 16384. The 14-point A/B in run 37375212446 against baseline run 37080504692 found gains only at c32 and above (TP2 c32 ITL p90 -14.4%, TP4 c64 -27.8%) and repeated TTFT regressions below it (TP4 c4 +29.5%, TP2 c4 +18.8%), matching the profiled chunked-prefill step counts. perf-changelog.yaml conflicted at its append-only tail; resolved to main's bytes verbatim plus the #3690 and #3735 entries. 合并更新后的 #3690 分支，使本分支与其重新只差一处，并收窄这处差异： --max-num-batched-tokens 8192 现在仅作用于 override_tp4_c32、 override_tp4_c64 与 override_tp2_c32（override_tp2_c64 本就是 8192）， 并发 1-16 回到 16384。运行 37375212446 对基线运行 37080504692 的 14 点 A/B 显示收益只出现在并发 32 及以上（TP2 c32 的 ITL p90 -14.4%，TP4 c64 -27.8%），低于该并发则 TTFT 反复恶化（TP4 c4 +29.5%，TP2 c4 +18.8%）， 与 profile 中分块 prefill 步的计数一致。 perf-changelog.yaml 在仅可追加的尾部产生冲突，解析为逐字节保留 main 的 内容并追加 #3690 与 #3735 两个条目。 Co-authored-by: Cursor <cursoragent@cursor.com>
Fangzhou-Ai
added
the
full-sweep-enabled
a4w4 dropped from this branch; the ladder now sweeps a8w4Run 37424787697 on this image failed all nine TP=2 points and passed all nine TP=4 points. Root cause, reproduced locally on MI355X and bisected with a TP2/TP4 A/B on one build:
DSv4.1-Flash shards TP=4 only escapes because its What changed here
Fix statusvllm-project/vllm#60273 rounds the a4w4 intermediate to 256. It works (TP=2 boots, GSM8K 0.9629 flexible-extract over all 1319 questions) but costs +11% MoE weights at TP=2, +33% at TP=4, and moves both rows off every tuned AITER row, so it is not the fix this recipe should wait for. The defect is AITER-side and the cheaper fix is validated locally: AI assistance was used for the root-cause investigation and this change. |
Fangzhou-Ai
changed the title
Repin the MI355X DeepSeek-V4.1-Flash DSpark recipe to nightly-rocm100-21d93d0d and retune the ladder around it. - Leave VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4 unset. The a4w4 opt-in faults during graph capture at TP=2; all nine TP=2 points failed and all nine TP=4 points passed in run 37424787697. vllm-project/vllm#60273 tracks the fix, so the experts stay on the generic AITER CK a8w4 selector. - Enable VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 so tensor-parallel all-reduces above the quick-reduce size threshold use the INT4 codec. - Hold both TP rows at concurrency 64. Run 37510565585 measured TP=2 c128 at 150,683 tok/s/chip against 151,896 at c64 with prompt tokens within 0.5%, so the extra concurrency bought real prefill rather than scored tokens. - Set VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD=1024. Upstream caps the shared-expert overlap at 256 tokens, which predates speculative decoding: five DSpark drafts make a decode step CONC x 6 tokens, so c64 submits 384 and the overlap never engages above c42. 1024 covers the ladder and matches the ATOM default, and stays below the prefill chunk so memory profiling never takes the overlap. - Restore the upstream 16384 prefill chunk at TP2 c64, making the chunk uniform across all 14 points. That point had capped it at 8192 to buy KV room, but the room was taken by a profiling bug rather than by the chunk: the ROCm paged MXFP4 indexer sizes its decode logits workspace as (rows, max-model-len), and with no ambient vLLM config in scope the row count fell back to the batched-token count, so at this recipe's 1M context the reservation scaled linearly with the chunk. vllm-project/vllm#58014 bounds rows by max-num-seqs x (1 + 5 drafts), and the repinned image carries that fix. Rebased onto main: the configuration-procedure guides this branch had edited were removed by #3789, so those edits are dropped. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
d0a98ff to
8a96da0
Compare
- Set VLLM_ROCM_MONO_DECODE=1 (vllm-project/vllm#60397): two persistent launches a layer for decode steps of up to 48 rows; larger steps take the regular path. - Image TBD until the first ROCm 10.0 nightly with #60397 is published. - Drop the a4w4 notes and shorten the comments and the changelog entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fangzhou-Ai
changed the title
Fangzhou-Ai
removed
the
full-sweep-enabled
Keep the upstream default for simplicity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the TBD placeholder with nightly-rocm100-7d0b4e57, the first ROCm 10.0 nightly built after vllm-project/vllm#60397 merged, so the VLLM_ROCM_MONO_DECODE=1 setting on this recipe has kernels to run. 将 TBD 占位符替换为 nightly-rocm100-7d0b4e57，这是 vllm-project/vllm#60397 合并后构建的首个 ROCm 10.0 nightly，使本 recipe 中的 VLLM_ROCM_MONO_DECODE=1 设置有对应的 kernel 可用。 Co-authored-by: Cursor <cursoragent@cursor.com>
Fangzhou-Ai
added
the
full-sweep-enabled
|
A notice on the failed check 13 #3841 |
|
recipe PR vllm-project/recipes#1094 |
|
/reuse-sweep-run 38028752236 |
- Update the image from nightly-rocm100-ac9126e58aa7bbab1856ba6593ba4d5003fea516 to nightly-rocm100-7d0b4e57aac4c4323225b0eee9ae960ce75ffffe, the first ROCm 10.0 nightly with the DeepSeek-V4.1 mono decode kernels of vllm-project/vllm#60397, and update model.container to match. The image also brings vllm-project/vllm#59685, #56638, #60159 and #60389. - Enable the mono decode kernels (VLLM_ROCM_MONO_DECODE=1) at TP2 and TP4: a decode step of up to 48 rows, i.e. concurrency 1-8 with five DSpark drafts, runs each backbone layer as two persistent launches (30 of 40 layers whole, the other 10 the FFN launch only). Larger steps take the regular path, so C16-C64 are expected unchanged. - Enable VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 (vllm-project/recipes#1061) for tensor-parallel all-reduces above the quick-reduce size threshold. - Restore --max-num-batched-tokens 16384 at TP2 c64: vllm-project/vllm#58014 removed the profiling over-reservation the 8192 cap worked around. The prefill chunk is now 16384 at every point and the ladder stays at concurrency 1-64. - No data-type or precision change to the DSpark draft: vllm-project/vllm#60397 skips the draft layers (layer id >= num_hidden_layers), so the bundled DSpark draft of deepseek-ai/DeepSeek-V4.1-Flash runs on vLLM's regular path as shipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- insert any additional info here
- recipe at https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash?hardware=mi355x&kv_offload=lmcache
- every draft tensor's runtime weight has the same dtype and scale granularity as in the HF checkpoint. Some activations are quantized at runtime, but that is the same on main's image ac9126e5, so this PR doesn't change it. The only change the PR makes for the draft is the INT4 compression of large all-reduces.
Per-tensor comparison (draft layers mtp.0-2, ROCm gfx950, vLLM 7d0b4e57)
┌────────────────────────────────┬─────────────┬───────────────────────────────┬─────────────────────────────────────────┬────────────────────────────────────────────┐
│ Draft tensor │ HF repo │ Runtime weight after this PR │ Runtime activation / compute │ Changed by this PR? │
│ │ dtype │ │ │ │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ attn.wq_a+wkv (fused), wq_b, │ FP8 E4M3, │ FP8 E4M3, 32×32 E8M0 (kept) │ BF16 input → MXFP8 1×32 → block-32 GEMM │ No │
│ wo_b, main_proj │ 32×32 E8M0 │ │ → BF16 │ │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ │ FP8 E4M3, │ FP8 E4M3, 32×32 E8M0 (kept; │ Attention output → MXFP8 1×32 → batched │ │
│ attn.wo_a │ 32×32 E8M0 │ the BF16-dequant path is │ GEMM → BF16 │ No │
│ │ │ off-gfx950 only) │ │ │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ ffn.shared_experts.w1/w3/w2 │ FP8 E4M3, │ FP8 E4M3, 32×32 E8M0 (kept) │ MXFP8 1×32 → BF16 (separate MLP, not │ No │
│ │ 32×32 E8M0 │ │ fused) │ │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ │ MXFP4 │ MXFP4, 1×32 E8M0, shuffled │ Backend AITER_MXFP4_BF16, which in │ │
│ ffn.experts.N.w1/w3/w2 (128 │ (packed │ for AITER (lossless); │ practice runs A8W4: both stages │ No │
│ experts, top-3) │ I8), 1×32 │ intermediate zero-padded │ quantize activations to MXFP8, FP32 │ │
│ │ E8M0 │ 576→640 at TP4 │ accumulation │ │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ ffn.gate.weight │ BF16 │ BF16 │ BF16×BF16 → FP32 logits, FP32 routing │ No │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ gate.bias, bias_vl, hc_*, │ FP32 │ FP32 │ FP32 │ No │
│ attn_sink │ │ │ │ │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ norms │ BF16 │ BF16 │ BF16 │ No │
│ (attn/ffn/q/kv/main/final) │ │ │ │ │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ markov_head.embed/head │ BF16 │ BF16, unquantized │ BF16 │ No │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ confidence_head.proj │ BF16 │ FP32 (upcast at load) │ FP32 │ No │
├────────────────────────────────┼─────────────┼───────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────┤
│ SWA KV cache │ (n/a) │ fp8_ds_mla: 448 FP8 + 64 BF16 │ (n/a) │ No │
│ │ │ RoPE + UE8M0 scales │ │ │
The draft layers have no compressor or indexer (compress_ratios[40:43] = 0), so the MXFP4 indexer setting doesn't reach them.
Signed: seungrokj
|
seungrokj
changed the title
Oseltamivir
left a comment
Oseltamivir
left a comment
There was a problem hiding this comment.
lgtm, Functionstackx's requested changes to recipe for rocm10 added
5 participants
Enables the DeepSeek-V4.1 mono decode kernels from vllm-project/vllm#60397 in the MI355X DeepSeek-V4.1-Flash vLLM DSpark AgentX recipe (
dsv41flash-fp4-mi355x-vllm-agentic-dspark), to measure their end-to-end gain on the full ladder.Changes
nightly-rocm100-ac9126e58aa7bbab1856ba6593ba4d5003fea516→nightly-rocm100-7d0b4e57aac4c4323225b0eee9ae960ce75ffffe, the first ROCm 10.0 nightly that contains vllm#60397 (also brings vllm#59685, #56638, #60159 and #60389).model.containermatches.VLLM_ROCM_MONO_DECODE=1. The kernels run a decode step of up to 48 rows as two persistent launches a layer (30 of 40 backbone layers whole, the other 10 the FFN launch only), at TP2 and TP4. With five DSpark drafts that is concurrency 1-8; larger steps take the regular path, so c16-c64 should be unchanged.VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4(vllm-project/recipes#1061);layer_id >= num_hidden_layersinMonoDecodeLayer.create), so the bundled DSpark draft runs on vLLM's regular path as shipped.For reviewers
中文
在 MI355X DeepSeek-V4.1-Flash vLLM DSpark AgentX 配置（
dsv41flash-fp4-mi355x-vllm-agentic-dspark）中启用 vllm-project/vllm#60397 的 DeepSeek-V4.1 mono decode 内核，以在完整并发梯度上测量其端到端收益。改动
nightly-rocm100-ac9126e58aa7bbab1856ba6593ba4d5003fea516→nightly-rocm100-7d0b4e57aac4c4323225b0eee9ae960ce75ffffe，即第一个包含 vllm#60397 的 ROCm 10.0 nightly（同时带入 vllm#59685、#56638、#60159 和 #60389）。model.container同步更新。VLLM_ROCM_MONO_DECODE=1：在 TP2 和 TP4 下，每层用两个常驻内核处理不超过 48 行的 decode 步（40 个主干层中 30 层整层走 mono，其余 10 层仅 FFN 走 mono）。DSpark 有 5 个草稿 token，对应并发 1-8；更大的步走原有路径，因此 c16-c64 预期不变。VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4（vllm-project/recipes#1061）；MonoDecodeLayer.create中layer_id >= num_hidden_layers），模型自带的 DSpark 草稿按原样走 vLLM 原有路径。评审注意
AI model disclosure
Prepared with Claude Opus 5.5 (
claude-opus-5-5) in Claude Code: the recipe, master-config and changelog edits, the merge withmain, and this description. Earlier revisions of this PR were prepared with Claude Opus 5.5 through a Cursor agent.🤖 Generated with Claude Code