Repository navigation
perf(amd): bump DSv4.1-Flash ATOM AgentX to 1012 nightly and drop five points / DSv4.1-Flash ATOM AgentX 更新至 1012 nightly 并去掉五个点 - #3928
Conversation
…r points Move dsv41flash-fp4-mi355x-atom-agentic-dspark to rocm/atom-dev:nightly_202610121516 (ATOM a0527c9db) and drop TP2 c5/c128 and TP4 c4/c6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yhl-amd
added
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Looks good, straightforward config change. Verified: image/digest/ATOM-revision bump is consistent across agentic.yaml (model.container and AToMesh frontend container_image) and amd-master.yaml's image field; the remaining override_tp2_c*/override_tp4_c* blocks in the recipe exactly match the new conc-list entries in amd-master.yaml (TP2: 1,2,4,6,8,16,32,64; TP4: 1,5,8,32); the perf-changelog.yaml entry is appended at the tail, English-only, and accurately describes the image bump and dropped concurrency points.
Extended reasoning...
Three-file config/recipe change: an image tag/digest bump for the DSv4.1-Flash ATOM MI355X agentic recipe plus removal of four concurrency sweep points (TP2 c5/c128, TP4 c4/c6), mirrored between the srt-slurm recipe YAML and the master config, with a new append-only perf-changelog entry. No code paths, auth, or data-handling logic touched, only declarative benchmark config; cross-checking confirmed the recipe's remaining override blocks and the master config's conc-lists agree exactly, which is the main correctness risk in this kind of change.
approve
This review covers commit c9fc3b3, which is no longer the latest commit on this pull request; later commits are not covered by it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38276304505 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38276304505 |
|
/reuse-sweep-run 38276304505 |
… the diff - Update the image from rocm/atom-dev:nightly_202610100929 to nightly_202610121516 (digest sha256:a60811a8467b43e3e50e9e9eb8cec34f68b7bca03f8c6dbd6a0ff813dde49d4c; ATOM 0.1.8.dev29+ga0527c9db, ROCm 7.2.4) and update model.container and the AToMesh frontend image to match. - Drop TP2 concurrency 5 and 128 and TP4 concurrency 4, 5 and 6. TP2 runs concurrency 1, 2, 4, 6, 8, 16, 32 and 64 and TP4 runs 1, 8 and 32: 11 points instead of 16. - Server settings are unchanged. Upstream recipes/DeepSeek-V4.1-Flash-Agentic.md is identical between 73defd9b8 and a0527c9db. - ROCm/ATOM#2545 reworks decoder SWA bounded replay to follow the V4.1 report: the forward that finishes a prompt runs the encoder (layers 0-19) and layer 20's compression on every row, layers 20-38 on the last 128 rows with their windows truncated there, and layer 39 on the last row only. - ROCm/ATOM#2545 also turns on encoder SWA bounded replay by default: a prefix hit rebuilds the encoder's sliding-window state by recomputing one KV page below the hit instead of reading a state checkpoint, and no state checkpoints are kept. The DeepSeek-V4.1 KV page, which is also the prefix-cache hash block, drops from 256 to 128 tokens. - ATOM's mono decode path (on by default) now takes any decode step of up to 48 rows, i.e. concurrency 1-8 with five DSpark drafts, not only DSpark verify steps. Steps of 12 or more rows that are a multiple of TP run its sublayers token-parallel, with reduce-scatters instead of all-reduces. The draft runner never runs token-parallel. - No data-type or precision change to the DSpark draft. The checkpoint's bundled mtp.0-2 stages are stored like the target's layers: FP8 E4M3 weights with 32x32 UE8M0 block scales for the attention projections, shared experts and main_proj; MXFP4 routed experts; BF16 norms, router gate, Markov head and confidence head; F32 attention sinks and hc mixers. ATOM builds the draft with the target's native_quant_config() and make_v4_quant_config(), with no online quantization in this recipe, so it runs as shipped: FP8 activations on the FP8 GEMMs and the target's BF16 KV cache. Between 73defd9b8 and a0527c9db only the rows the draft reads under bounded replay change. - 镜像从 rocm/atom-dev:nightly_202610100929 更新为 nightly_202610121516（digest sha256:a60811a8467b43e3e50e9e9eb8cec34f68b7bca03f8c6dbd6a0ff813dde49d4c；ATOM 0.1.8.dev29+ga0527c9db，ROCm 7.2.4），model.container 和 AToMesh frontend 镜像同步更新。 - 去掉 TP2 的并发 5 和 128，以及 TP4 的并发 4、5 和 6。TP2 跑并发 1、2、4、6、8、16、32 和 64，TP4 跑 1、8 和 32：共 11 个点，原来是 16 个。 - server 参数不变。上游 recipes/DeepSeek-V4.1-Flash-Agentic.md 在 73defd9b8 和 a0527c9db 之间没有改动。 - ROCm/ATOM#2545 按 V4.1 报告重写了 decoder SWA bounded replay：完成 prompt 的那次 forward 对所有行运行 encoder（第 0-19 层）和第 20 层的压缩，第 20-38 层只算最后 128 行并在此截断窗口，第 39 层只算最后一行。 - ROCm/ATOM#2545 还默认开启 encoder SWA bounded replay：前缀命中时，从命中点下方重算一个 KV page 来重建 encoder 的滑动窗口状态，不再读取 state checkpoint，也不再保留 state checkpoint。DeepSeek-V4.1 的 KV page（也是前缀缓存的 hash block）从 256 降为 128 个 token。 - ATOM 的 mono decode 路径（默认开启）现在接管任何不超过 48 行的 decode step（五个 DSpark draft 时对应并发 1-8），不再只限于 DSpark verify step。行数不少于 12 且为 TP 倍数的 step 以 token 并行运行子层，用 reduce-scatter 代替 all-reduce。draft runner 从不使用 token 并行。 - DSpark draft 的数据类型和精度不变。checkpoint 自带的 mtp.0-2 与 target 层的存储方式相同：attention 投影、shared experts 和 main_proj 为 FP8 E4M3 权重加 32x32 UE8M0 block scale；routed experts 为 MXFP4；norm、router gate、Markov head 和 confidence head 为 BF16；attention sink 和 hc mixer 为 F32。ATOM 用与 target 相同的 native_quant_config() 和 make_v4_quant_config() 构建 draft，本 recipe 没有 online quantization，因此 draft 按原样运行：FP8 GEMM 使用 FP8 激活，KV cache 与 target 相同为 BF16。在 73defd9b8 和 a0527c9db 之间，只有 bounded replay 下 draft 读取的行发生变化。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seungrokj
changed the title
|
Draft data types: no change. The DSpark draft (
At ATOM 中文Draft 数据类型：不变。 DSpark draft（
在 ATOM |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- insert any additional info here
- recipe https://github.com/ROCm/ATOM/blob/main/recipes/DeepSeek-V4.1-Flash-Agentic.md
- Draft data types: no change. The DSpark draft (
mtp.0-2indeepseek-ai/DeepSeek-V4.1-Flash) is stored exactly like the target's layer 39 (read from the safetensors headers):
| Part | Data type |
|---|---|
Attention projections, shared experts, main_proj |
FP8 E4M3 with 32×32 UE8M0 block scales |
| Routed experts | MXFP4 |
| Norms, router gate, Markov head, confidence head | BF16 |
| Attention sinks, hc mixers | F32 |
At ATOM a0527c9db (the image's revision), the draft is built with the target's own quantization setup: native_quant_config() and make_v4_quant_config(). The recipe sets no online quantization, so the draft runs as shipped: FP8 activations on the FP8 GEMMs and the target's BF16 KV cache. The only draft change in 73defd9b8..a0527c9db is which rows it reads under bounded replay.
Signed: seungrokj
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep on in-PR commit): PASS — run 38276304505 ran on in-PR commit ✅ Check 2 (Evals): PASS — ➖ Check 3 (Upstream recipe): N/A — The only affected entry is ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest template): PASS — Every item in the current ✅ Check 6 (Upstream images / engine-first): PASS — This is an ATOM entry, so the vLLM/SGLang image rule does not apply. Engine-first ordering is met by the existing ✅ Check 7 (Deprecated models): PASS — In ✅ Check 8 (No arch hacks): PASS — There are no ✅ Check 9 (Spec-decode chat template): PASS — The AgentX client ( ✅ Check 10 (No engine patches): PASS — The diff changes only the image tags/digest, removes override points and adds a changelog entry. The effective recipe has no patches, setup scripts or wheel installs. ✅ Check 11 (Agentic golden AL): PASS — The job log shows ➖ Check 12 (Append-only): N/A — The new changelog entry has no ✅ Check 13 (Draft as shipped): PASS — The draft is the bundled DSpark head ( ✅ Check 14 (Pareto coverage): PASS — One curve: dsv41flash / agentic-coding / MI355X ATOM / fp4, using ✅ Check 15 (PR description): PASS — The PR body's claims match the pinned code and evidence: image and digest, TP2 Assessed commit: |
|
@Oseltamivir can you plz take a look at this ? |
Oseltamivir
left a comment
Oseltamivir
left a comment
There was a problem hiding this comment.
lgtm, image SWA bounded replay + mono decode
3 participants
Moves
dsv41flash-fp4-mi355x-atom-agentic-dsparkto the 1012 ATOM nightly and drops five points.rocm/atom-dev:nightly_202610100929→rocm/atom-dev:nightly_202610121516(digestsha256:a60811a8467b43e3e50e9e9eb8cec34f68b7bca03f8c6dbd6a0ff813dde49d4c; ATOM0.1.8.dev29+ga0527c9db, ROCm 7.2.4), in the master config,model.containerand the AToMesh frontend image.[1, 2, 4, 6, 8, 16, 32, 64]; TP4 drops c4, c5 and c6 →[1, 8, 32]. 11 points instead of 16.recipes/DeepSeek-V4.1-Flash-Agentic.mdis identical between73defd9b8anda0527c9db.mtp.0-2) runs as shipped: FP8 E4M3 attention, shared experts andmain_projwith 32x32 UE8M0 scales, MXFP4 routed experts, BF16 norms, gate, Markov and confidence heads. ATOM builds it with the target'snative_quant_config()andmake_v4_quant_config(), and the recipe sets no online quantization.中文
把
dsv41flash-fp4-mi355x-atom-agentic-dspark换成 1012 的 ATOM nightly，并去掉五个测试点。rocm/atom-dev:nightly_202610100929→rocm/atom-dev:nightly_202610121516（digestsha256:a60811a8467b43e3e50e9e9eb8cec34f68b7bca03f8c6dbd6a0ff813dde49d4c；ATOM0.1.8.dev29+ga0527c9db，ROCm 7.2.4）。master config、model.container和 AToMesh frontend 镜像一并更新。[1, 2, 4, 6, 8, 16, 32, 64]；TP4 去掉 c4、c5 和 c6，剩[1, 8, 32]。从 16 个点减到 11 个。recipes/DeepSeek-V4.1-Flash-Agentic.md在73defd9b8和a0527c9db之间没有改动。mtp.0-2）按原样运行：attention、shared experts 和main_proj为 FP8 E4M3（32x32 UE8M0 scale），routed experts 为 MXFP4，norm、gate、Markov head 和 confidence head 为 BF16。ATOM 用与 target 相同的native_quant_config()和make_v4_quant_config()构建它，recipe 没有设置 online quantization。AI model disclosure
claude-opus-5-5): made the config, recipe and changelog changes and prepared this PR.claude-opus-5-5): aligned the perf-changelog block and this description with the diff, and checked the draft data types against the checkpoint and ATOMa0527c9db.🤖 Generated with Claude Code