Repository navigation
Add GB300 DeepSeek-V4.1-Flash Dynamo + vLLM disaggregated AgentX recipes / 新增 GB300 DeepSeek-V4.1-Flash Dynamo + vLLM 分离式 AgentX 配方 - #3941
Conversation
Add 8P1D, 6P1D and 5P1D points (DEP2 prefill, DEP16 decode) behind the Dynamo KV router with native vLLM sidecars. 新增 GB300 DeepSeek-V4.1-Flash Dynamo + vLLM 分离式 AgentX 配方：8P1D、6P1D、5P1D 三个点（DEP2 prefill、DEP16 decode），使用 Dynamo KV 路由与原生 vLLM sidecar。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
| - deepseek_v41 | ||
| environment: | ||
| PYTHONHASHSEED: "0" | ||
| setup_script: vllm-container-deps.sh |
There was a problem hiding this comment.
can u delete this? what is the need for this
isnt the numa conflict fix already commited ages ago?
https://github.com/NVIDIA/srt-slurm/blob/main/configs/patches/vllm-container-deps.sh
There was a problem hiding this comment.
Thanks for the comments! I've addressed them
…gle-node arm on the new image Drop the vllm-container-deps.sh setup script (the image already has numactl and msgpack), enable FlashInfer autotune on prefill, move the GB300 single-node arm to vllm/vllm-openai:nightly-aarch64-dev-20261012-59e75f4 with its TP4 points at conc 1 and 8, and restore a historical changelog pr-link that the previous commit changed by mistake. 处理评审意见：删除 vllm-container-deps.sh 安装脚本（镜像已自带 numactl 和 msgpack），prefill 开启 FlashInfer autotune；GB300 单节点配置改用 vllm/vllm-openai:nightly-aarch64-dev-20261012-59e75f4，TP4 只跑 conc 1 和 8；并恢复上一个提交误改的一条历史 changelog pr-link。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… vLLM arm unchanged Add dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg (one TP4 worker with a native Dynamo sidecar, conc 1 and 8) and restore the GB300 single-node vLLM key and recipe to main. 将 TP4 点移到新的 Dynamo 聚合配置项，单节点 vLLM 配置保持不变：新增 dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg（单个 TP4 worker + 原生 Dynamo sidecar，conc 1 和 8），并将 GB300 单节点 vLLM 配置项和配方恢复为 main 的版本。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ten DEP2 prefill instances feed the DEP16 decode at max-num-seqs 132. 在 Dynamo 分离式配置项中新增 10P1D 点（conc 6656）：10 个 DEP2 prefill 实例配 DEP16 decode，max-num-seqs 132。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Opened #3950 against this branch with the Dynamo frontend/runtime settings we used on the GB300 Dynamo + vLLM P/D line: conditional disaggregation off, router temperature/queue threshold pinned, tcp request plane + zmq event plane (no NATS), vLLM Two things I did not change but would check before the sweep: (1) decode Also: the 10P1D override is c6656 while the shared table says c7168; and could you add TTFT p50/p90 next to each point? With 18 to 132 decode slots per rank most users queue in front of decode, so TTFT is where the cap shows up. |
Although I tried c7168 locally, it seemed a bit workload overwhelming, I think c6656 performed at least as well. So I switched to c6656 in my commit |
…ntX recipes (#3950) * Dynamo frontend settings for the GB300 DSV4.1-Flash Dynamo + vLLM AgentX recipes Carry the Dynamo runtime and router settings that held up on the GB300 Dynamo + vLLM P/D frontier (c1 to c640, full-hour AgentX replays) into the disaggregated and aggregated recipes: - Conditional disaggregation off: active-decode-blocks-threshold, active-prefill-tokens-threshold and active-prefill-tokens-threshold-frac set to None, so every request is prefilled on a prefill rank and never on one of the 16 lockstep MegaMoE decode ranks. - Router defaults pinned: DYN_ROUTER_TEMPERATURE=0 and DYN_ROUTER_QUEUE_THRESHOLD None (temperature 1 measured as noise, +1 to 4%). - Dynamo request plane tcp and event plane zmq; the NATS service is dropped (no broker hop; the frontier ran this way at c640). - Disaggregated recipe: etcd on a dedicated infra node and the frontend placed beside it, so the single Rust frontend does not share a node with two DEP2 prefill ranks running Engram CPU offload and 70 GiB host KV each. Adds one CPU-only node to the job; the GPU count of the measurement is unchanged. - vLLM engines run with --reasoning-parser None on every role: the deepseek_v41 tool-call and reasoning parsers are applied once, by the Dynamo frontend (published through the sidecar --dyn-*-parser flags), not a second time inside vllm-rs. - DYN_LOG=warn for measured runs. Validated with srtctl dry-run (5P1D c2560 and TP4 c1) on oci-jhb. * Keep the Dynamo frontend on the head node; etcd stays on infra srtctl's orchestrator accepts only head|first_decode|last_decode for frontend.placement.node (src/srtctl/core/topology.py), even though the schema and `srtctl dry-run` accept `infra`; a 10P1D job with the previous commit's placement failed at launch with "frontend.placement.node='infra' is invalid". Drop the frontend placement and the dedicated infra node for etcd; the frontend runs on the head node as before. Verified: the same 10P1D job launches with this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Juntian777
added
full-sweep-fail-fast
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it's a large multi-node benchmark recipe addition with many interdependent parallelism/offloading settings, and the touched paths (nvidia-master.yaml, gb300-* recipes) are CODEOWNER-protected, a human look is still worthwhile.
What was reviewed:
- New agg (TP4, conc 1/8) and disagg (DEP2 prefill / DEP16 decode, 10P1D-5P1D) vLLM recipes for dsv41flash on GB300-FP4, their nvidia-master.yaml entries, and the perf-changelog append.
- Image/container consistency between recipe
container/identity.container.imageand master-configimage- matches in both new entries. - Disagg topology node math (prefill nodes x 2 workers/node, decode 16 GPUs over 4 nodes) against
gpus_per_node: 4- consistent. - The disagg recipe's speculative-config omitting
rejection_sample_method(unlike agg's explicit"block") - already investigated and ruled out as a discrepancy by the bug hunting system.
Extended reasoning...
Pure benchmark-config addition (no source/doc changes): two new srt-slurm recipe override files and matching nvidia-master.yaml entries plus a perf-changelog append for a GB300/FP4 DeepSeek-V4.1-Flash Dynamo+vLLM AgentX family. No auth/crypto/permissions surface is touched; the risk is purely in dense numeric correctness (parallelism degrees, cudagraph capture sizes, KV-offload ratios, node counts) that is easy to get subtly wrong and hard to fully verify without running the pending sweep. All four changed files fall under CODEOWNERS (nvidia-master.yaml and gb300-* recipes explicitly, perf-changelog.yaml via the default rule), and the PR itself is marked draft with sweep/evals pending and the sweep-label/maintainer-reuse checklist items still unchecked, so human review is expected before merge regardless.
This review covers commit 5deca12, which is no longer the latest commit on this pull request; later commits are not covered by it.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38351782338 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38351782338 |
xinli-sw
added
full-sweep-fail-fast
…auto-detect With cpus-per-task 72 every worker step was pinned to CPUs 0-71 (NUMA 0), including the DEP2 prefill worker on GPUs 2-3, and vLLM refuses automatic GPU-to-NUMA detection under a constrained affinity, so the prefill engines exited. With 144 each step (one task, --overlap) gets CPUs 0-143 and vLLM binds each DEP2 prefill worker to its GPUs' NUMA node (GPUs 0-1 -> node 0, GPUs 2-3 -> node 1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1955dcc to
8ce65c4
Compare
Juntian777
added
full-sweep-enabled
Picks up NVIDIA/srt-slurm#602: on the Python sidecar launcher (multi-node hybrid-DP decode) srtctl drops `reasoning-parser: none`, which Python vLLM rejects as an unknown parser name, while single-node `vllm-rs serve` endpoints keep it as Dynamo sidecars require. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Juntian777
removed
the
full-sweep-enabled
Juntian777
added
the
full-sweep-fail-fast
This branch has not been deployed
4 participants
Summary
Add GB300 DeepSeek-V4.1-Flash AgentX on Dynamo + vLLM:
dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg(TP4, conc 1 and 8) anddsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg(10P1D, 8P1D, 6P1D, 5P1D; DEP2 prefill, DEP16 decode). Imagevllm/vllm-openai:nightly-aarch64-dev-20261012-59e75f4, Dynamo1.6.0.dev20261010with native vLLM sidecars and the KV router.Draft: full sweep and evals pending.
AI model disclosure
claude-opus-5-5) in Claude Code, including one delegated research agent on the same model.PR checklist and change type
Type of Change
Checklist
inferencex-e2e/perf-changelog.yamland have not edited historical entriesfull-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
新增 GB300 上 DeepSeek-V4.1-Flash 的 Dynamo + vLLM AgentX 配方：
dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg（TP4，conc 1 和 8）与dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-disagg（10P1D、8P1D、6P1D、5P1D；DEP2 prefill，DEP16 decode）。镜像vllm/vllm-openai:nightly-aarch64-dev-20261012-59e75f4（vLLMfe4df3df），Dynamo1.6.0.dev20261010，使用原生 vLLM sidecar 与 KV 路由。草稿：完整 sweep 与评测待运行。
AI 模型使用说明：Claude Code 中的 Claude Opus 5.5（
claude-opus-5-5），含一个同模型的委派调研 agent；负责起草配方、master 配置项、changelog 条目和 PR 文本，并调研已有 AgentX P/D 提交的惯例；作者审阅并验证改动。改动类型：新功能。🤖 Generated with Claude Code