Add B300 Dynamo+SGLang AgentX configs for DeepSeek V4 with NIXL KV transfer / 添加使用 NIXL KV 传输的 DeepSeek V4 B300 Dynamo+SGLang AgentX 配置 by nvpohanh · Pull Request #3817 · SemiAnalysisAI/InferenceX

nvpohanh · 2026-10-08T02:00:41Z

[by Claude Code]

Summary

Adds DeepSeek-V4-Pro-0813 FP4 B300 Dynamo+SGLang AgentX recipes. The recipes come from #3631, with disaggregated KV transfer over NIXL instead of Mooncake, and now run on lmsysorg/sglang:nightly-dev-20261013-7224fce1.

Points: aggregate TP8 c1, TP4 c4, TP4 c8 (HiCache) and DEP8 c384 (HiCache); disaggregated 1P1D DEP8 c64/c240 and 1P2D DEP8 c64 (HiCache prefill). Topology, concurrency, cache sizes and memory fractions are unchanged.
Image: nightly-dev-20261013-7224fce1 replaces nightly-dev-20260916-c9a8fba9.
Attention DP: DEP8 workers set attn-dp-size: 8 in place of dp-size: 8 + enable-dp-attention: true, which [Feature] Add --attn-dp-size and deprecate --enable-dp-attention sgl-project/sglang#41818 deprecated.
Dynamo: the recipes install Dynamo from commit b208989fbe, which is also the router version in the master config.
- The image's Dynamo 1.6.0.dev20261010 predates fix(sglang): count attention-DP ranks from attn_dp_size ai-dynamo/dynamo#16005. In this SGLang, both DP spellings resolve to dp_size=1, attn_dp_size=8. That Dynamo reads only dp_size and enable_dp_attention, so it would register one DP rank per DEP8 worker and subscribe only to rank 0's KV events. The KV router would then send every request to DP rank 0.
- b208989fbe is the first commit after #16005 whose dependency pins match the image (aisimulate==0.13.0.dev2). srt-slurm builds it at job start and caches the build on /configs, as for qwen3.5-fp8-b300-dynamo-sglang-agentic-disagg on the same cluster.
Disaggregated KV transfer: NIXL with the LIBFABRIC backend (SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC). The image's NIXL 1.5.0 ships the plugin, so there is no setup script, waiver or package change.
Image behavior to note: this SGLang fuses the shared expert into the DeepGEMM Mega-MoE kernel by default (fuse shared experts with routed experts in MegaMoE's DeepGEMM sgl-project/sglang#39313), so DEP8 throughput may differ from earlier runs of these recipes for that reason too.
No recipe enables the W4A4 MXFP4 Mega-MoE path. Prefill and decode use the same DSpark K=6 method.
On b300-dsxe, Dynamo+SGLang serves DeepSeek-V4-Pro-0813 from the node-local NVMe copy (DeepSeek-V4-Pro-0813@scratch), as vLLM already does.

Settings removed or renamed for the 1013 image

Checked against the SGLang source at 7224fce1 and the Dynamo source at b208989fbe. Topology declarations (tp-size, ep-size, and dp-size: 1 on TP points) are kept.

Setting	Roles	Why
`SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT`, `SGLANG_OPT_USE_JIT_NORM`, `SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE`, `SGLANG_DSV4_MHC_PREWARM`	various	The image no longer reads them. `moe-a2a-backend: megamoe` selects Mega-MoE, and MHC prewarm always runs.
`SGLANG_OPT_DEEPGEMM_MEGA_MOE`, `NCCL_TIMEOUT`	disagg	Never defined: the first is a misspelling, and the second is not an NCCL variable.
`SGLANG_DEFAULT_THINKING`, `SGLANG_DSV4_REASONING_EFFORT`, `SGLANG_REQUEST_STATE_WAIT_TIMEOUT`	all, disagg	Only SGLang's own HTTP server reads them. Behind Dynamo, the frontend applies the chat template with thinking and reasoning effort taken from the request, and the worker calls the engine directly.
`SGLANG_RAGGED_VERIFY_MODE=static`, `SGLANG_JIT_DEEPGEMM_PRECOMPILE=1`, `SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1`, `SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=1`, `SGLANG_OPT_USE_ONLINE_COMPRESS=0`, `SGLANG_OPT_USE_JIT_INDEXER_METADATA=1`, `SGLANG_OPT_USE_TOPK_V2=True`	various	Defaults on B300.
`attention-backend: dsv4`, `page-size: 256`	various	Set by SGLang's DeepSeek-V4 model override.
`speculative-num-steps: 1`, `speculative-eagle-topk: 1`, `speculative-num-draft-tokens: 7`	all	Derived by DSpark from `speculative-dspark-block-size: 6`, which also selects the golden acceptance length.
`swa-full-tokens-ratio: 0.1`, `hicache-write-policy: write_through`	TP agg	Defaults for this model.
`cuda-graph-max-bs-prefill`	disagg prefill	SGLang disables the prefill CUDA graph for DeepSeek-V4.
`disable-shared-experts-fusion`	agg DEP8	DeepSeek-V4 only fuses shared experts into the routed weights when enforced. Mega-MoE's in-kernel fusion is controlled by `SGLANG_OPT_DEEPGEMM_MEGA_MOE_FUSE_SHARED_EXPERTS`, not by this flag.
`incremental-streaming-output`, `enable-cache-report`	agg DEP8, all	Dynamo always turns incremental streaming on. The cache report only affects SGLang's OpenAI server.
Frontend `active-decode-blocks-threshold`, `active-prefill-tokens-threshold`, `active-prefill-tokens-threshold-frac` set to `"None"`	frontend	Dynamo's router rejection thresholds are off by default.
`SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW` → `SGLANG_OPT_RELEASE_PREFILL_SWA`	disagg	Renamed in the image. The new name defaults to off, so the setting is kept under the new name.

Recipe details (carried over from #3631)

Aggregate DEP8 c384

Ported from the single-node SGLang DEP8 c384 point (dsv4-fp4-b300-sglang-agentic-hicache-mtp, benchmarks/single_node/srt-slurm-recipes/dsv4/sglang/b300-fp4-mtp/agentic.yaml, override_dep8_c384):

Engine settings kept: DP8 attention + EP8 Mega-MoE, FP4 indexer, DP LM head, local control broadcast, prefill delayer with prefill-decode-interval: 20, stream-interval: 20, chunked-prefill-size: 65536 with SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320, max-running-requests: 768 (96 per DP rank, twice concurrency), mem-fraction-static: 0.88, swa-full-tokens-ratio: 0.075, HiCache ratio 3 (write_back, direct, page_first_direct), DSpark K=6.
Decode graph limit: cuda-graph-max-bs-decode: 96, the per-rank request cap (768 / 8).
Routing: the Dynamo KV router replaces the SGLang router's DP-aware consistent hashing. Sessions stay sticky through X-Dynamo-Session-ID, and each DP rank publishes KV events (kv-events-config). The frontend sets DYN_TCP_REQUEST_TIMEOUT: "60", as the disaggregated recipes do, so the c384 warmup burst does not exceed Dynamo's 5 s request-plane ack timeout.
Dropped: settings that only apply to SGLang's own HTTP server and have no effect behind the Dynamo frontend: tokenizer-worker-num, tool-call and reasoning parsers, chat template, skip-server-warmup, SGLANG_TIMEOUT_KEEP_ALIVE.

Gap-filling points: agg TP4 c8 and disagg 1P2D DEP8 c64

Agg TP4 c8: the TP4 c4 recipe with max-running-requests: 16 and cuda-graph-max-bs-decode: 16, plus a HiCache DRAM tier (hicache-ratio: 2.75, write_through, direct, page_first_direct), because TP4 c4 already fills about 60% of the GPU KV pool.
Disagg 1P2D DEP8 c64: the 1P1D c64 recipe with a second decode worker (3 nodes), which halves each decode rank's load.

NIXL instead of Mooncake

On AWS EFA, Mooncake needs the separate mooncake-transfer-engine-efa-cuda13 wheel in place of the image's Mooncake package, which #3631 installed during container setup (review). NIXL packages its own libfabric backend, so these recipes drop that setup script, its waiver and the Mooncake-only settings (MOONCAKE_PROTOCOL, MC_MS_FILTERS, WITH_NVIDIA_PEERMEM). The NIXL LIBFABRIC settings come from #3597.

AI model disclosure

Claude Opus 5.5 (claude-opus-5-5, via Claude Code), including two delegated read-only sub-agents on the same model:
- This update: merged current main; audited every recipe flag and env var against SGLang 7224fce1 and the bundled and pinned Dynamo sources; moved the recipes to nightly-dev-20261013-7224fce1; switched DEP8 workers to attn-dp-size; pinned Dynamo; removed or renamed settings; updated the changelog entry and this description.
- Earlier: rebased the Add B300 Dynamo+SGLang AgentX configs for DeepSeek V4 with aggregate DEP8 c384 / 添加含聚合式 DEP8 c384 的 DeepSeek V4 B300 Dynamo+SGLang AgentX 配置 #3631 commits onto main, switched the disaggregated recipes from Mooncake to NIXL, and removed the Mooncake setup script and waiver.
Commits carried over from Add B300 Dynamo+SGLang AgentX configs for DeepSeek V4 with aggregate DEP8 c384 / 添加含聚合式 DEP8 c384 的 DeepSeek V4 B300 Dynamo+SGLang AgentX 配置 #3631:
- GPT-5 (Codex, including one delegated agent): the original submission and the Mooncake configuration.
- GPT-6 (Codex): diagnosed and prepared earlier runtime repairs.
- Claude Opus 5.5: the DeepGEMM fast-warmup and node-local checkpoint fixes, the aggregate DEP8 c384 recipe, the TP4 c8 and 1P2D c64 points, and the Mooncake waiver that this PR removes.

PR checklist and change type

Type of Change

Checklist

I have completed the AI model disclosure and kept it current
I have tested my changes locally
I have updated documentation if necessary
For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of inferencex-e2e/perf-changelog.yaml and have not edited historical entries
If this PR can affect benchmark performance or adds or modifies a recipe, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.

中文

摘要

新增 DeepSeek-V4-Pro-0813 FP4 B300 Dynamo+SGLang AgentX 配方。配方沿用 #3631，分离式 KV 传输由 Mooncake 改为 NIXL，现运行于 lmsysorg/sglang:nightly-dev-20261013-7224fce1。

**测试点：**聚合式 TP8 c1、TP4 c4、TP4 c8（HiCache）与 DEP8 c384（HiCache）；分离式 1P1D DEP8 c64/c240 与 1P2D DEP8 c64（prefill 使用 HiCache）。拓扑、并发、缓存大小与显存比例均不变。
**镜像：**以 nightly-dev-20261013-7224fce1 取代 nightly-dev-20260916-c9a8fba9。
**注意力 DP：**DEP8 worker 改用 attn-dp-size: 8，取代已被 [Feature] Add --attn-dp-size and deprecate --enable-dp-attention sgl-project/sglang#41818 弃用的 dp-size: 8 + enable-dp-attention: true。
**Dynamo：**配方从提交 b208989fbe 安装 Dynamo，主配置中的 router 版本也记为该提交。
- 镜像自带的 Dynamo 1.6.0.dev20261010 早于 fix(sglang): count attention-DP ranks from attn_dp_size ai-dynamo/dynamo#16005。在该版本 SGLang 中，两种 DP 写法都会解析为 dp_size=1, attn_dp_size=8。而该 Dynamo 只读取 dp_size 与 enable_dp_attention，因此每个 DEP8 worker 只注册一个 DP rank，并且只订阅 rank 0 的 KV 事件，KV 路由器会把所有请求发往 DP rank 0。
- b208989fbe 是 #16005 之后第一个依赖版本约束与镜像一致（aisimulate==0.13.0.dev2）的提交。srt-slurm 在作业启动时构建它，并把构建结果缓存在 /configs，与同一集群上的 qwen3.5-fp8-b300-dynamo-sglang-agentic-disagg 做法相同。
**分离式 KV 传输：**使用 LIBFABRIC 后端的 NIXL（SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC）。镜像中的 NIXL 1.5.0 已自带该插件，因此无需初始化脚本、豁免文件或软件包改动。
**需注意的镜像行为：**该版本 SGLang 默认把共享专家融合进 DeepGEMM Mega-MoE kernel（fuse shared experts with routed experts in MegaMoE's DeepGEMM sgl-project/sglang#39313），因此 DEP8 吞吐与这些配方早先的结果相比，也可能因此有所不同。
所有配方均不启用 W4A4 MXFP4 Mega-MoE 路径。Prefill 与 decode 使用相同的 DSpark K=6 方法。
在 b300-dsxe 上，Dynamo+SGLang 从节点本地 NVMe 副本（DeepSeek-V4-Pro-0813@scratch）加载 DeepSeek-V4-Pro-0813，与 vLLM 现有做法相同。

针对 1013 镜像删除或重命名的设置

均已对照 SGLang 7224fce1 与 Dynamo b208989fbe 源码核实。拓扑声明（tp-size、ep-size 以及 TP 测试点上的 dp-size: 1）予以保留。

镜像已不再读取：SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT、SGLANG_OPT_USE_JIT_NORM、SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE、SGLANG_DSV4_MHC_PREWARM。Mega-MoE 由 moe-a2a-backend: megamoe 选择，MHC 预热始终执行。
从未定义：SGLANG_OPT_DEEPGEMM_MEGA_MOE（拼写错误）与 NCCL_TIMEOUT（并非 NCCL 变量）。
仅 SGLang 自身 HTTP 服务端读取：SGLANG_DEFAULT_THINKING、SGLANG_DSV4_REASONING_EFFORT、SGLANG_REQUEST_STATE_WAIT_TIMEOUT。在 Dynamo 之后，前端按请求中的思考模式与推理强度套用聊天模板，worker 直接调用引擎。
B300 上的默认值：SGLANG_RAGGED_VERIFY_MODE=static、SGLANG_JIT_DEEPGEMM_PRECOMPILE=1、SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1、SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=1、SGLANG_OPT_USE_ONLINE_COMPRESS=0、SGLANG_OPT_USE_JIT_INDEXER_METADATA=1、SGLANG_OPT_USE_TOPK_V2=True。
由 SGLang 的 DeepSeek-V4 模型覆盖设置：attention-backend: dsv4、page-size: 256。
由 DSpark 根据 speculative-dspark-block-size: 6 推导：speculative-num-steps: 1、speculative-eagle-topk: 1、speculative-num-draft-tokens: 7。golden 接受长度同样由 block size 选择。
该模型的默认值：TP 聚合配方中的 swa-full-tokens-ratio: 0.1 与 hicache-write-policy: write_through。
SGLang 对 DeepSeek-V4 关闭 prefill CUDA graph：cuda-graph-max-bs-prefill。
disable-shared-experts-fusion：DeepSeek-V4 仅在强制时才把共享专家融合进路由专家权重；Mega-MoE 的 kernel 内融合由 SGLANG_OPT_DEEPGEMM_MEGA_MOE_FUSE_SHARED_EXPERTS 控制，而非此参数。
incremental-streaming-output 由 Dynamo 强制开启；enable-cache-report 只影响 SGLang 的 OpenAI 服务端。
前端设为 "None" 的 active-decode-blocks-threshold、active-prefill-tokens-threshold、active-prefill-tokens-threshold-frac：Dynamo 路由拒绝阈值默认关闭。
SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW 在镜像中重命名为 SGLANG_OPT_RELEASE_PREFILL_SWA。新名称默认关闭，因此以新名称保留该设置。

配方细节（沿用自 #3631）

聚合式 DEP8 c384：移植自单节点 SGLang DEP8 c384 测试点（dsv4-fp4-b300-sglang-agentic-hicache-mtp，benchmarks/single_node/srt-slurm-recipes/dsv4/sglang/b300-fp4-mtp/agentic.yaml，override_dep8_c384）。

**保留的引擎设置：**DP8 注意力 + EP8 Mega-MoE、FP4 indexer、DP LM head、local control broadcast、prefill-decode-interval: 20 的 prefill delayer、stream-interval: 20、chunked-prefill-size: 65536 配合 SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320、max-running-requests: 768（每个 DP rank 96，为并发数两倍）、mem-fraction-static: 0.88、swa-full-tokens-ratio: 0.075、HiCache 比例 3（write_back、direct、page_first_direct）、DSpark K=6。
Decode CUDA graph 上限：cuda-graph-max-bs-decode: 96，即每个 rank 的请求上限（768 / 8）。
**路由：**以 Dynamo KV 路由器取代 SGLang 路由器的 DP 感知一致性哈希。会话通过 X-Dynamo-Session-ID 保持亲和，各 DP rank 发布 KV 事件（kv-events-config）。前端与分离式配方一样设置 DYN_TCP_REQUEST_TIMEOUT: "60"，避免 c384 预热突发请求超过 Dynamo 请求平面默认 5 秒的确认超时。
**移除：**仅适用于 SGLang 自身 HTTP 服务端、在 Dynamo 前端之后不起作用的设置：tokenizer-worker-num、工具调用与推理解析器、聊天模板、skip-server-warmup、SGLANG_TIMEOUT_KEEP_ALIVE。

补充测试点：

**聚合式 TP4 c8：**基于 TP4 c4 配方，max-running-requests: 16、cuda-graph-max-bs-decode: 16，并增加 HiCache DRAM 层（hicache-ratio: 2.75、write_through、direct、page_first_direct），因为 TP4 c4 已占用约 60% 的 GPU KV 池。
**分离式 1P2D DEP8 c64：**即 1P1D c64 配方增加第二个 decode worker（共 3 个节点），使每个 decode rank 的负载减半。

**以 NIXL 取代 Mooncake：**在 AWS EFA 上，Mooncake 需要用单独的 mooncake-transfer-engine-efa-cuda13 wheel 替换镜像自带的 Mooncake 包，#3631 在容器初始化时完成这一替换（review）。NIXL 自带 libfabric 后端，因此这些配方移除了该初始化脚本、其豁免文件以及仅供 Mooncake 使用的设置（MOONCAKE_PROTOCOL、MC_MS_FILTERS、WITH_NVIDIA_PEERMEM）。NIXL LIBFABRIC 设置来自 #3597。

AI 模型披露

Claude Opus 5.5（claude-opus-5-5，通过 Claude Code），包括两个使用同一模型的只读委派子代理：
- **本次更新：**合并最新 main；对照 SGLang 7224fce1 以及镜像自带与固定版本的 Dynamo 源码审查所有配方参数与环境变量；将配方迁移到 nightly-dev-20261013-7224fce1；DEP8 worker 改用 attn-dp-size；固定 Dynamo 版本；删除或重命名相关设置；更新变更日志条目与本说明。
- **此前：**将 Add B300 Dynamo+SGLang AgentX configs for DeepSeek V4 with aggregate DEP8 c384 / 添加含聚合式 DEP8 c384 的 DeepSeek V4 B300 Dynamo+SGLang AgentX 配置 #3631 的提交 rebase 到 main，将分离式配方由 Mooncake 切换为 NIXL，并移除 Mooncake 初始化脚本与豁免文件。
沿用自 Add B300 Dynamo+SGLang AgentX configs for DeepSeek V4 with aggregate DEP8 c384 / 添加含聚合式 DEP8 c384 的 DeepSeek V4 B300 Dynamo+SGLang AgentX 配置 #3631 的提交：
- GPT-5（Codex，包括一个委派代理）：最初提交与 Mooncake 配置。
- GPT-6（Codex）：诊断并准备了前序运行时修复。
- Claude Opus 5.5：DeepGEMM 快速预热与节点本地权重路径修复、聚合式 DEP8 c384 配方、TP4 c8 与 1P2D c64 测试点，以及本 PR 移除的 Mooncake 豁免文件。

🤖 Generated with Claude Code

将 B300 Dynamo+SGLang AgentX 变更日志条目的 pr-link 指向 #3817。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

新增基于 Mooncake 的 B300 DeepSeek V4 AgentX 配方，并适配 Python 启动器。

移除配方中与集群级每 GPU CPU 配置冲突的任务级 CPU 参数。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…gentX The pluggable launcher resolves DeepSeek-V4-Pro-0813 on b300-dsxe to the shared /data/models copy for every non-vLLM framework, including multi-node Dynamo+SGLang, which previously read the node-local /scratch/models copy. With several points loading concurrently from shared storage, the TP8 c1 agg worker spent ~2.9h in weight loading and lost its etcd lease; the TP4 c4 agg worker never became healthy within the 4h window on the previous head. Pin every point of both configs to /scratch/models/DeepSeek-V4-Pro-0813 via the MODEL_PATH additional-setting, restoring the storage path under which this recipe set last passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…oint A point-level MODEL_PATH is opaque to the launcher, so srtctl's model preflight (srt-slurm v2.39.1) ran on the runner host and rejected the node-local /scratch/models path. Instead, add dynamo-sglang to the b300-dsxe override that already routes vLLM to DeepSeek-V4-Pro-0813@scratch: the checkpoint is then known to be node-local, model_paths maps the recipe alias to /scratch/models/DeepSeek-V4-Pro-0813, and preflight is skipped as it is for vLLM. Single-node SGLang keeps the shared /data/models copy. Drop the MODEL_PATH additional-settings added in the previous commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Replace the disaggregated 2P1D DEP8 c480 point with a one-node aggregate DEP8 c384 Dynamo+SGLang recipe that follows the single-node SGLang DEP8 c384 point: attention DP8 + EP8 Mega-MoE, FP4 indexer, prefill delayer (interval 20), chunked prefill 65536, max-running-requests 768, cuda-graph-max-bs-decode 544, mem-fraction-static 0.88, swa-full-tokens-ratio 0.075 and a HiCache ratio 3 write_back DRAM tier with the page_first_direct layout. The recipe uses this PR's lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 image and its arg names (tp-size/dp-size/ep-size, attention-backend dsv4 instead of the deprecated compressed alias, no deprecated SGLANG_ENABLE_UNIFIED_RADIX_TREE). The Dynamo frontend tokenizes and routes, so SGLang-server-only settings (tokenizer workers, parsers, chat template, server warmup, keep-alive) are dropped; per-DP-rank KV events feed the Dynamo KV router. Drop enable-w4a4-mxfp4-megamoe from every recipe, so Mega-MoE runs its default FP8xFP4 kernels. 将 B300 分离式 2P1D DEP8 c480 测试点替换为单节点聚合式 DEP8 c384 Dynamo+SGLang 配方，沿用单节点 SGLang DEP8 c384 测试点：注意力 DP8 + EP8 Mega-MoE、FP4 indexer、prefill delayer（间隔 20）、chunked prefill 65536、max-running-requests 768、cuda-graph-max-bs-decode 544、 mem-fraction-static 0.88、swa-full-tokens-ratio 0.075，以及 HiCache 比例 3、write_back、page_first_direct 布局的 DRAM 层。配方使用本 PR 的 lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 镜像及其参数命名（tp-size/dp-size/ep-size、以 attention-backend dsv4 替代已弃用的 compressed 别名、不再设置已弃用的 SGLANG_ENABLE_UNIFIED_RADIX_TREE）。由于 Dynamo 前端负责分词与路由，移除仅适用于 SGLang 服务端的设置（tokenizer worker、解析器、聊天模板、服务端预热、keep-alive）；各 DP rank 的 KV 事件供 Dynamo KV 路由器使用。所有配方移除 enable-w4a4-mxfp4-megamoe，Mega-MoE 使用默认的 FP8xFP4 kernel。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

将 B300 Dynamo+SGLang AgentX 变更日志条目的 pr-link 指向 #3631。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Set cuda-graph-max-bs-decode to 96, the per-DP-rank request cap (max-running-requests 768 / dp-size 8). SGLang already clamps captured decode batch sizes to the per-rank request pool, so 544 never captured more than this; the explicit value states the intended limit. 将 B300 聚合式 DEP8 c384 配方的 cuda-graph-max-bs-decode 设为 96，即每个 DP rank 的请求上限（max-running-requests 768 / dp-size 8）。SGLang 本就会把捕获的 decode batch size 限制在每个 rank 的请求池以内，544 实际上从未捕获超过该值；显式设置可明确预期的上限。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The c384 warmup burst hit Dynamo's 5 s default request-plane ack timeout while the worker ingested the large AgentX requests; the frontend then marked the only worker unreachable and returned 500/503 for every warmup request. Set DYN_TCP_REQUEST_TIMEOUT to 60 s, as in the disaggregated recipes of this PR. c384 预热突发请求期间，worker 接收大量 AgentX 大请求时触发了 Dynamo 请求平面默认 5 秒的确认超时；前端随后将唯一的 worker 标记为不可达，所有预热请求均返回 500/503。与本 PR 的分离式配方一致，将 DYN_TCP_REQUEST_TIMEOUT 设为 60 秒。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Fill the interactivity gap between agg TP4 c4 (~244 tok/s/user) and disagg 1P1D DEP8 c64 (~128 tok/s/user) with two points: - agg TP4 c8: the TP4 c4 recipe at max-running-requests 16 and cuda-graph-max-bs-decode 16, plus a HiCache DRAM tier (ratio 2.75, write_through, direct I/O, page_first_direct, as in the B200 TP8 AgentX recipe on the same image) because TP4 c4 already fills about 60% of the GPU KV pool. - disagg 1P2D DEP8 c64: the 1P1D c64 recipe with a second decode worker, which halves each decode rank's load. 在聚合式 TP4 c4（约 244 tok/s/user）与分离式 1P1D DEP8 c64（约 128 tok/s/user）之间增加两个测试点： - 聚合式 TP4 c8：基于 TP4 c4 配方，max-running-requests 16、 cuda-graph-max-bs-decode 16；由于 TP4 c4 已占用约 60% 的 GPU KV 池，增加 HiCache DRAM 层（比例 2.75、write_through、direct I/O、 page_first_direct，与同一镜像上的 B200 TP8 AgentX 配方相同）。 - 分离式 1P2D DEP8 c64：即 1P1D c64 配方增加第二个 decode worker，使每个 decode rank 的负载减半。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The B300 DSXE disaggregated recipes replace the image's mooncake-transfer-engine-cuda13 0.3.13 with the public EFA build mooncake-transfer-engine-efa-cuda13 0.3.13.post1 at container setup, as SGLang's AWS EFA documentation prescribes, because the nodes connect over AWS EFA and the standard build lacks the EFA transport. Document what is changed, why the unmodified image cannot run the disaggregated points, the upstream links, and the removal plan. B300 DSXE 分离式配方在容器初始化时，按照 SGLang 的 AWS EFA 文档，将镜像中的 mooncake-transfer-engine-cuda13 0.3.13 替换为公开的 EFA 版本 mooncake-transfer-engine-efa-cuda13 0.3.13.post1：节点之间通过 AWS EFA 互联，而标准版本不包含 EFA 传输。本豁免说明所做的改动、未经修改的镜像无法运行分离式测试点的原因、上游链接以及移除计划。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Switch the B300 DeepSeek-V4-Pro-0813 disaggregated recipes (1P1D DEP8 c64/c240 and 1P2D DEP8 c64) from Mooncake to NIXL with the LIBFABRIC backend for the DSXE EFA fabric. The NIXL build already in the pinned lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 image provides the libfabric plugin, so the container runs as shipped: drop the install-mooncake-efa-cu13.sh setup script, which replaced the image's Mooncake package with the EFA wheel, along with its waiver and the Mooncake-only MOONCAKE_PROTOCOL, MC_MS_FILTERS and WITH_NVIDIA_PEERMEM settings. Prefill and decode both set SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC, and the master config reports kv-p2p-transfer: nixl. 将 B300 DeepSeek-V4-Pro-0813 分离式配方（1P1D DEP8 c64/c240 与 1P2D DEP8 c64）的 KV 传输由 Mooncake 改为使用 LIBFABRIC 后端的 NIXL，以适配 DSXE 的 EFA 网络。固定镜像 lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 中自带的 NIXL 已包含 libfabric 插件，容器可按原样运行：移除用 EFA wheel 替换镜像内 Mooncake 包的 install-mooncake-efa-cu13.sh 初始化脚本及其豁免文件，并移除仅供 Mooncake 使用的 MOONCAKE_PROTOCOL、MC_MS_FILTERS 与 WITH_NVIDIA_PEERMEM 设置。Prefill 与 decode 均设置 SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC，主配置中的 kv-p2p-transfer 改为 nixl。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…g AgentX entry 更新 B300 Dynamo+SGLang AgentX 变更日志条目，说明分离式 KV 传输改用 NIXL。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

将 B300 Dynamo+SGLang AgentX 变更日志条目的 pr-link 指向 #3817。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude

Findings marked 🟡 are optional suggestions and need no follow-up push.

github-actions · 2026-10-08T16:35:01Z

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38339272863

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38339272863

nvpohanh · 2026-10-09T01:43:21Z

/use 37715955859

github-actions · 2026-10-09T01:43:37Z

@nvpohanh staged run 37715955859: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-10-08~r37715955859

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

nvpohanh · 2026-10-09T03:05:22Z

doc migration: sgl-project/sglang#43083

AGENTS.md requires new perf-changelog entries to be English-only, so drop the Simplified Chinese description bullets from this PR's entry. Also drop the b300-dsxe aiperf-cache volume from runners.yaml: no b300-dsxe launch lane or srt-slurm volume-mounts entry mounts it, so it had no effect on any job. 按照 AGENTS.md，新增的 perf-changelog 条目须仅用英文，因此删除本 PR 条目中的简体中文描述。另删除 runners.yaml 中 b300-dsxe 的 aiperf-cache 卷：b300-dsxe 的启动路径与 srt-slurm volume-mounts 均未挂载该卷，它对任何任务均无影响。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

functionstackx

does the nixl package itself has the efa userspace lib? curious how thsi works cuz sglang container doesnt install the efa userspace yet

nvpohanh · 2026-10-09T03:23:10Z

@functionstackx Yes, I think NIXL packages the libfabric along with the NIXL so it does not rely on user-provided or container environment libfabric or EFA.

The Mooncake + EFA hassle is because Mooncake does not package the libfabric along with it and it instead dynamically load the .so file at runtime, so all kinds of incompatibility issue shows up.

Correct me if I am wrong

nvpohanh · 2026-10-09T06:00:15Z

[by Claude Code] sgl-project/sglang#43083 has been merged. This PR's AgentX configs (dsv4-fp4-b300-dynamo-sglang-agentic-agg, dsv4-fp4-b300-dynamo-sglang-agentic-disagg) are now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4#agentic-long-context-deployment

The commands mirror the launch measured in run 37715955859, pinned by this PR's /use comment.

nvpohanh · 2026-10-09T06:11:40Z

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Additional detail section:

PR validation: Run Sweep 37715955859 ran on in-PR commit b8bafb340b8be6369e5f4df272cd60d5f7e3c587 (attempt 1, success) and is pinned with /use 37715955859.
- All 7 benchmark jobs succeeded:
  - aggregate TP8 c1, TP4 c4, TP4 c8 (HiCache) and DEP8 c384 (HiCache);
  - disaggregated 1P1D DEP8 c64/c240 and 1P2D DEP8 c64, with NIXL over LIBFABRIC.
- The skipped jobs are empty single-node, fixed-sequence and canary lanes.
- The later head 8a31d2f05bfbe6ae38c8ac7cbb9bde689552fc78 changes only two things, and the generated sweep matrix is identical to the pinned commit's:
  - it removes the Chinese bullets from this PR's changelog description;
  - it removes an unused b300-dsxe aiperf-cache volume from runners.yaml, which no launch lane mounted.
Evals: gsm8k em_strict on all four eval points, which ran on the same image:
- 1P1D c64: 0.9750 (job)
- 1P1D c240: 0.9757 (job)
- 1P2D c64: 0.9719 (job)
- aggregate DEP8 c384: 0.9727 (job)
- All clear the dsv4 gsm8k threshold of 0.91 in infx/evals/thresholds.yaml.
Image and engine order: upstream lmsysorg/sglang:nightly-dev-20260916-c9a8fba9 (SGLang c9a8fba9). Both config keys are framework: dynamo-sglang, so the engine-ordering rule does not apply.
Chat template: AgentX replay runs aiperf profile --scenario agentx. In the pinned aiperf (89b21867), that scenario sets endpoint_type: chat (inferencex_agentx_mvp.py), so every request goes through the model's chat template.
Draft precision:
- Draft: the DSpark head that ships in deepseek-ai/DeepSeek-V4-Pro-0813 (rev 72e1d3230f6c080a530b0a1d46f8eb4602340597, mtp.0 to mtp.2), served with speculative-dspark-block-size: 6.
- Shipped precision, read from the safetensors headers of shards 64-66:
  - attention (wq_a, wq_b, wkv, wo_a, wo_b), shared experts and main_proj are FP8 E4M3 with E8M0 block scales;
  - the 384 routed experts per layer are MXFP4, packed as I8 with E8M0 scales;
  - norms, gate, markov_head and confidence_head are BF16, and hc_* and attn_sink are F32.
  - quantization_config is FP8 E4M3, 128x128 blocks with UE8M0 scales, dynamic activations, and no exclusions.
- Effective precision: the same, through the pinned image's default handling.
  - The logged prefill and decode server args show speculative_draft_model_path=/model (the served checkpoint).
  - They also show speculative_draft_model_quantization=None, speculative_draft_kv_cache_dtype=None, quantization=None and json_model_override_args='{}'.
  - enable_w4a4_mxfp4_megamoe=False, so Mega-MoE runs its default FP8xFP4 kernels. The KV cache is the default fp8_e4m3 for both target and draft.
- SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: not set in any recipe, launcher or job log. It defaults to EnvBool(False) at SGLang c9a8fba9.
Golden AL:
- infx/golden_al_distribution/dsv4-pro-0813-dspark.yaml gives 3.77 for thinking_on at draft length 6.
- All 7 benchmark jobs logged SGLANG_SIMULATE_ACC_LEN=3.77, SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token, with SGLANG_DEFAULT_THINKING=1.
MODELS.md: dsv4 agentic coding is active, and it lists the native DSpark heads in deepseek-ai/DeepSeek-V4-Pro-0813 for agentic coding.
No architecture hacks:
- There is no --hf-overrides or model-config editing.
- The FP4 indexer is target-side lower precision, covered by passing evals. The draft layers carry no indexer weights.
- publish_events_and_metrics is a Dynamo TRT-LLM srt-slurm field and does not apply to this SGLang path. enable-metrics is on.
No engine patches:
- No recipe has a setup_script, and nothing is installed into the container.
- The disaggregated recipes use the image's own NIXL; the worker logs show NIXL ... Backend LIBFABRIC was instantiated.
- No waiver is needed.
Recipe publication:
- Coverage: every B300 deployment in this PR is on the published SGLang cookbook DeepSeek-V4 page, Agentic Long-Context Deployment. It comes from [Docs] Cookbook: AgentX launches for Qwen3.5 / Qwen3.8-Flash-Next; refresh GLM-5.2, DeepSeek-V4, DeepSeek-V4.1 sgl-project/sglang#43083, MERGED 2026-10-09T05:55:37Z.
- Source: the entries are rendered from this PR's pinned run 37715955859 at b8bafb340, through the /use comment.
- Variants:
  - aggregate TP8 c1, TP4 c4/c8 (HiCache) and DEP8 c384 (HiCache);
  - disaggregated 1P1D DEP8 c64/c240 and 1P2D DEP8 c64.
- Commands: they include etcd, NATS, nginx, the Dynamo frontend (--router-mode kv with session affinity and the DYN_*_TIMEOUT settings), and every worker role. The disaggregated prefill and decode commands carry --disaggregation-transfer-backend nixl and SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC.
- InferenceX-only settings that the cookbook omits:
  - --enable-metrics and --enable-cache-report: Prometheus scraping and AIPerf cache-hit accounting, with no effect on serving.
  - DYN_LOG, DYN_SKIP_SGLANG_LOG_FORMATTING, SGLANG_LOG_FORWARD_ITERS, SGLANG_LOG_MS and PYTHONUNBUFFERED: logging only.
  - SGLANG_DG_CACHE_DIR: SA's shared DeepGEMM cache mount; SGLang uses its default directory.
  - PIP_BREAK_SYSTEM_PACKAGES and PYTHONNOUSERSITE: srt-slurm install and Python-path plumbing.
  - SGLANG_SIMULATE_ACC_*: benchmark-only synthetic acceptance, removed and disclosed on each entry.
  - Container paths and ports become checkpoint names and placeholders: /model/, and the decode --kv-events-config port, rendered as 5557 because decode runs on its own node.
- Check: I resolved every entry's command blocks (port of _agentx.jsx materialize) and compared them with the 7 recipes at b8bafb340. Every remaining frontend and worker flag and environment variable matches.
append-only: N/A, because the new changelog entry does not use append-only: true.
Pareto:
- Curve: one curve, dsv4 / agentic-coding / b300_dynamo-sglang / fp4 / run 37715955859.
- Data: the 7 bmk_agentic_* artifacts of the pinned run, using request_metrics.throughput.per_gpu.total_tput_tps against E2EL, counted with infx.workflows.pareto_coverage.
- P90 E2EL frontier: 5 of 7 points (TP4 c8, 1P2D c64, 1P1D c64, 1P1D c240, DEP8 c384). TP8 c1 and TP4 c4 are dominated.
- P75 E2EL frontier: 6 of 7 points. TP4 c4 joins; TP8 c1 stays dominated.
- Dashboard: https://inferencex.semianalysis.com/inference?unofficialRun=37715955859

Signed: nvpohanh

github-actions · 2026-10-09T06:20:00Z

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @nvpohanh is a named owner of inferencex-e2e/configs/nvidia-master.yaml and of *_node/srt-slurm-recipes/**/b300-*/ (which covers all 7 new recipes); models.py and perf-changelog.yaml fall under the catch-all.

✅ Check 1 (Passing sweep on in-PR commit): PASS — Run Sweep 37715955859 (attempt 1, success) ran on in-PR commit b8bafb340. All 7 multi-node agentic / jobs and all 4 multi-node agentic eval / jobs ran and succeeded; the skipped single-node, fixed-sequence and canary lanes have no work in this PR. Head 8a31d2f0 matches the pinned SHA.

✅ Check 2 (Evals): PASS — gsm8k em_strict from eval_results_all of run 37715955859: 1P1D c64 0.9750, 1P1D c240 0.9757, 1P2D c64 0.9719, agg DEP8 c384 0.9727. All clear the dsv4 threshold of 0.91 and ran on lmsysorg/sglang:nightly-dev-20260916-c9a8fba9, the same image as the config.

✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#43083 is MERGED (2026-10-09T05:55:37Z). At merge commit 1fc6b896, docs/src/snippets/agentx/deepseek-ai/deepseek-v4.jsx has B300 dynamo-sglang entries for agg TP8 c1, TP4 c4/c8 (HiCache), DEP8 c384 (HiCache) and disagg 1P1D c64/c240 and 1P2D c64, with provenance run 37715955859 @ b8bafb340. After resolving the ref/diff chains (base entries 12, 14, 16 and envs 0, 5, 6), every engine flag and env var, disaggregation-transfer-backend nixl, SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC, the KV-router frontend args and the DYN_* timeouts match the recipes. The omitted settings (metrics/cache-report, logging, DG cache dir, pip/python-path plumbing, simulated acceptance) are InferenceX-only and are justified in the sign-off. Informational: the cookbook lists 2 (1P1D) and 3 (1P2D) Dynamo frontend replicas behind nginx, but the run launched 1 and 2. This is only a replica count; the router settings match.

✅ Check 4 (Reuse command): PASS — /use 37715955859 was posted on its own line by nvpohanh (COLLABORATOR) on 2026-10-09.

✅ Check 5 (Latest checklist template): PASS — all 17 items in the current PR_REVIEW_CHECKLIST.md template, including the recipe sub-item, are present and checked.

✅ Check 6 (Upstream image / engine order): PASS — both new entries (dsv4-fp4-b300-dynamo-sglang-agentic-agg/-disagg) use upstream lmsysorg/sglang:nightly-dev-20260916-c9a8fba9, and framework: dynamo-sglang is an open-source engine, so the ordering rule does not apply.

✅ Check 7 (Deprecated models): PASS — MODELS.md (as of 2026-10-09) lists dsv4 Agentic coding as active, with native DSpark heads in deepseek-ai/DeepSeek-V4-Pro-0813 allowed for agentic coding.

✅ Check 8 (No architecture hacks): PASS — the diff has no --hf-overrides, json-model-override-args or model-config edits. The FP4 indexer and Mega-MoE are target-side kernel/precision choices covered by passing evals. publish_events_and_metrics appears only in TRT-LLM recipes and is not a setting on this SGLang path; nothing disables event publication, and KV events are published via kv-events-config where the router needs them.

✅ Check 9 (Chat template): PASS — srt_agentic.sh runs aiperf profile --scenario agentx (confirmed in the job logs). At pinned harness 89b21867, inferencex_agentx_mvp.py sets "endpoint_type": "chat".

✅ Check 10 (No engine patches): PASS — no setup_script, .patch, git apply, heredoc rewrites or engine wheel installs; dynamo.install: false. The job logs show only host-side srt-slurm and aiperf client installs, and the Mooncake setup script and waiver were removed from this PR.

✅ Check 11 (Agentic golden AL): PASS — the launcher injected SGLANG_SIMULATE_ACC_LEN=3.77, METHOD=match-expected and TOKEN_MODE=real-draft-token into the agg, prefill and decode roles (Applied override lines in the job logs). This equals dsv4-pro-0813-dspark.yaml thinking_on at block size 6 (3.77), with SGLANG_DEFAULT_THINKING=1.

➖ Check 12 (Append-only): N/A — the new perf-changelog.yaml entry does not set append-only: true.

✅ Check 13 (Draft as shipped): PASS — the draft is the embedded DSpark head of deepseek-ai/DeepSeek-V4-Pro-0813 @ 72e1d323. Its quantization_config is FP8 e4m3, 128x128 blocks, ue8m0 scales, dynamic activations, with no exclusions. The recipes set no draft path, draft quantization, dtype or KV-dtype override and do not enable the W4A4 Mega-MoE. The effective worker env logged in the jobs contains no SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE, and no recipe, launcher or repo default sets it for dsv4.

✅ Check 14 (Pareto coverage): PASS — one curve, dsv4 / agentic-coding / b300_dynamo-sglang / fp4 / run 37715955859, built from the 7 bmk_agentic_* artifacts (per_gpu.total_tput_tps vs E2EL). infx.workflows.pareto_coverage gives P90 5/5 (TP4 c8, 1P2D c64, 1P1D c64, 1P1D c240, DEP8 c384) and P75 6/5. No points are invalid.

✅ Check 15 (PR description): PASS — the body's claims (NIXL/LIBFABRIC on prefill and decode, kv-p2p-transfer: nixl, TP4 c8 HiCache 2.75 write_through, 1P2D = 1P1D c64 plus a second decode worker, DEP8 c384 settings including max-running-requests 768, cuda-graph-max-bs-decode 96, prefill-decode-interval 20, HiCache ratio 3 and DYN_TCP_REQUEST_TIMEOUT=60, node-local checkpoint override, no W4A4 Mega-MoE) match the code at 8a31d2f0.

Assessed commit: 8a31d2f05bfbe6ae38c8ac7cbb9bde689552fc78.

functionstackx

looking

nvpohanh · 2026-10-12T07:57:40Z

@functionstackx Any updates on this? Thanks!

functionstackx · 2026-10-12T18:18:14Z

commented on slasck

functionstackx

to test reproducibability, i deleted the existing cached squash file and rebased the main and resweeped it from main but it doesnt seem to work #3932

…sglang-agentx-nixl # Conflicts: # inferencex-e2e/perf-changelog.yaml

Move all seven B300 DeepSeek-V4-Pro-0813 Dynamo+SGLang AgentX recipes and both master-config keys to lmsysorg/sglang:nightly-dev-20261013-7224fce1. - DEP8 workers use attn-dp-size 8 in place of the deprecated dp-size 8 + enable-dp-attention. - Install Dynamo from commit b208989fbe, which carries ai-dynamo/dynamo#16005. The image's Dynamo 1.6.0.dev20261010 predates it and registers one DP rank for an attention-DP worker of this SGLang, so the KV router would send every request to DP rank 0. b208989fbe is the first commit after the fix whose aisimulate pin matches the image. - Drop env vars and args that have no effect on this image: names it no longer reads, settings only SGLang's HTTP server reads, values equal to their defaults or set automatically for DeepSeek-V4, DSpark or Dynamo, and the Dynamo router rejection thresholds, which are off by default. - Rename SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW to SGLANG_OPT_RELEASE_PREFILL_SWA. 将 B300 DeepSeek-V4-Pro-0813 Dynamo+SGLang AgentX 的七个配方及两个主配置项更新为 lmsysorg/sglang:nightly-dev-20261013-7224fce1。 - DEP8 worker 改用 attn-dp-size 8，取代已弃用的 dp-size 8 + enable-dp-attention。 - 从 Dynamo 提交 b208989fbe 安装 Dynamo，其中包含 ai-dynamo/dynamo#16005。镜像自带的 Dynamo 1.6.0.dev20261010 早于该修复，对该版本 SGLang 的注意力 DP worker 只注册一个 DP rank，KV 路由器会把所有请求发往 DP rank 0。b208989fbe 是该修复之后第一个 aisimulate 版本约束与镜像一致的提交。 - 删除在该镜像中不起作用的环境变量与参数：镜像不再读取的名称、仅 SGLang 自身 HTTP 服务端读取的设置、等于默认值或由 DeepSeek-V4、DSpark、Dynamo 自动设置的值，以及默认关闭的 Dynamo 路由拒绝阈值。 - 将 SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW 重命名为 SGLANG_OPT_RELEASE_PREFILL_SWA。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-13T02:55:43Z

inferencex-e2e/infx diff: +1 -1

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

nvpohanh · 2026-10-13T05:02:05Z

[by Claude Code] The SGLang cookbook update of this PR's AgentX configs (dsv4-fp4-b300-dynamo-sglang-agentic-agg, dsv4-fp4-b300-dynamo-sglang-agentic-disagg) will happen in sgl-project/sglang#43983, in the Agentic Long-Context Deployment section of the DeepSeek-V4 cookbook page. The configs have been live in that section since sgl-project/sglang#43083; sgl-project/sglang#43983 refreshes them with the pin below.

The commands mirror the launch measured in run 37715955859, pinned by this PR's /use comment.

nvpohanh · 2026-10-13T06:45:23Z

[by Claude Code] sgl-project/sglang#43983 has been merged. This PR's AgentX configs (dsv4-fp4-b300-dynamo-sglang-agentic-agg, dsv4-fp4-b300-dynamo-sglang-agentic-disagg) are now in the Agentic Long-Context Deployment section of the SGLang DeepSeek-V4 cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4#agentic-long-context-deployment

The commands mirror the launch measured in run 37715955859, pinned by this PR's /use comment.

nvpohanh · 2026-10-13T08:58:42Z

Need more fixes

nvpohanh requested a review from a team October 8, 2026 02:00

nvpohanh requested review from Ankur-singh, RohitNagraj, kedarpotdar-nv and wzhao18 as code owners October 8, 2026 02:00

github-project-automation Bot added this to InferenceX Board Oct 8, 2026

nvpohanh added a commit that referenced this pull request Oct 8, 2026

chore(changelog): link the B300 Dynamo+SGLang AgentX entry to #3817

fbea025

将 B300 Dynamo+SGLang AgentX 变更日志条目的 pr-link 指向 #3817。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nvpohanh and others added 14 commits October 7, 2026 19:02

feat(dsxe): add Mooncake B300 AgentX recipes

83e4260

新增基于 Mooncake 的 B300 DeepSeek V4 AgentX 配方，并适配 Python 启动器。

fix(dsxe): use cluster CPU directives

3c6cdc4

移除配方中与集群级每 GPU CPU 配置冲突的任务级 CPU 参数。

fix(dsxe): enable DeepGEMM fast warmup for B300 agg recipes

0a4f0f6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chore(changelog): link the B300 Dynamo+SGLang AgentX entry to #3631

d7f5fa6

将 B300 Dynamo+SGLang AgentX 变更日志条目的 pr-link 指向 #3631。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chore(changelog): describe NIXL KV transfer for the B300 Dynamo+SGLan…

99b3d7c

…g AgentX entry 更新 B300 Dynamo+SGLang AgentX 变更日志条目，说明分离式 KV 传输改用 NIXL。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chore(changelog): link the B300 Dynamo+SGLang AgentX entry to #3817

b8bafb3

将 B300 Dynamo+SGLang AgentX 变更日志条目的 pr-link 指向 #3817。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nvpohanh force-pushed the dsv4-fp4-b300-dynamo-sglang-agentx-nixl branch from fbea025 to b8bafb3 Compare October 8, 2026 02:02

nvpohanh added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 8, 2026

claude Bot reviewed Oct 8, 2026

View reviewed changes

Comment thread inferencex-e2e/perf-changelog.yaml Outdated

Comment thread inferencex-e2e/configs/runners.yaml Outdated

functionstackx requested changes Oct 9, 2026

View reviewed changes

nvpohanh mentioned this pull request Oct 9, 2026

[Docs] Cookbook: AgentX launches for Qwen3.5 / Qwen3.8-Flash-Next; refresh GLM-5.2, DeepSeek-V4, DeepSeek-V4.1 sgl-project/sglang#43083

Merged

kedarpotdar-nv approved these changes Oct 9, 2026

View reviewed changes

functionstackx requested changes Oct 9, 2026

View reviewed changes

functionstackx mentioned this pull request Oct 12, 2026

Recreate B300 DeepSeek-V4 AgentX NIXL recipes for a full sweep / 重建 B300 DeepSeek-V4 AgentX NIXL 配方并运行完整测试 #3932

Open

7 of 11 tasks

functionstackx requested changes Oct 12, 2026

View reviewed changes

nvpohanh and others added 2 commits October 12, 2026 19:10

Merge remote-tracking branch 'origin/main' into dsv4-fp4-b300-dynamo-…

a8cd60a

…sglang-agentx-nixl # Conflicts: # inferencex-e2e/perf-changelog.yaml

nvpohanh removed the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 13, 2026

nvpohanh added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 13, 2026

nvpohanh mentioned this pull request Oct 13, 2026

[Docs] Cookbook: refresh SemiAnalysis AgentX launches (DeepSeek-V4.1 GB200/GB300, GLM-5.2 FP8 B300) sgl-project/sglang#43983

Merged

Conversation

nvpohanh commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Aggregate DEP8 c384

Gap-filling points: agg TP4 c8 and disagg 1P2D DEP8 c64

NIXL instead of Mooncake

Type of Change

Checklist

摘要

针对 1013 镜像删除或重命名的设置

配方细节（沿用自 #3631）

AI 模型披露

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvpohanh commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvpohanh commented Oct 9, 2026

Uh oh!

functionstackx left a comment

Choose a reason for hiding this comment

Uh oh!

nvpohanh commented Oct 9, 2026

Uh oh!

nvpohanh commented Oct 9, 2026

Uh oh!

nvpohanh commented Oct 9, 2026

Additional detail section:

Uh oh!

github-actions Bot commented Oct 9, 2026

✅✅✅ Verdict: PASS ✅✅✅

Uh oh!

functionstackx left a comment

Choose a reason for hiding this comment

Uh oh!

nvpohanh commented Oct 12, 2026

Uh oh!

functionstackx commented Oct 12, 2026

Uh oh!

functionstackx left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 13, 2026

Uh oh!

nvpohanh commented Oct 13, 2026

Uh oh!

nvpohanh commented Oct 13, 2026

Uh oh!

nvpohanh commented Oct 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nvpohanh commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 9, 2026 •

edited

Loading

functionstackx left a comment •

edited

Loading