Repository navigation
Add Qwen3.5 B300 Dynamo-SGLang AgentX recipes / 添加 Qwen3.5 B300 Dynamo-SGLang AgentX 配方 - #2978
RohitNagraj wants to merge 9 commits into
Conversation
添加 Qwen3.5 NVFP4 B300 Dynamo-SGLang AgentX 配置和配方，并校验合成接受长度与已提交的黄金曲线一致。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
在性能变更日志中补充 Qwen3.5 B300 PR #2978 链接。
RohitNagraj
added
the
full-sweep-enabled
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond what's already flagged inline, I checked whether the launch script's job-name sed override (sed -i "s/^name:.*/.../") would silently no-op on the two new recipes that wrap everything under a top-level base: key (where name: is indented two spaces) — that base:/zip_override_running_cap: matrix format predates this PR (already used by the glm5/glm5.2 recipes), so it's pre-existing behavior rather than a regression introduced here.
Extended reasoning...
This run reported a confirmed finding (perf-changelog.yaml's new entry still has a placeholder pull/XXX pr-link) that will be posted inline, plus additional verified findings that are not being surfaced individually in this review body per the dedup/cap policy. Since verified findings exist, approval is not appropriate. I independently re-examined the candidate concern that two of the four new recipe YAMLs indent name: under a top-level base: key, which would make the launch script's unconditional sed -i "s/^name:.*/name: \"${RUNNER_NAME}\"/" fail to match and silently leave the recipe's default job name in place. I confirmed via git grep against the pre-PR commit that this base: + zip_override_running_cap: matrix wrapping format already exists in the repository (glm5 and glm5.2 recipes use it), so the same sed limitation already applied before this change — it is not a new regression introduced by this diff, just an existing gap this PR's new recipes happen to also hit. Given the outstanding verified-but-unposted findings, a human should still review before merge.
Additional findings (no inline location):
-
🟡
perf-changelog.yaml— The new changelog entry's pr-link is still the placeholder.../pull/XXXinstead of the real PR URL, unlike every other historical entry which links a real PR. Fix: replace with the actual pull request URL before merge, per the append-only changelog convention that requires a real, verifiable PR link for traceability.Extended reasoning...
perf-changelog.yaml is append-only and byte-sensitive; the new entry at the physical tail (config-keys: qwen3.5-fp4-b300-dynamo-sglang-agentic-pareto) ends with
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. All prior entries (2925, 2954, 2955, 2969, ...) use real PR numbers. If merged as-is, anyone auditing this recipe/config change later via the changelog gets a dead/placeholder link instead of the actual PR, breaking the traceability the append-only convention exists to guarantee.Verification: nit. The changelog entry added by this diff at line 7149 of perf-changelog.yaml ends with
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX(a placeholder), for config-keysqwen3.5-fp4-b300-dynamo-sglang-agentic-pareto, while adjacent entries such as #2925 (line 7143) use real PR URLs. The finding is factually real. Caveat to the candidate's wording: it claims every other… | nit.…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34515330805 |
延长 Qwen3.5 AgentX 配方的 Slurm 时间限制。
合并 origin/main，并将本分支的性能变更日志条目重新追加到文件末尾。
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38339289942 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38339289942 |
|
/reuse-sweep-run 34548026742 |
Align the Qwen3.5 B300 recipes with the current SRT schema and recipe layout, use automatic golden acceptance selection, and preserve the changelog tail. 修复 Qwen3.5 B300 配方的合并冲突：对齐当前 SRT 配置格式与配方目录， 使用自动黄金接受长度选择，并将变更日志条目保留在文件末尾。
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep on in-PR commit): PASS — commit ✅ Check 2 (Evals pass): PASS — ➖ Check 3 (Recipe linked and merged): N/A — disaggregated/multi-node submission ( ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist template): PASS — all sixteen items of the current ✅ Check 6 (Upstream image, engine-first): PASS — image is upstream ✅ Check 7 (No deprecated models/scenarios): PASS — ✅ Check 8 (No architecture hacks): PASS — no ✅ Check 9 (Spec-decode via chat template): PASS — ✅ Check 10 (No engine patches): PASS — no ✅ Check 11 (Agentic spec-decode golden AL): PASS — ➖ Check 12 (Append-only): N/A — the new Assessed commit: |
|
Sorry, over the weekend, there was 2 major refactors to clean up the technical debt accumalated over the past 11 months of moving at the speed of light. We don't see any major refactors in the forthseeable future besides cleaning up AMD multinode AgentX pile of bash. As much, due to the refactors, u would need to ask your agent to rebase from remote main@latest. Thank you in advance for ur understanding |
|
Heads-up: #3576 (merged) replaced the bash launchers with a Python launcher, so this PR will conflict when you merge |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( |
|
[by Claude Code] sgl-project/sglang#43083 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 34548026742, pinned by this PR's |
移除 Qwen3.5 B300 AgentX 配方中指向 /tmp 的缓存环境变量设置。
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
| SGLANG_FLASHINFER_FP4_GEMM_BACKEND: cutlass | ||
| SGLANG_MOE_NVFP4_DISPATCH: '1' | ||
| SGLANG_CUTEDSL_MOE_NVFP4_DISPATCH: '1' | ||
| SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE: '1' |
There was a problem hiding this comment.
this is not allowed
|
i have removed nextn quant here which isn't allowed by the agreed upon guidelines #3931 https://github.com/SemiAnalysisAI/InferenceX/blob/main/inferencex-e2e/docs/PR_REVIEW_CHECKLIST.md |
There was a problem hiding this comment.
to test reproducitivity, i deleted the existing cached squash file, rebased it onto main and removing nextn quant it doesnt seem to work
https://github.com/SemiAnalysisAI/InferenceX/actions/runs/38289020106/job/114945328806?pr=3931
Resolve the perf-changelog.yaml conflict by keeping main's file unchanged and re-appending this PR's entry at the tail. 合并 origin/main，解决 perf-changelog.yaml 冲突：保持 main 中的文件内容不变， 并将本 PR 的条目重新追加到文件末尾。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-7224fce1 - Bump the four recipes and the master entry to lmsysorg/sglang:nightly-dev-20261013-7224fce1 and rename the recipe variants to match. - Drop SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE so the NextN draft layer is served at its shipped precision. - Replace aliases this image removed: cuda-graph-max-bs becomes cuda-graph-max-bs-decode (the old spelling is now an ambiguous prefix), and SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 becomes SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0. - Drop settings with no effect in this image: environment variables it no longer reads (SGLANG_ENABLE_SPEC_V2, SGLANG_ENABLE_FLASHINFER_GEMM, SGLANG_FLASHINFER_FP4_GEMM_BACKEND, SGLANG_CUTEDSL_MOE_NVFP4_DISPATCH), false booleans srt-slurm never passes (including the deprecated enable-dp-attention), and settings whose readers these TP-only aggregate workers never reach (DP-attention scheduler gathers, Mooncake and NVSHMEM variables, load-balance-method, moe-dense-tp-size, and hicache-io-backend: kernel, which page_first_direct rewrites to direct). - Drop the recipe-level cpus-per-task directive, which conflicts with the b300-dsxe cpus-per-gpu default, and run the AgentX client from benchmarks/srt_agentic.sh. 将 Qwen3.5 B300 SGLang AgentX 迁移到 nightly-dev-20261013-7224fce1： - 将四个配方与主配置条目升级到 lmsysorg/sglang:nightly-dev-20261013-7224fce1，并相应重命名配方变体。 - 移除 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE，使 NextN 草稿层以发布时的精度运行。 - 替换该镜像已移除的别名：cuda-graph-max-bs 改为 cuda-graph-max-bs-decode （旧写法现在是有歧义的前缀），SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 改为 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0。 - 移除在该镜像中无效的设置：不再被读取的环境变量（SGLANG_ENABLE_SPEC_V2、 SGLANG_ENABLE_FLASHINFER_GEMM、SGLANG_FLASHINFER_FP4_GEMM_BACKEND、 SGLANG_CUTEDSL_MOE_NVFP4_DISPATCH），srt-slurm 从不传递的 false 布尔参数 （包括已弃用的 enable-dp-attention），以及这些仅 TP 的聚合 worker 不会 触达的设置（DP 注意力调度器 gather、Mooncake 与 NVSHMEM 变量、 load-balance-method、moe-dense-tp-size，以及会被 page_first_direct 改写为 direct 的 hicache-io-backend: kernel）。 - 移除与 b300-dsxe 默认 cpus-per-gpu 冲突的配方级 cpus-per-task 指令，并改为 通过 benchmarks/srt_agentic.sh 运行 AgentX 客户端。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nvpohanh
added
full-sweep-fail-fast
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( The commands mirror the launch measured in run 34548026742, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#43983 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 34548026742, pinned by this PR's |
This branch has not been deployed
5 participants
[by Claude Code]
Summary
Adds the Qwen3.5 NVFP4 B300 Dynamo+SGLang AgentX configuration
qwen3.5-fp4-b300-dynamo-sglang-agentic-pareto. It runs five aggregate MTP points onb300-dsxebehindsglang-router, built from four srt-slurm recipes:Synthetic acceptance uses the committed Qwen3.5 thinking-on golden acceptance length for each recipe's MTP step count. For Qwen3.5 SGLang, the injector also requires
speculative-num-draft-tokens = speculative-num-steps + 1.This update moves all four recipes to
lmsysorg/sglang:nightly-dev-20261013-7224fce1(SGLang7224fce1). That image is already CUDA 13, so its tag has nocu13suffix. The update also aligns the recipes with that image and with currentmain:SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEfrom all four recipes.cuda-graph-max-bs-decodeinstead ofcuda-graph-max-bs. The old alias is gone, and--cuda-graph-max-bsis now an ambiguous prefix of--cuda-graph-max-bs-decodeand--cuda-graph-max-bs-prefill.SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1becomesSGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0. The old name is no longer forwarded, so without this change the TP memory-imbalance check would raise instead of warn.enable-dp-attention: falseis dropped rather than converted toattn-dp-size.SGLANG_ENABLE_SPEC_V2,SGLANG_ENABLE_FLASHINFER_GEMM,SGLANG_FLASHINFER_FP4_GEMM_BACKEND, andSGLANG_CUTEDSL_MOE_NVFP4_DISPATCH. Its replacement,SGLANG_MOE_NVFP4_DISPATCH=1, stays.falsebooleans that srt-slurm never passes to SGLang:enable-dp-attention,enable-dp-lm-head,enable-symm-mem, andenable-dynamic-chunking.SGLANG_SCHEDULER_SKIP_ALL_GATHERandSGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH: read only by the DP-attention MLP sync.MC_FORCE_MNNVLandMC_TE_METRIC: no Mooncake component runs.NVSHMEM_REMOTE_TRANSPORT: no DeepEP or pplx dispatcher runs.load-balance-method: read only when DP > 1.moe-dense-tp-size: read only by the attention-DP and attention-TP-gather paths, and the recipes setdisable-attn-tp-gather.hicache-io-backend: kernel: SGLang rewrites it todirectforpage_first_direct.main: mergesmainand makes two more fixes.cpus-per-task: '144'. It conflicts with theb300-dsxecpus-per-gpudefault and fails withsbatch: fatal: --cpus-per-task, --tres-per-task=cpu:#, and --cpus-per-gpu are mutually exclusive, as in the Recreate Qwen3.5 B300 AgentX recipes for a full sweep / 重建 Qwen3.5 B300 AgentX 配方并运行完整测试 #3931 canary.benchmarks/srt_agentic.sh, becausebenchmarks/multi_node/agentic_srt.shno longer exists.Note: the earlier
/reuse-sweep-run 34548026742comment pins a run on the previous image. Before merging, a new/use <run_id>must name a green sweep of this head.AI model disclosure
claude-opus-5-5(Claude Opus 5.5), through Claude Code.nightly-dev-20261013-7224fce1update in this revision: image bump, flag and environment audit against the SGLang7224fce1source, merge ofmain, local validation, and this description.PR checklist and change type
Type of Change
Checklist
inferencex-e2e/perf-changelog.yamland have not edited historical entriesfull-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
概述
新增 Qwen3.5 NVFP4 B300 Dynamo+SGLang AgentX 配置
qwen3.5-fp4-b300-dynamo-sglang-agentic-pareto。它在b300-dsxe上通过sglang-router运行五个聚合式 MTP 测试点，由四个 srt-slurm 配方组成：合成接受长度按各配方的 MTP 步数取自已提交的 Qwen3.5 thinking-on 黄金接受长度。对 Qwen3.5 SGLang，注入器还要求
speculative-num-draft-tokens = speculative-num-steps + 1。本次更新将四个配方迁移到
lmsysorg/sglang:nightly-dev-20261013-7224fce1（SGLang7224fce1）。该镜像本身已基于 CUDA 13，因此标签中不带cu13后缀。同时，本次更新使配方与该镜像和当前main保持一致：SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE。cuda-graph-max-bs-decode，不再使用cuda-graph-max-bs。旧别名已被移除，而--cuda-graph-max-bs现在是--cuda-graph-max-bs-decode和--cuda-graph-max-bs-prefill的歧义前缀。SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1改为SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0。旧变量名不再被转发；若不修改，TP 显存不均衡检查会直接报错，而不是只发出警告。enable-dp-attention: false，而不是改写为attn-dp-size。SGLANG_ENABLE_SPEC_V2、SGLANG_ENABLE_FLASHINFER_GEMM、SGLANG_FLASHINFER_FP4_GEMM_BACKEND和SGLANG_CUTEDSL_MOE_NVFP4_DISPATCH。其替代变量SGLANG_MOE_NVFP4_DISPATCH=1保留。false布尔参数：enable-dp-attention、enable-dp-lm-head、enable-symm-mem和enable-dynamic-chunking。SGLANG_SCHEDULER_SKIP_ALL_GATHER和SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH：只被 DP 注意力的 MLP 同步读取。MC_FORCE_MNNVL和MC_TE_METRIC：没有运行任何 Mooncake 组件。NVSHMEM_REMOTE_TRANSPORT：没有运行 DeepEP 或 pplx 分发器。load-balance-method：只在 DP > 1 时读取。moe-dense-tp-size：只被注意力 DP 和注意力 TP gather 路径读取，而配方已设置disable-attn-tp-gather。hicache-io-backend: kernel：在page_first_direct布局下，SGLang 会将其改写为direct。main： 合并main，并另做两项修复。cpus-per-task: '144'。它与b300-dsxe默认的cpus-per-gpu冲突，会报错sbatch: fatal: --cpus-per-task, --tres-per-task=cpu:#, and --cpus-per-gpu are mutually exclusive，与 Recreate Qwen3.5 B300 AgentX recipes for a full sweep / 重建 Qwen3.5 B300 AgentX 配方并运行完整测试 #3931 的 canary 失败相同。benchmarks/srt_agentic.sh运行 AgentX 客户端，因为benchmarks/multi_node/agentic_srt.sh已不存在。注意： 此前的
/reuse-sweep-run 34548026742评论固定的是旧镜像上的运行。合并前，需要用新的/use <run_id>指定当前 head 上全部通过的 sweep。AI 模型使用说明： 本次修订由
claude-opus-5-5（Claude Opus 5.5）通过 Claude Code 完成，工作内容包括：升级镜像，对照 SGLang7224fce1源码审查参数和环境变量，合并main，本地验证，以及撰写本说明。此前的提交没有记录所用模型，因此无法确认。