[Klaud Cold] Update qwen3.5-fp4-gb200-dynamo-sglang-mtp SGLang image to v0.5.21-cu130 / [Klaud Cold] 将 qwen3.5-fp4-gb200-dynamo-sglang-mtp 的 SGLang 镜像更新至 v0.5.21-cu130 by Klaud-Cold · Pull Request #4090 · SemiAnalysisAI/InferenceX

Klaud-Cold · 2026-10-15T04:44:11Z

Goal: Update SGLang image from lmsysorg/sglang:nightly-dev-cu13-20260709-074bb928 to lmsysorg/sglang:v0.5.21-cu130.
Baseline: 2026-10-13 · lmsysorg/sglang:nightly-dev-cu13-20260709-074bb928
Mean latency · Sources: API 1, API 2, API 3

Point	Total tok/s/GPU ↑	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
8k/1k c1 PTP4x1/DTP4x1 9c5e21	352.08	78.65	376.01	2.8
8k/1k c4 PTP4x1/DTP4x1 e4311a	1,123.39	249.88	527.94	3.23
8k/1k c8 PTP4x1/DTP4x1 08138a	1,803.69	405.43	667.49	4.02
8k/1k c128 PTP4x3/DTP4x5 c7da62	4,878.33	864.3	1,317.44	5.54
8k/1k c512 PTP2x2/DTP2x2 3b5db2	12,664.92	2,814.06	27,633.23	13.1
8k/1k c768 PTP2x2/DTP2x3 7de907	12,606.11	2,333.47	37,238.78	11.38
8k/1k c5120 PTP4x7/DTP16x1 1caedd	17,479.73	5,338.62	35,477.48	18.36
8k/1k c8192 PTP4x7/DTP16x1 748a08	17,594.47	5,374.85	61,207.2	21.8
8k/1k c12288 PTP4x7/DTP4x4 40c7c4	18,413.4	5,624.9	91,869.48	30.65

Eval	Score ↑	Samples
gsm8k/em_strict · c128	96.97%	1,319
gsm8k/em_strict · c512	96.66%	1,319
gsm8k/em_strict · c768	96.89%	1,319
gsm8k/em_strict · c5120	97.19%	1,319
gsm8k/em_strict · c8192	97.04%	1,319
gsm8k/em_strict · c12288	96.82%	1,319

中文

**目标：**将 SGLang 镜像从 lmsysorg/sglang:nightly-dev-cu13-20260709-074bb928 更新为 lmsysorg/sglang:v0.5.21-cu130。
**基线：**2026-10-13 · lmsysorg/sglang:nightly-dev-cu13-20260709-074bb928
平均延迟 · 来源： API 1, API 2, API 3；数值及异常说明见上表。

….5.21-cu130 Pin lmsysorg/sglang:v0.5.21-cu130 by digest and rename options removed in v0.5.21: mamba-scheduler-strategy -> mamba-radix-cache-strategy, cuda-graph-max-bs -> cuda-graph-max-bs-decode, and SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 -> SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0. The old image accepted the old names as deprecated aliases for the same settings. 将 SGLang 镜像更新为 v0.5.21-cu130（按 digest 固定），并重命名 v0.5.21 中已移除的选项：mamba-scheduler-strategy 改为 mamba-radix-cache-strategy， cuda-graph-max-bs 改为 cuda-graph-max-bs-decode， SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 改为 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0。旧镜像将旧名称作为同一设置的弃用别名接受。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-15T04:44:19Z

Thanks for the contribution!

Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

**审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
**PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
**合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

Klaud-Cold · 2026-10-15T04:45:37Z

Initial attempt · Failed · Run 38683066951 / attempt 1 · 2026-10-15 06:15 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 4f77f88eac28 · Mean latency
Change: Pin lmsysorg/sglang:v0.5.21-cu130 by digest and rename options v0.5.21 removed: mamba-scheduler-strategy to mamba-radix-cache-strategy, cuda-graph-max-bs to cuda-graph-max-bs-decode, SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 to SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 (old image accepted the old names as aliases of the same settings; sources: sgl-project/sglang 074bb928 python/sglang/srt/server_args.py DeprecatedAliasStoreAction, v0.5.21 python/sglang/srt/arg_groups/fields/exec_.py and environ.py).

Point	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
8k/1k c1 PTP4x1/DTP4x1 9c5e21	N/A	N/A	N/A
8k/1k c4 PTP4x1/DTP4x1 e4311a	N/A	N/A	N/A
8k/1k c8 PTP4x1/DTP4x1 08138a	N/A	N/A	N/A
8k/1k c128 PTP4x3/DTP4x5 c7da62	N/A	N/A	N/A
8k/1k c512 PTP2x2/DTP2x2 3b5db2	N/A	N/A	N/A
8k/1k c768 PTP2x2/DTP2x3 7de907	N/A	N/A	N/A
8k/1k c5120 PTP4x7/DTP16x1 1caedd	N/A	N/A	N/A
8k/1k c8192 PTP4x7/DTP16x1 748a08	N/A	N/A	N/A
8k/1k c12288 PTP4x7/DTP4x4 40c7c4	N/A	N/A	N/A

Note: All rows: request errors unavailable.
Note: 8k/1k c5120 PTP4x7/DTP16x1 1caedd, 8k/1k c8192 PTP4x7/DTP16x1 748a08: cancelled.
Note: 8k/1k c5120 PTP4x7/DTP16x1 1caedd, 8k/1k c8192 PTP4x7/DTP16x1 748a08, 8k/1k c12288 PTP4x7/DTP4x4 40c7c4: Δ N/A: point unavailable.
Note: 8k/1k c12288 PTP4x7/DTP4x4 40c7c4: failed.

Eval	Score ↑	Samples
gsm8k/em_strict · c128	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c512	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c768	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c5120	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c8192	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c12288	N/A	1,319/N/A (old/new)

Note: gsm8k/em_strict · c128, gsm8k/em_strict · c512, gsm8k/em_strict · c768, gsm8k/em_strict · c5120, gsm8k/em_strict · c8192: unavailable.
Note: All rows: Δ N/A: eval score unavailable.
Note: gsm8k/em_strict · c12288: failed.

Next: Superseded by repair 1.

中文

初次尝试 · 失败 · Run 38683066951 / attempt 1 · 2026-10-15 06:15 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 4f77f88eac28 · 平均延迟
**变更：**按 digest 固定 lmsysorg/sglang:v0.5.21-cu130，并重命名 v0.5.21 已移除的选项：mamba-scheduler-strategy 改为 mamba-radix-cache-strategy，cuda-graph-max-bs 改为 cuda-graph-max-bs-decode，SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK=1 改为 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0（旧镜像将旧名称作为同一设置的别名接受；来源：sgl-project/sglang 074bb928 的 python/sglang/srt/server_args.py DeprecatedAliasStoreAction，以及 v0.5.21 的 python/sglang/srt/arg_groups/fields/exec_.py 和 environ.py）。实测数值及异常说明见上表。
**下一步：**已被修复 1 取代。

…c12288 The c12288 recipe pinned the ai-dynamo 1.2.1 wheel over the image's bundled ai-dynamo. dynamo.sglang 1.2.1 imports sglang.srt.server_args_config_parser, which SGLang v0.5.21 moved under sglang.srt.utils, so workers failed at import. Use the bundled Dynamo like the family's other recipes. c12288 配方用 ai-dynamo 1.2.1 wheel 覆盖了镜像自带的 ai-dynamo。 dynamo.sglang 1.2.1 导入 sglang.srt.server_args_config_parser，而 SGLang v0.5.21 已将其移至 sglang.srt.utils，导致 worker 导入失败。改为与本配置族其他配方一样使用镜像自带的 Dynamo。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Klaud-Cold · 2026-10-15T05:20:30Z

Repair 1/5 · Failed · Run 38686225197 / attempt 1 · 2026-10-15 06:52 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 8852297023cf · Mean latency
Change: Measured f182efb: set c12288 dynamo.install to false and drop its ai-dynamo 1.2.1 wheel pin so it uses the image's bundled ai-dynamo 1.6.0.dev20260930 like the other eight recipes (dynamo.sglang 1.2.1 imports sglang.srt.server_args_config_parser, moved to sglang.srt.utils in sgl-project/sglang 94183a8d2b; ai-dynamo main imports it through dynamo.sglang._compat).

Point	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
8k/1k c1 PTP4x1/DTP4x1 9c5e21	N/A	N/A	N/A
8k/1k c4 PTP4x1/DTP4x1 e4311a	N/A	N/A	N/A
8k/1k c8 PTP4x1/DTP4x1 08138a	N/A	N/A	N/A
8k/1k c128 PTP4x3/DTP4x5 c7da62	N/A	N/A	N/A
8k/1k c512 PTP2x2/DTP2x2 3b5db2	N/A	N/A	N/A
8k/1k c768 PTP2x2/DTP2x3 7de907	N/A	N/A	N/A
8k/1k c5120 PTP4x7/DTP16x1 1caedd	N/A	N/A	N/A
8k/1k c8192 PTP4x7/DTP16x1 748a08	N/A	N/A	N/A
8k/1k c12288 PTP4x7/DTP4x4 40c7c4	N/A	N/A	N/A

Note: 8k/1k c1 PTP4x1/DTP4x1 9c5e21, 8k/1k c4 PTP4x1/DTP4x1 e4311a, 8k/1k c128 PTP4x3/DTP4x5 c7da62, 8k/1k c512 PTP2x2/DTP2x2 3b5db2, 8k/1k c768 PTP2x2/DTP2x3 7de907, 8k/1k c5120 PTP4x7/DTP16x1 1caedd, 8k/1k c8192 PTP4x7/DTP16x1 748a08, 8k/1k c12288 PTP4x7/DTP4x4 40c7c4: cancelled.
Note: All rows: request errors unavailable.
Note: All rows: Δ N/A: point unavailable.
Note: 8k/1k c8 PTP4x1/DTP4x1 08138a: failed.

Eval	Score ↑	Samples
gsm8k/em_strict · c128	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c512	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c768	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c5120	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c8192	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c12288	N/A	1,319/N/A (old/new)

Note: All rows: Δ N/A: eval score unavailable.
Note: gsm8k/em_strict · c5120, gsm8k/em_strict · c8192: unavailable.
Note: gsm8k/em_strict · c12288: failed.

Next: Repair 2: set cuda-graph-backend-prefill to disabled on every prefill role to keep the old image's effective prefill graph setting.

中文

修复 1/5 · 失败 · Run 38686225197 / attempt 1 · 2026-10-15 06:52 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 8852297023cf · 平均延迟
**变更：**测量提交 f182efb：将 c12288 的 dynamo.install 设为 false 并移除 ai-dynamo 1.2.1 wheel 固定，使其与其他八个配方一样使用镜像自带的 ai-dynamo 1.6.0.dev20260930（dynamo.sglang 1.2.1 导入 sglang.srt.server_args_config_parser，该模块已在 sgl-project/sglang 94183a8d2b 中移至 sglang.srt.utils；ai-dynamo main 通过 dynamo.sglang._compat 导入）。实测数值及异常说明见上表。
**下一步：**修复 2：在每个 prefill 角色上将 cuda-graph-backend-prefill 设为 disabled，以保持旧镜像实际生效的 prefill 图设置。

…sabled SGLang 074bb928 auto-disabled breakable prefill CUDA graphs for multimodal models and DP attention, so these Qwen3.5 prefill workers ran without them. v0.5.21 marks Qwen3_5MoeForConditionalGeneration as breakable-supported and drops the DP-attention rule; the DEP4 prefill workers then OOM capturing a 65536-token graph. Set cuda-graph-backend-prefill: disabled on every prefill role to keep the old effective setting. SGLang 074bb928 会对多模态模型和 DP attention 自动禁用 breakable prefill CUDA graph，因此这些 Qwen3.5 prefill worker 此前不使用它。 v0.5.21 将 Qwen3_5MoeForConditionalGeneration 标记为支持 breakable，并移除了 DP attention 规则；DEP4 prefill worker 在捕获 65536 token 的图时显存不足。在每个 prefill 角色上设置 cuda-graph-backend-prefill: disabled，以保持旧的实际设置。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Klaud-Cold · 2026-10-15T06:53:05Z

Repair 2/5 · Running · Run 38695089128 / attempt 1 · 2026-10-15 10:15 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · b301e68225f8 · Mean latency
Change: Measured 57441f3: add cuda-graph-backend-prefill disabled to every prefill role, matching the old image where python/sglang/srt/server_args.py at sgl-project/sglang 074bb928 disabled breakable prefill graphs for multimodal models and DP attention, which v0.5.21 python/sglang/srt/arg_groups/cuda_graph_hook.py and configs/model_config.py no longer do for Qwen3_5MoeForConditionalGeneration.

Point	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
8k/1k c1 PTP4x1/DTP4x1 9c5e21	N/A	N/A	N/A
8k/1k c4 PTP4x1/DTP4x1 e4311a	N/A	N/A	N/A
8k/1k c8 PTP4x1/DTP4x1 08138a	N/A	N/A	N/A
8k/1k c128 PTP4x3/DTP4x5 c7da62	N/A	N/A	N/A
8k/1k c512 PTP2x2/DTP2x2 3b5db2	N/A	N/A	N/A
8k/1k c768 PTP2x2/DTP2x3 7de907	N/A	N/A	N/A
8k/1k c5120 PTP4x7/DTP16x1 1caedd	N/A	N/A	N/A
8k/1k c8192 PTP4x7/DTP16x1 748a08	N/A	N/A	N/A
8k/1k c12288 PTP4x7/DTP4x4 40c7c4	N/A	N/A	N/A

Note: All rows: request errors unavailable.
Note: 8k/1k c5120 PTP4x7/DTP16x1 1caedd, 8k/1k c12288 PTP4x7/DTP4x4 40c7c4: pending.
Note: 8k/1k c5120 PTP4x7/DTP16x1 1caedd, 8k/1k c12288 PTP4x7/DTP4x4 40c7c4: Δ N/A: point unavailable.

Eval	Score ↑	Samples
gsm8k/em_strict · c128	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c512	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c768	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c5120	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c8192	N/A	1,319/N/A (old/new)
gsm8k/em_strict · c12288	N/A	1,319/N/A (old/new)

Note: All rows: Δ N/A: eval score unavailable.
Note: gsm8k/em_strict · c5120: unavailable.

Next: Wait for the smoke benchmarks and evals to finish.

中文

修复 2/5 · 运行中 · Run 38695089128 / attempt 1 · 2026-10-15 10:15 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · b301e68225f8 · 平均延迟
**变更：**测量提交 57441f3：在每个 prefill 角色上添加 cuda-graph-backend-prefill disabled，与旧镜像一致；sgl-project/sglang 074bb928 的 python/sglang/srt/server_args.py 会对多模态模型和 DP attention 禁用 breakable prefill 图，而 v0.5.21 的 python/sglang/srt/arg_groups/cuda_graph_hook.py 和 configs/model_config.py 对 Qwen3_5MoeForConditionalGeneration 不再这样做。实测数值及异常说明见上表。
**下一步：**等待冒烟基准测试和评测完成。

Klaud-Cold · 2026-10-15T12:23:32Z

unexpected-error · Cleanup pending. Stop and confirm owned runs before closing.

中文

unexpected-error · 清理待完成。先停止并确认自有运行结束，再关闭 PR。

Klaud-Cold · 2026-10-15T12:23:48Z

unexpected-error · Repairs: unknown · Runs: 38683066951, 38686225197, 38695089128
All owned runs ended. PR closed; branch deleted for retry.

中文

unexpected-error · 修复次数：unknown · 运行：38683066951, 38686225197, 38695089128
所有自有运行均已结束。PR 已关闭；分支已删除，可重新尝试。

github-project-automation Bot added this to InferenceX Board Oct 15, 2026

Klaud-Cold closed this Oct 15, 2026

Klaud-Cold deleted the klaud/auto-eec85ea84e2c2c2e-24b897d31b2c8322 branch October 15, 2026 12:23

github-project-automation Bot moved this to Done in InferenceX Board Oct 15, 2026

Navigation Menu

Repository navigation

[Klaud Cold] Update qwen3.5-fp4-gb200-dynamo-sglang-mtp SGLang image to v0.5.21-cu130 / [Klaud Cold] 将 qwen3.5-fp4-gb200-dynamo-sglang-mtp 的 SGLang 镜像更新至 v0.5.21-cu130 - #4090

[Klaud Cold] Update qwen3.5-fp4-gb200-dynamo-sglang-mtp SGLang image to v0.5.21-cu130 / [Klaud Cold] 将 qwen3.5-fp4-gb200-dynamo-sglang-mtp 的 SGLang 镜像更新至 v0.5.21-cu130#4090
Klaud-Cold wants to merge 3 commits into
mainfrom
klaud/auto-eec85ea84e2c2c2e-24b897d31b2c8322

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Uh oh!

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Uh oh!

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Uh oh!

Klaud-Cold commented Oct 15, 2026

Uh oh!

Klaud-Cold commented Oct 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Klaud-Cold commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

Klaud-Cold commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Klaud-Cold commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Klaud-Cold commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Klaud-Cold commented Oct 15, 2026

Uh oh!

Klaud-Cold commented Oct 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Klaud-Cold commented Oct 15, 2026 •

edited

Loading