[Klaud Cold] Update Qwen3.5 FP8 B300 SGLang AgentX MTP image to v0.5.21-cu130 / [Klaud Cold] 将 Qwen3.5 FP8 B300 SGLang AgentX MTP 镜像更新至 v0.5.21-cu130 by Klaud-Cold · Pull Request #4115 · SemiAnalysisAI/InferenceX

Klaud-Cold · 2026-10-15T15:00:21Z

Goal: Update SGLang image from lmsysorg/sglang:v0.5.16-cu130 to lmsysorg/sglang:v0.5.21-cu130.
Baseline: 2026-08-07 · lmsysorg/sglang:v0.5.16-cu130
Mean latency · Sources: API 1, API 2

Point	Total tok/s/GPU ↑	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
AgentX c1 TP2 EP2 1b1efd	6,521.61	67.04	1,706.35	8.06
AgentX c1 TP4 EP1 365af6	3,544.83	37.76	1,371	6.39
AgentX c4 TP2 EP2 39e3f3	11,543.71	124.88	856.54	8.39
AgentX c4 TP4 EP1 8f5459	6,446.35	71.71	413.24	3.4
AgentX c8 TP2 EP2 09a92c	19,447.07	181.12	1,276.18	6.63
AgentX c8 TP4 EP1 814aa4	10,593.97	104.98	442.79	3.02
AgentX c12 TP2 EP2 64905a	24,761.95	237.82	2,048.25	11.69
AgentX c12 TP4 EP1 3ecb3c	18,270.73	161.17	519.64	3.64
AgentX c16 TP2 EP2 9f6509	13,020.66	117.54	17,826.21	50.45
AgentX c16 TP4 EP1 d139ae	23,354.98	202.14	585.19	4.51
AgentX c20 TP2 EP2 f4b7f4	10,968	101.38	36,228.99	65.67
AgentX c20 TP4 EP1 ca9ded	28,278.84	235.78	652.12	4.57
AgentX c24 TP2 EP2 024cc3	11,360.03	104.77	54,466.35	62.27
AgentX c24 TP4 EP1 aef8ff	33,652.89	278.99	671.34	5.64
AgentX c28 TP2 EP2 fb8b81	11,202.22	98.51	79,270.73	54.79
AgentX c28 TP4 EP1 966a4d	36,430.28	317.6	705.82	6.63
AgentX c32 TP2 EP2 92786e	55,593.45	460.74	10,861.89	11.64
AgentX c32 TP2 EP2 b5ae14	11,688.47	97.97	101,166.92	55.32
AgentX c32 TP4 EP1 eb1cfc	38,164.6	337.49	747.67	7.34
AgentX c34 TP2 EP2 324e27	54,985.67	467.86	12,857.81	12.19
AgentX c36 TP2 EP2 fabb7b	54,272.25	460.86	15,644.09	11.79
AgentX c38 TP2 EP2 67d2b2	53,103.36	484.58	18,908.69	11.48
AgentX c40 TP2 EP2 534e71	52,828.67	484.74	20,878.87	11.14
AgentX c40 TP4 EP1 855ad5	43,541.78	414.76	805.71	8.75
AgentX c44 TP2 EP2 3e4349	51,691.36	469.74	24,197.05	11.61
AgentX c48 TP2 EP2 e136fc	51,404.79	471.39	29,628.26	11.93
AgentX c48 TP4 EP1 12039d	45,376.32	453.76	1,042.83	12.98
AgentX c52 TP2 EP2 980634	49,621.92	479.96	36,385.37	12.74
AgentX c56 TP2 EP2 05008f	49,618.19	477.83	41,409.75	12.54
AgentX c56 TP4 EP1 73dee6	46,211.39	475.98	1,271.15	18.35
AgentX c60 TP4 EP1 d743be	45,804.97	479.18	1,447.1	22.09
AgentX c64 TP4 EP1 aa6d8a	43,570.02	457.23	1,578.06	28.69
AgentX c68 TP4 EP1 e800a7	40,130.06	428.86	1,763.95	39.59
AgentX c72 TP4 EP1 ed5057	36,861.16	402.37	2,233.66	45.26

Eval: N/A

中文

**目标：**将 SGLang 镜像从 lmsysorg/sglang:v0.5.16-cu130 更新至 lmsysorg/sglang:v0.5.21-cu130。
**基线：**2026-08-07 · lmsysorg/sglang:v0.5.16-cu130
平均延迟 · 来源： API 1, API 2；数值及异常说明见上表。

将 Qwen3.5 FP8 B300 SGLang AgentX MTP 镜像更新至 v0.5.21-cu130（按 digest 固定）。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-15T15:00:32Z

Thanks for the contribution!

Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

**审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
**PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
**合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

Klaud-Cold · 2026-10-15T15:02:30Z

Initial attempt · Failed · Run 38756649823 / attempt 1 · 2026-10-15 15:11 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 214026f9fd9a · Mean latency
Change: Pin the SGLang v0.5.21-cu130 image by digest with no recipe changes, because the used server flags, NEXTN, HiCache options, parsers and environment variables keep their semantics between the v0.5.16 and v0.5.21 tags, and the launcher already maps cuda-graph-max-bs to cuda-graph-max-bs-decode.

Point	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
AgentX c1 TP2 EP2 1b1efd	N/A	N/A	N/A
AgentX c1 TP4 EP1 365af6	N/A	N/A	N/A
AgentX c32 TP2 EP2 92786e	N/A	N/A	N/A

Note: AgentX c1 TP2 EP2 1b1efd: failed.
Note: All rows: request errors unavailable.
Note: All rows: Δ N/A: point unavailable.
Note: AgentX c1 TP4 EP1 365af6, AgentX c32 TP2 EP2 92786e: cancelled.

Next: Rename cuda-graph-max-bs to cuda-graph-max-bs-decode in the recipe, because v0.5.21 removed the deprecated alias and argparse rejects the ambiguous prefix, then rerun smoke.

中文

初次尝试 · 失败 · Run 38756649823 / attempt 1 · 2026-10-15 15:11 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 214026f9fd9a · 平均延迟
**变更：**按 digest 固定 SGLang v0.5.21-cu130 镜像，不修改配方，因为 v0.5.16 与 v0.5.21 标签之间所用的服务参数、NEXTN、HiCache 选项、解析器和环境变量语义保持不变，且启动器已将 cuda-graph-max-bs 映射为 cuda-graph-max-bs-decode。实测数值及异常说明见上表。
**下一步：**在配方中将 cuda-graph-max-bs 改为 cuda-graph-max-bs-decode，因为 v0.5.21 移除了该弃用别名且 argparse 拒绝有歧义的前缀，然后重新运行冒烟测试。

… MTP SGLang v0.5.21 removed the deprecated --cuda-graph-max-bs alias, so the prefix is now ambiguous between the decode and prefill flags. SGLang v0.5.21 移除了已弃用的 --cuda-graph-max-bs 别名，该前缀现在会在 decode 与 prefill 参数之间产生歧义，因此改用 cuda-graph-max-bs-decode。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Klaud-Cold · 2026-10-15T15:12:09Z

Repair 1/5 · Passed · Run 38758181864 / attempt 1 · 2026-10-15 17:00 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 214026f9fd9a · Mean latency
Change: Rename cuda-graph-max-bs to cuda-graph-max-bs-decode in the B300 FP8 agentic recipe, matching the v0.5.16 alias target, because v0.5.21 removed the alias.

Point	Output tok/s/GPU ↑	TTFT ms ↓	TPOT ms ↓
AgentX c32 TP2 EP2 92786e	462.74 (N/A)	10,855.25 (N/A)	11.31 (N/A)
AgentX c1 TP4 EP1 365af6	29.86 (N/A)	2,428.14 (N/A)	13.14 (N/A)
AgentX c1 TP2 EP2 1b1efd	50.63 (N/A)	3,162.02 (N/A)	15.3 (N/A)

Note: All rows: Δ N/A: dataset/scenario/statistic mismatch.

Eval	Score ↑	Samples
gsm8k/em_strict · c72	97.8% (N/A)	N/A/1,319 (old/new)

Note: All rows: Δ N/A: no matched eval baseline.

Next: Append the perf-changelog entry, generate the final matrix and run the full sweep; c1 throughput is below the baseline, while TP2 HiCache c32 is on par.

中文

修复 1/5 · 已通过 · Run 38758181864 / attempt 1 · 2026-10-15 17:00 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 214026f9fd9a · 平均延迟
**变更：**在 B300 FP8 agentic 配方中将 cuda-graph-max-bs 改为 cuda-graph-max-bs-decode（即 v0.5.16 别名指向的参数），因为 v0.5.21 移除了该别名。实测数值及异常说明见上表。
**下一步：**追加 perf-changelog 条目，生成最终矩阵并运行完整扫描；c1 吞吐低于基线，TP2 HiCache c32 与基线持平。

…e update 为 Qwen3.5 FP8 B300 SGLang AgentX MTP 镜像更新追加 perf-changelog 条目。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

合并 main 分支并在 perf-changelog 末尾保留本 PR 条目。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Klaud-Cold · 2026-10-15T17:13:52Z

Final full sweep · Running · Run 38775476750 / attempt 1 · 2026-10-15 17:13 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 95a31953c658
Change: Run the full fail-fast sweep for all 34 baseline points with the v0.5.21-cu130 image and the cuda-graph-max-bs-decode rename after merging main.

Next: Wait for every sweep job and its artifacts, then run finish.

中文

最终完整 sweep · 运行中 · Run 38775476750 / attempt 1 · 2026-10-15 17:13 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 95a31953c658
**变更：**合并 main 后，使用 v0.5.21-cu130 镜像和 cuda-graph-max-bs-decode 重命名对全部 34 个基线点运行完整 fail-fast 扫描。实测数值及异常说明见上表。
**下一步：**等待所有扫描任务及其产物完成，然后运行 finish。

Klaud-Cold · 2026-10-15T17:14:20Z

Final full sweep · Running · Run 38775545421 / attempt 1 · 2026-10-15 20:00 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 95a31953c658
Change: Run the full fail-fast sweep for all 34 baseline points with the v0.5.21-cu130 image and the cuda-graph-max-bs-decode rename after merging main.

Next: 9 of 34 benchmarks have passed with no failures so far; the remaining jobs are queued for b300-dsxe runners, so keep waiting.

中文

最终完整 sweep · 运行中 · Run 38775545421 / attempt 1 · 2026-10-15 20:00 UTC
lmsysorg/sglang:v0.5.21-cu130@sha256:b1259f3ea3275f66237c498ea388919729018bc9f01c3d638391e06e2cf3f469 · 95a31953c658
**变更：**合并 main 后，使用 v0.5.21-cu130 镜像和 cuda-graph-max-bs-decode 重命名对全部 34 个基线点运行完整 fail-fast 扫描。实测数值及异常说明见上表。
**下一步：**目前 34 个基准测试中已有 9 个通过且无失败；其余任务正在排队等待 b300-dsxe runner，继续等待。

github-actions · 2026-10-16T02:56:29Z

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38775545421

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38775545421

Klaud-Cold · 2026-10-16T20:17:03Z

unexpected-error · Cleanup pending. Stop and confirm owned runs before closing.

中文

unexpected-error · 清理待完成。先停止并确认自有运行结束，再关闭 PR。

Klaud-Cold · 2026-10-16T20:17:20Z

unexpected-error · Repairs: unknown · Runs: 38756649823, 38758181864, 38775476750, 38775545421
All owned runs ended. PR closed; branch deleted for retry.

中文

unexpected-error · 修复次数：unknown · 运行：38756649823, 38758181864, 38775476750, 38775545421
所有自有运行均已结束。PR 已关闭；分支已删除，可重新尝试。

chore(qwen3.5): update B300 FP8 SGLang AgentX MTP image to v0.5.21-cu130

64ce92e

将 Qwen3.5 FP8 B300 SGLang AgentX MTP 镜像更新至 v0.5.21-cu130（按 digest 固定）。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-project-automation Bot added this to InferenceX Board Oct 15, 2026

chore(perf-changelog): record Qwen3.5 FP8 B300 SGLang AgentX MTP imag…

0441d76

…e update 为 Qwen3.5 FP8 B300 SGLang AgentX MTP 镜像更新追加 perf-changelog 条目。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Klaud-Cold added full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Oct 15, 2026

Merge main into klaud/auto-c840d71da0ca7bde-555679de269db36f

95a3195

合并 main 分支并在 perf-changelog 末尾保留本 PR 条目。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Klaud-Cold added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 15, 2026

adibarra mentioned this pull request Oct 15, 2026

fix(klaud): verify older and aggregate baselines, free abandoned work / fix(klaud): 验证旧版与聚合基线并释放放弃的工作 #4137

Merged

Klaud-Cold removed the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 16, 2026

Klaud-Cold closed this Oct 16, 2026

Klaud-Cold deleted the klaud/auto-c840d71da0ca7bde-555679de269db36f branch October 16, 2026 20:17

github-project-automation Bot moved this to Done in InferenceX Board Oct 16, 2026

Navigation Menu

Repository navigation

[Klaud Cold] Update Qwen3.5 FP8 B300 SGLang AgentX MTP image to v0.5.21-cu130 / [Klaud Cold] 将 Qwen3.5 FP8 B300 SGLang AgentX MTP 镜像更新至 v0.5.21-cu130 - #4115

[Klaud Cold] Update Qwen3.5 FP8 B300 SGLang AgentX MTP image to v0.5.21-cu130 / [Klaud Cold] 将 Qwen3.5 FP8 B300 SGLang AgentX MTP 镜像更新至 v0.5.21-cu130#4115
Klaud-Cold wants to merge 4 commits into
mainfrom
klaud/auto-c840d71da0ca7bde-555679de269db36f

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Uh oh!

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Uh oh!

Klaud-Cold commented Oct 15, 2026

Uh oh!

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

Klaud-Cold commented Oct 16, 2026

Uh oh!

Klaud-Cold commented Oct 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Klaud-Cold commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 15, 2026

Uh oh!

Klaud-Cold commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Klaud-Cold commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Klaud-Cold commented Oct 15, 2026

Uh oh!

Klaud-Cold commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

Klaud-Cold commented Oct 16, 2026

Uh oh!

Klaud-Cold commented Oct 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Klaud-Cold commented Oct 15, 2026 •

edited

Loading

Klaud-Cold commented Oct 15, 2026 •

edited

Loading