Repository navigation
[NVIDIA][AgentX] Retune B300 DeepSeek-V4.1-Flash SGLang aggregate curve on 1014 nightly / [NVIDIA][AgentX] 在 1014 nightly 上重新调优 B300 DeepSeek-V4.1-Flash SGLang 聚合曲线 - #4094
Conversation
Repin to nightly-dev-cu13-20261014-684fea8a, enable the fastokens tokenizer backend, and publish eleven points (TP4 C1/4/8/16/20, TP2 C16/32/48, DEP2 C64/256, DEP4 C256). 在 1014 nightly 上重新调优 AgentX 曲线：固定 nightly-dev-cu13-20261014-684fea8a, 启用 fastokens tokenizer 后端，发布十一个点(TP4 C1/4/8/16/20、TP2 C16/32/48、 DEP2 C64/256、DEP4 C256)。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
shyeh25
added
the
full-sweep-fail-fast
|
Claude finished @shyeh25's task in 1m 25s —— View job Review of PR #4094
I found one 🟡 issue, posted inline. The DEP2 and DEP4 rows say Nothing else is blocking:
I couldn't run |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - { tp: 2, ep: 2, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml } | ||
| - { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml } |
There was a problem hiding this comment.
🟡 WARNING: The DEP2 and DEP4 rows are labeled kv-offloading: none, but their recipe variants (override_dep2_c64, override_dep2_c256, override_dep4_c256) turn on SGLang HiCache: enable-hierarchical-cache: true, hicache-ratio: 4/2, hicache-write-policy: write_back. That is CPU DRAM KV offloading.
Why it matters: These three points will be published as if they used no KV offload, even though they rely on host DRAM. That gives the wrong picture against TP-only arms and other vendors. single_node.validate_recipe only cross-checks KV_OFFLOADING when the recipe sets it, so nothing catches the mismatch. Every other HiCache AgentX row in this file is labeled kv-offloading: dram, kv-offload-backend: { name: hicache } (for example, lines 888–889, 1094, 1123 and 1171–1173).
Fix:
| - { tp: 2, ep: 2, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml } | |
| - { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml } | |
| - { tp: 2, ep: 2, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [64, 256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml } | |
| - { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml } |
…RAM offload The DEP2 and DEP4 variants enable SGLang HiCache, so their master rows use kv-offloading dram with the hicache backend. Server arguments are unchanged. DEP2 与 DEP4 配置启用了 SGLang HiCache,因此主配置行改为 kv-offloading dram 并使用 hicache 后端。服务器参数不变。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=38705241139 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=38705241139 |
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline finding on kv-offloading: none vs the new HiCache-enabled DEP arms, I also checked whether the new DEP4 (TP4/EP4) c256 point silently loses its default GSM8K accuracy eval because it ties with DEP2's c256 point on concurrency within the same dp-attn group — this is confirmed intentional: mark_eval_entries in infx/matrix/generate.py explicitly groups single-node agentic eval selection by (model, runner, framework, precision, spec-decoding, dp-attn, image), not by tp/ep, per its own docstring, so only one representative per dp-attn group gets picked. Not a bug.
Extended reasoning...
The diff is a benchmark-config repin (new SGLang nightly image, tokenizer backend) plus a retune of srt-slurm recipe overrides and matching master-config search-space rows for a single agentic benchmark; no security-sensitive surface (auth, crypto, permissions) is touched. A confirmed inline finding already flags a real config-correctness issue (kv-offloading field not updated to reflect newly-enabled HiCache on the new DEP arms), which is reason enough to defer to a human regardless. I additionally traced the GSM8K-eval-skip candidate through infx/matrix/generate.py and found it matches documented, intentional grouping behavior rather than a regression.
| - { tp: 2, ep: 2, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [64, 256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml } | ||
| - { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml } |
There was a problem hiding this comment.
🟡 (optional) The new DEP2/DEP4 arms turn on SGLang HiCache but still declare kv-offloading: none, so operators and dashboards get a silently wrong DRAM-offload budget and report for these points. nvidia-master.yaml:9181-9182 keep kv-offloading: none while agentic.yaml's override_dep2_c64/c256 (lines 228-232) and override_dep4_c256 (lines 326-330) set enable-hierarchical-cache: true. Because kv_offloading != 'dram', agentic_dram_offload_gb() (infx/matrix/generate.py:491-493) returns 0, skipping the dram-utilization capacity check, and results/agentic/init.py:192 will record offload_mode=off and allocated_cpu_dram_gb=0 for jobs that actually reserve host DRAM for KV tiering. Fix: set kv-offloading: dram and kv-offload-backend: {name: hicache} on every search-space row whose recipe enables enable-hierarchical-cache, covering both the DEP2 and DEP4 rows here.
Why this was flagged
Trigger: the matrix generator builds points for the dsv41flash-fp4-b300-sglang-agentic-dspark DEP2 (nvidia-master.yaml:9181) and DEP4 (9182) rows, both kv-offloading: none. Their srt-recipe (agentic.yaml override_dep2_c64/c256, override_dep4_c256) newly sets enable-hierarchical-cache: true and hicache-ratio 4/4/2, i.e. real host-DRAM KV cache tiering. generate.py's agentic_dram_offload_gb (line 491-493) short-circuits to 0 whenever kv_offloading != 'dram', so no DRAM capacity is computed or validated against available-cpu-dram-mib, unlike the dram_utilization path CONFIGS.md:211-212 describes. Base behavior: the dp4_c192 arm this replaces never used HiCache, so kv-offloading: none was accurate there; this PR adds real offload without updating the metadata. Downstream, klaud/reporting.py:868 and results/agentic/init.py:192 will publish offload_mode=off and allocated_cpu_dram_gb=0 for these runs.
Verification: nit. nvidia-master.yaml:9181-9182 declare kv-offloading: none for the new DEP2/DEP4 arms, while agentic.yaml enables HiCache host-DRAM tiering for the same arms: override_dep2_c64 (line 228), override_dep2_c256 (line 278), override_dep4_c256 (line 326). generate.py:491-493 returns 0 whenever kv_offloading != "dram", so results/agentic/init.py:192 reports allocated_cpu_dram_gb = 0 and kv_offloading = none despite host-DRAM KV tiering.
|
/use 38705241139 |
|
@nvpohanh staged run 38705241139. This run remains available across future |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( The commands mirror the launch measured in run 38705241139, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#44654 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 38705241139, pinned by this PR's |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Passing sweep on in-PR commit): PASS — this config is agentic-only (fixed-sequence ✅ Check 2 (Evals): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#44654 is MERGED (2026-10-16T04:45:08Z, ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — all 18 items of the current ✅ Check 6 (Upstream image / ordering): PASS — ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (Architecture hacks): PASS — no ✅ Check 9 (Chat template): PASS — the AIPerf export for the run records endpoint ✅ Check 10 (Engine patches): PASS — the diff adds no patch files, ✅ Check 11 (Golden AL): PASS — ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 13 (Draft precision): PASS — the draft is the DSpark head bundled in ✅ Check 14 (Pareto coverage): PASS — one curve ✅ Check 15 (PR description): PASS — the body's image, fastokens backend, arms, concurrency points, and per-arm settings (lpm/1 request at TP4 C1, trtllm at TP4 C8/C20 and TP2 C48, bounded replay at C16, MegaMoE plus HiCache plus consistent hashing on DEP2/DEP4, smaller HiCache ratio and larger chunks on DEP4) match the pinned recipe. ✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview Framework Recipes table has a DeepSeek-V4.1-Flash row with an SGLang ✅ linking the SGLang cookbook page. ➖ Check 17 (Metrics publication): N/A — direct Assessed commit: |
…glang-agentic-1014-retune Resolve the inferencex-e2e/perf-changelog.yaml append conflict by keeping main's entries and re-appending this PR's entry. [skip-sweep] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nvpohanh
removed
the
full-sweep-fail-fast
3 participants
Retunes
dsv41flash-fp4-b300-sglang-agentic-dspark(B300, SGLang, single-node aggregate, DSpark block 5) onlmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a(multiarch digest pinned) with the fastokens tokenizer backend, and replaces the published points with a new set:中文
在
lmsysorg/sglang:nightly-dev-cu13-20261014-684fea8a(固定 multiarch digest)上重新调优dsv41flash-fp4-b300-sglang-agentic-dspark(B300、SGLang、单节点聚合、DSpark block 5),启用 fastokens tokenizer 后端，并以新的点位替换原有点位:TP4/EP1 并发 1、4、8、16、20;TP2/EP1 并发 16、32、48;DEP2(TP2/EP2、DP attention 2)并发 64、256;DEP4(TP4/EP4、DP attention 4)并发 256。AI model disclosure
Prepared with Claude Code using claude-opus-5-5 (recipe and config changes, validation, PR text).
🤖 Generated with Claude Code