Skip to content

docs(cookbook): add Qwen3.8 GB300 long-context recipe - #43159

Merged
Fridge003 merged 3 commits into
sgl-project:mainfrom
Ankur-singh:docs/qwen38-gb300-agentx-3816
Oct 8, 2026
Merged

Fridge003 merged 3 commits into
sgl-project:mainfrom
Ankur-singh:docs/qwen38-gb300-agentx-3816

Conversation

@Ankur-singh

@Ankur-singh Ankur-singh commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Integrate the single-GPU GB300 long-context serving configuration qualified by InferenceX #3816 into the existing Qwen3.8-Flash-Next command builder.

Replace the unverified GB300 / NVFP4 (NVDA) / Low Latency cell with the qualified environment and serving arguments: embedded MTP, ReplaySSM, HiCache DRAM offload, and pinned-memory PLE offload. Pin the stock SGLang image digest and target/draft checkpoint revisions. Keep the memory budget, startup output and qualification notes in the MDX page, and remove its separate static launch command. PLE Auto/On retains the qualified configuration; Off removes the offload flags and marks the modified configuration unverified.

Omit 15 redundant arguments after verifying the pinned SGLang build's CLI defaults and model/backend inheritance. The generated command retains the qualified non-default settings, image digest, checkpoint revisions and serving environment.

This PR changes the per-model config and its MDX page. It can merge independently of #43083.

Run 37714835620, attempt 1 passed all eight concurrency points and the matching real-verification GSM8K eval: 97.35% on 1,319 questions. The generated command uses normal MTP verification; simulated-acceptance throughput is not presented as production performance. The builder's Docker wrapper is adapted from the validated Slurm/Enroot launch and has not been run separately on a GB300 host.

Qualification evidence
  • Source head: e67ca6fc9ecf4cd57f27654bda8a3e9059923820 in InferenceX How to contribute an optimized R1 operator in SGlang? #3816.
  • Image: lmsysorg/sglang:v0.5.21-cu130@sha256:1f176b3a1dd1205d2a462ab80ea9fc17769a32e53cb6f3c690efa2dc2c384977, built from SGLang e00930c5489053f26d86b179cee0d087f846acbb.
  • Target and embedded draft: nvidia/Qwen3.8-Flash-Next-NVFP4 revision fc694b54fb0174e0913e6adf86691ef85a4ead47.
  • AgentX concurrency: 1, 2, 4, 8, 16, 32, 48, 64. Matching GSM8K eval job.
  • Benchmark-only trace loading, simulated acceptance, client concurrency overrides, result collection, and Slurm allocation controls are excluded from the serving recipe.
Contributor checklist
  • Follow the native cookbook config and command-builder conventions.
  • Run the applicable pre-commit checks.
  • Update documentation; runtime implementation is unchanged.
  • Unit tests: not applicable to this documentation-only change.
  • Link the existing accuracy and workload qualification evidence, with the simulated-acceptance limitation stated explicitly.
中文

将 InferenceX #3816 已验证的 GB300 单 GPU 长上下文配置集成到现有 Qwen3.8-Flash-Next 命令生成器。

用已验证的环境变量及非默认服务端参数替换尚未验证的 GB300 / NVFP4 (NVDA) / Low Latency 配置单元,保留内置 MTP、ReplaySSM、HiCache DRAM 卸载及固定内存 PLE 卸载,并固定官方镜像摘要和目标及草稿检查点版本。MDX 页面保留内存预算、真实启动日志与验证说明,移除独立的静态启动命令。PLE 选择 Auto 或 On 时保留已验证配置;选择 Off 时移除卸载参数,并将修改后的配置标记为未验证。

根据固定 SGLang 版本的命令行默认值以及模型和后端参数继承规则,省略 15 个重复参数。生成的命令仍保留已验证的非默认设置、镜像摘要、检查点版本和服务环境变量。

本 PR 仅修改该模型的配置文件与 MDX 页面,可独立于 #43083 合并。

运行 37714835620,attempt 1 的八个并发点及对应 GSM8K 评测均通过;使用真实验证的 GSM8K 在 1,319 道题上取得 97.35%。生成的命令使用正常 MTP 验证,不将模拟接受率测试的吞吐量视为生产流量性能。命令生成器的 Docker 包装方式改编自已验证的 Slurm/Enroot 启动方式,尚未在 GB300 主机上单独执行。

AI model disclosure

Prepared with an AI assistant identified by the runtime as the GPT-6 family. The exact model/version could not be verified. Role: source/evidence review, recipe authoring and feedback revisions, local documentation validation, and PR preparation. No delegated agents contributed.

由运行时标识为 GPT-6 系列的 AI 助手协助完成;无法验证精确的模型及版本。工作包括来源与验证证据审查、配方编写与反馈修改、本地文档验证及 PR 准备。没有委派其他代理参与。


CI States

Latest PR Test (Base): ✅ Run #37835395177
Latest PR Test (Extra): ❌ Run #37835394473
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

为 Qwen3.8-Flash-Next 添加独立的 GB300 单 GPU 长上下文配方,保留已验证的环境变量、服务端参数与镜像及检查点版本,并说明 HiCache、PLE 卸载和验证证据。

Signed-off-by: Ankur-singh <ankusingh@nvidia.com>
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 8, 2026
Comment thread docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx Outdated
将已验证的 GB300 NVFP4 长上下文配置移入现有命令生成器,保留完整环境变量、服务参数与镜像和检查点版本;移除重复的静态启动命令,并正确处理 PLE 卸载切换及验证状态。

Signed-off-by: Ankur-singh <ankusingh@nvidia.com>
对照固定 SGLang 版本的默认值和参数继承规则,删除 GB300 配方中 15 个重复参数,保留已验证的非默认服务设置。

Signed-off-by: Ankur-singh <ankusingh@nvidia.com>
@Ankur-singh

Copy link
Copy Markdown
Contributor Author

@faradawn address the comment above plus some clean-up. Feel free to push directly to the PR. Appreciate your help and feedback.

@Fridge003 Fridge003 changed the title docs(cookbook): add Qwen3.8 GB300 long-context recipe / docs(cookbook): 新增 Qwen3.8 GB300 长上下文部署配方 docs(cookbook): add Qwen3.8 GB300 long-context recipe Oct 8, 2026
@Fridge003
Fridge003 merged commit 2b94cb4 into sgl-project:main Oct 8, 2026
104 of 108 checks passed
nvpohanh added a commit to nvpohanh/sglang that referenced this pull request Oct 9, 2026
…fresh GLM-5.2, DeepSeek-V4, DeepSeek-V4.1

Mirror every NVIDIA sglang / dynamo-sglang AgentX lane on SemiAnalysis
InferenceX main (ceb50ba2) into the Agentic Long-Context Deployment
explorer added in sgl-project#42726, with these open InferenceX PRs overlaid: sgl-project#3816
(Qwen3.8 GB300 SGLang), sgl-project#3840 (Qwen3.5 FP8 GB300 Dynamo+SGLang), sgl-project#2978
(Qwen3.5 B300 Dynamo+SGLang), sgl-project#3817 (DSV4 B300 Dynamo+SGLang), sgl-project#3606
(DSV4 B300 SGLang), sgl-project#3701 (DSV4 GB200 Dynamo+SGLang), sgl-project#3782 (DSV4.1 GB300
SGLang), sgl-project#3744 (DSV4.1 B300 SGLang) and sgl-project#3836 (GLM-5.2 FP8 B300 SGLang).
Lanes of PRs without an authorized /use run are shown as in progress.

- Qwen3.5: new "3.3 Agentic Long-Context Deployment" subsection (the page
  keeps its legacy generator); the hand-written 5.2.3 AgentX HiCache recipe
  is removed.
- Qwen3.8-Flash-Next: new "Agentic Long-Context Deployment" section after
  the Playground with the Deployment panel's agenticLink footer. The static
  "2.3 GB300 long-context serving with HiCache and MTP" section added in
  sgl-project#43159 is removed (its InferenceX sgl-project#3816 configuration is now in the
  explorer), and the Deployment panel warning that linked to it now links
  to the explorer section.
- GLM-5.2 / DeepSeek-V4 / DeepSeek-V4.1: regenerated data; commands change
  only for the overlaid lanes and one derived HiCache variant.
- _agentx.jsx: notes for --mamba-scheduler-strategy and
  SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC, and wording for derived
  Dynamo variants whose image does not bundle ai-dynamo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants