[Docs] Cookbook: AgentX launches for Qwen3.5 / Qwen3.8-Flash-Next; refresh GLM-5.2, DeepSeek-V4, DeepSeek-V4.1

docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx

@@ -232,6 +232,21 @@ This section provides deployment configurations optimized for different hardware
 
 - **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
 
+<a id="agentic-long-context-deployment" />
+
+### 3.3 Agentic Long-Context Deployment
+
+Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments of Qwen3.5-397B-A17B, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.
+
+- **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
+- **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
+- **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.
+
+import { AgentX } from "/src/snippets/_agentx.jsx";
+import { agentx } from "/src/snippets/agentx/Qwen/qwen3.5.jsx";
+
+<AgentX data={agentx} />
+
 ## 4. Model Invocation
 
 **NVIDIA:**
@@ -1023,44 +1038,6 @@ Max ITL (ms):                            1220.68
 ==================================================
 ```
 
-#### 5.2.3 Agentic Long-Context with HiCache DRAM Offload (H200 FP8, MTP)
-
-For agentic workloads, here is how to enable HiCache and MTP.
-
-Container image (pinned for reproducibility):
-`lmsysorg/sglang:nightly-dev-cu13-20260815-a5ba081f`.
-
-Server Launch Command:
-```bash Command
-python3 -m sglang.launch_server \
-  --model-path Qwen/Qwen3.5-397B-A17B-FP8 \
-  --served-model-name Qwen/Qwen3.5-397B-A17B-FP8 \
-  --trust-remote-code \
-  --tensor-parallel-size 8 \
-  --data-parallel-size 1 \
-  --expert-parallel-size 1 \
-  --quantization fp8 \
-  --kv-cache-dtype fp8_e4m3 \
-  --mamba-ssm-dtype bfloat16 \
-  --attention-backend flashinfer \
-  --enable-flashinfer-allreduce-fusion \
-  --mem-fraction-static 0.8 \
-  --stream-interval 50 \
-  --scheduler-recv-interval 10 \
-  --tokenizer-worker-num 6 \
-  --enable-metrics \
-  --speculative-algorithm EAGLE \
-  --speculative-num-steps 3 \
-  --speculative-eagle-topk 1 \
-  --speculative-num-draft-tokens 4 \
-  --page-size 64 \
-  --enable-hierarchical-cache \
-  --hicache-size 77 \
-  --hicache-io-backend kernel \
-  --hicache-mem-layout page_first \
-  --hicache-write-policy write_through_selective
-```
-
 ### 5.3 Vision Speed Benchmark
 
 We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. Each request has 128 input tokens, two 720p images, and 1024 output tokens.

docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx

@@ -26,7 +26,7 @@ import { Deployment } from "/src/snippets/_deployment.jsx";
 import { config }     from "/src/snippets/configs/Qwen/qwen3.8-flash-next.jsx";
 import { benchmarks } from "/src/snippets/configs/Qwen/qwen3.8-flash-next-benchmarks.jsx";
 
-<Deployment config={config} benchmarks={benchmarks} />
+<Deployment config={config} benchmarks={benchmarks} agenticLink="agentic-long-context-deployment" />
 
 <a id="spark-note" />
 
@@ -69,6 +69,19 @@ import { Playground } from "/src/snippets/_playground.jsx";
 
 <Playground config={config} />
 
+## Agentic Long-Context Deployment
+
+Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.
+
+- **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
+- **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
+- **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.
+
+import { AgentX } from "/src/snippets/_agentx.jsx";
+import { agentx } from "/src/snippets/agentx/Qwen/qwen3.8-flash-next.jsx";
+
+<AgentX data={agentx} />
+
 ## 1. Model Introduction
 
 **Qwen3.8-Flash-Next** is a multimodal Mixture-of-Experts model released as an early preview of the architecture Qwen4 is being built on — the same role Qwen3-Next played for Qwen3.5, whose hybrid Gated DeltaNet + Gated Attention design then carried through the Qwen3.5, Qwen3.6, Qwen3.7 and Qwen3.8 series. Qwen is publishing the architectural changes ahead of the full Qwen4 family so the community can evaluate them independently.
@@ -218,45 +231,3 @@ Pending update — a sample transcript will be added here once the weights are p
 ```
 
 </Accordion>
-
-### 2.3 GB300 long-context serving with HiCache and MTP
-
-Use this single-GPU, aggregated TP1/EP1 configuration for long-context, multi-turn workloads on GB300. It combines the NVIDIA NVFP4 checkpoint's embedded MTP head, CPU DRAM KV offload through HiCache, and pinned-memory PLE embedding offload.
-
-This serving configuration comes from [InferenceX #3816](https://github.com/SemiAnalysisAI/InferenceX/pull/3816). [Run 37714835620, attempt 1](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37714835620) passed concurrency 1, 2, 4, 8, 16, 32, 48 and 64 on commit `e67ca6fc9ecf4cd57f27654bda8a3e9059923820`. Its [matching GSM8K eval](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/37714835620/job/113109083376) used real verification and scored 97.35% on 1,319 questions.
-
-The throughput sweep used InferenceX's simulated acceptance; the generated command uses normal MTP verification and does not claim those throughput numbers for production traffic.
-
-#### Prerequisites and memory budget
-
-- Use a Linux ARM64 GB300 host with Docker, NVIDIA Container Toolkit, and a driver compatible with the image's CUDA 13.0 runtime. See [Docker installation](/docs/get-started/install#method-3-using-docker).
-- Reserve one GB300 GPU. The recipe keeps TP, PP, DP and EP at 1.
-- Budget 224 GiB of CPU memory for the combined target/draft KV and Mamba HiCache host pool, plus approximately 48 GiB for the offloaded FP8 PLE table and additional headroom for checkpoint loading and host processes. `--hicache-size` sets the combined cache budget; it is separate from PLE weight offload. Keep sufficient free host memory for both.
-- The Docker command mounts your Hugging Face cache to retain downloaded model files across restarts. The image digest and both target/draft checkpoint revisions are pinned to the validated configuration.
-
-#### Generate the launch command
-
-In the [Deployment panel](#deployment), select **GB300 → NVFP4 (NVDA) → Low Latency → Single Node**, and leave **PLE Offload** at **Auto** or **On**. The matched cell includes all five serving environment variables and the full argument set, including ReplaySSM, HiCache and pinned PLE offload. Choose **Docker** to use the pinned image, or **Python** for a matching local installation. Changing PLE Offload to **Off** produces an unverified configuration.
-
-The Docker wrapper comes from the shared command builder and is adapted from the validated Slurm/Enroot launch; it has not been run separately on a GB300 host. Use the command generated by the panel rather than a separate static copy.
-
-<Accordion title="Reference startup output">
-
-The validated run loaded the target and embedded draft through the stock image's mixed-precision loader. This excerpt is from the real-verification eval; timings and memory availability vary by host.
-
-```text Output
-[2026-10-08 02:22:36] Load weight end. elapsed=1011.56 s, type=Qwen4ExpForConditionalGeneration, quant=modelopt_mixed, quant_algo=MIXED_PRECISION, avail mem=202.03 GB, mem usage=73.64 GB.
-[2026-10-08 02:23:03] Load weight end. elapsed=25.97 s, type=Qwen4ExpForCausalLMMTP, quant=modelopt_mixed, quant_algo=MIXED_PRECISION, avail mem=197.09 GB, mem usage=4.89 GB.
-```
-
-</Accordion>
-
-#### Serving choices
-
-- **Draft precision:** revision `fc694b54` stores the MTP routed experts in block-scaled FP8 and the rest of the head in BF16. The pinned image, built from SGLang `e00930c5`, preserves this per-layer `modelopt_mixed` configuration. The draft uses the same checkpoint revision as the target. `SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0` disables the optional draft precision conversion path; target and draft KV caches stay in BF16.
-- **Hybrid attention:** the recipe selects FlashInfer for linear-attention prefill/decode, Triton for linear verification, and `trtllm_mha` as the attention backend. ReplaySSM speculation, `extra_buffer_lazy`, a tracking interval of 256 and 480 Mamba cache slots retain the qualified state-cache configuration.
-- **Admission and graphs:** the server admits up to 96 requests, uses 16,384-token prefill chunks, and captures full decode graphs at the listed batch sizes up to 96. Prefill graphs remain disabled. The qualification varied client concurrency while retaining these server settings.
-- **CPU offload:** HiCache uses the `kernel` I/O backend, `page_first` layout and `write_back` policy. PLE uses a separate pinned-memory embedding table. See [HiCache design](/docs/advanced_features/hicache_design) for the cache tiers and policies.
-- **API and telemetry:** send requests through the chat-completions API at `http://localhost:30000/v1`, using model name `nvidia/Qwen3.8-Flash-Next-NVFP4`. `--reasoning-parser auto` separates reasoning from answer content. Engine metrics and cache reporting remain enabled.
-
-The generated command omits arguments that match the pinned image's defaults, including TP/PP/DP/EP=1, the inherited served-model name and draft MoE backend, full decode graphs, and default Mamba tracking, HiCache layout, PLE backend, tokenizer and scheduler settings. It retains the qualified lane's non-default serving settings. InferenceX-only trace loading, simulated-acceptance controls, client concurrency overrides, result collection and Slurm allocation settings are excluded.

docs/src/snippets/_agentx.jsx

@@ -191,8 +191,10 @@ export const AgentX = ({ data }) => {
   };
   const noteText = (S) => ({
     "rename-dp-attention": <>Flags are mirrored from the submission image: <code style={S.code}>--enable-dp-attention</code> with <code style={S.code}>--dp-size</code> / <code style={S.code}>--data-parallel-size</code>. Newer SGLang deprecates them in favor of <code style={S.code}>--attn-dp-size</code> (<a style={S.a} href="https://github.com/sgl-project/sglang/pull/41818">#41818</a>).</>,
+    "rename-mamba-strategy": <>Flags are mirrored from the submission image: <code style={S.code}>--mamba-scheduler-strategy</code> is the older spelling of <code style={S.code}>--mamba-radix-cache-strategy</code> (<a style={S.a} href="https://github.com/sgl-project/sglang/pull/28151">#28151</a>). Current SGLang accepts only the new name.</>,
     "hicache-size": <><code style={S.code}>--hicache-size</code> is host DRAM per rank in GB, sized to SemiAnalysis's nodes. Scale it to your host memory.</>,
     efa: <><code style={S.code}>MOONCAKE_PROTOCOL=efa</code> reflects SemiAnalysis's AWS-EFA B300 cluster. Use <code style={S.code}>rdma</code> on InfiniBand.</>,
+    "nixl-libfabric": <><code style={S.code}>SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC</code> reflects SemiAnalysis's AWS-EFA B300 cluster. On InfiniBand, leave it unset to use NIXL's default UCX backend.</>,
     "ib-devices": <>Set <code style={S.code}>IB_DEVICES</code> to your node's RDMA NICs (comma-separated, e.g. <code style={S.code}>mlx5_0,mlx5_1</code>).</>,
     "mooncake-segment": <><code style={S.code}>MOONCAKE_GLOBAL_SEGMENT_SIZE</code> is the host DRAM each worker process contributes to the Mooncake store. Size it to your host memory.</>,
   });
@@ -202,8 +204,10 @@ export const AgentX = ({ data }) => {
     const text = JSON.stringify(blocks);
     return [
       names.has("--enable-dp-attention") && "rename-dp-attention",
+      names.has("--mamba-scheduler-strategy") && "rename-mamba-strategy",
       names.has("--hicache-size") && "hicache-size",
       env.MOONCAKE_PROTOCOL === "efa" && "efa",
+      env.SGLANG_DISAGGREGATION_NIXL_BACKEND === "LIBFABRIC" && "nixl-libfabric",
       env.MOONCAKE_GLOBAL_SEGMENT_SIZE !== undefined && "mooncake-segment",
       text.includes("$IB_DEVICES") && "ib-devices",
     ].filter(Boolean);
@@ -582,7 +586,7 @@ export const AgentX = ({ data }) => {
           )}
           {notesFor(blocks).map((n) => noteText(S)[n] && <li key={n}>{noteText(S)[n]}</li>)}
           {routerData.install.map((cmd) => (
-            <li key={cmd}>{cmd.startsWith("#") ? <>Dynamo: {cmd.replace(/^# /, "")}</> : <>Dynamo install used by SA: <code style={S.code}>{cmd}</code></>}</li>
+            <li key={cmd}>{cmd.startsWith("#") ? <>Dynamo: {cmd.replace(/^# /, "")}</> : router === cell.submitted ? <>Dynamo install used by SA: <code style={S.code}>{cmd}</code></> : <>This image does not bundle <code style={S.code}>ai-dynamo</code>. Install the release built for its SGLang version first: <code style={S.code}>{cmd}</code></>}</li>
           ))}
           {router === "dynamo" && !routerData.install.length && <li>Dynamo: the <code style={S.code}>ai-dynamo</code> release bundled in the image.</li>}
           {cell.setup.map((s, i) => (