You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b9efee3
Browse filesBrowse the repository at this point in the historyBrowse files
|`--tensor-parallel-size 1 --data-parallel-size 4 --enable-expert-parallel`| DP=4 + EP across the 4 GPUs (TP=1) |
132
132
|`--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}'`| Single-node DEP compilation config from the upstream recipe |
133
+
|`--no-enable-flashinfer-autotune`| Skip per-shape FlashInfer autotuning at startup; required on dsv4 for correct accuracy |
133
134
|`--max-num-seqs 256`| Concurrency cap |
134
135
135
136
### vLLM GB200 (`vllm/agg_gb200/deploy.yaml`)
@@ -140,6 +141,7 @@ Same OpenAI-renderer wiring as the B200 variant; differences below come from the
140
141
|---|---|
141
142
|`--tensor-parallel-size 4 --enable-expert-parallel`|**TP=4 + EP** across the 4 GPUs of the NVL4 tray (DP dropped — the GB200 tray's intra-tray NVLink makes TP attractive for this size class) |
142
143
|`--moe-backend deep_gemm_mega_moe`| DeepGEMM "mega MoE" kernel — the optimized FP8 MoE path for V4 expert routing on Blackwell |
144
+
|`--no-enable-flashinfer-autotune`| Skip per-shape FlashInfer autotuning at startup; required on dsv4 for correct accuracy |
143
145
|`NCCL_NVLS_ENABLE=1`, `NCCL_P2P_LEVEL=NVL`, `VLLM_USE_NCCL_SYMM_MEM=1`| Enable NVLink Sharp (NVLS) multicast for one-shot all-reduce on the tray |
@@ -243,14 +245,15 @@ If `tool_calls` is missing and raw tool-call markers appear in `content`, confir
243
245
244
246
### vLLM-specific
245
247
246
-
-**Prebuilt images.** Both `vllm/agg_b200/deploy.yaml` and `vllm/agg_gb200/deploy.yaml` reference `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.0-deepseek-v4-cuda13-dev.2` (multi-arch). To rebuild from source (custom Dynamo branch, different vLLM base, etc.), see [`<repo_root>/container/README.md`](../../../container/README.md).
248
+
-**Prebuilt images.** Both `vllm/agg_b200/deploy.yaml` and `vllm/agg_gb200/deploy.yaml` reference `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.0-deepseek-v4-cuda13-dev.3` (multi-arch). To rebuild from source (custom Dynamo branch, different vLLM base, etc.), see [`<repo_root>/container/README.md`](../../../container/README.md).
247
249
-**Engine-ready timeout.**`VLLM_ENGINE_READY_TIMEOUT_S=3600` matches the startup probe budget on both variants.
248
250
-**DP stability (B200 only).**`VLLM_RANDOMIZE_DP_DUMMY_INPUTS=1` and `VLLM_SKIP_P2P_CHECK=1` mirror the DeepSeek-R1 vLLM recipe and stabilize DP dummy inputs. The GB200 variant uses TP (no DP), so `VLLM_RANDOMIZE_DP_DUMMY_INPUTS` is not set.
251
+
-**FlashInfer autotune.**`--no-enable-flashinfer-autotune` skips per-shape FlashInfer autotuning at startup and is set on both vLLM variants. Required on dsv4: the autotuner currently produces tunings that regress GSM8k accuracy. Skipping it also shortens first-launch warmup.
249
252
-**FlashInfer TRT-LLM allreduce on GB200.** You may see a non-fatal startup warning `Failed to initialize FlashInfer Allreduce norm fusion workspace ... Flashinfer allreduce-norm fusion will be disabled`. vLLM falls back to a non-fused allreduce + RMSNorm; correctness is unaffected. To enable the fused kernel, set the compilation pass: `--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"],"pass_config":{"fuse_allreduce_rms":true}}'`.
250
253
251
254
### SGLang-specific
252
255
253
-
-**Prebuilt images.**`sglang/agg/deploy.yaml` references `nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.2.0-deepseek-v4-cuda12-dev.2` and `sglang/agg-gb200/deploy.yaml` references the arm64 sibling `nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.2.0-deepseek-v4-cuda13-dev.2`. To rebuild either (custom Dynamo branch, different SGLang base, etc.), see [`recipes/deepseek-v4/container/README.md`](../container/README.md).
256
+
-**Prebuilt images.**`sglang/agg/deploy.yaml` references `nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.2.0-deepseek-v4-cuda12-dev.3` and `sglang/agg-gb200/deploy.yaml` references the arm64 sibling `nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.2.0-deepseek-v4-cuda13-dev.3`. To rebuild either (custom Dynamo branch, different SGLang base, etc.), see [`recipes/deepseek-v4/container/README.md`](../container/README.md).
254
257
-**DeepGEMM / FlashInfer warmup.**`SGLANG_JIT_DEEPGEMM_PRECOMPILE=0` + `SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1` skip the slow precompile and use the fast warmup path. `--disable-flashinfer-autotune` skips per-shape FlashInfer autotuning at startup; the dsv4 base ships pre-tuned defaults.
255
258
-**NCCL / Gloo.**`NCCL_CUMEM_ENABLE=1` is set for V4 NCCL collectives on Blackwell. `GLOO_SOCKET_IFNAME=eth0` pins Gloo to the standard pod interface.
0 commit comments