Skip to content

Commit f516173

Browse files
Juntian777claudepeiliicquil11
authored
Add four GB300 DeepSeek V4.1 Flash Dynamo vLLM AgentX points (#3941)
* Add GB300 DeepSeek-V4.1-Flash Dynamo + vLLM disaggregated AgentX recipes Add 8P1D, 6P1D and 5P1D points (DEP2 prefill, DEP16 decode) behind the Dynamo KV router with native vLLM sidecars. 新增 GB300 DeepSeek-V4.1-Flash Dynamo + vLLM 分离式 AgentX 配方：8P1D、6P1D、5P1D 三个点（DEP2 prefill、DEP16 decode），使用 Dynamo KV 路由与原生 vLLM sidecar。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: drop setup script, enable prefill autotune; GB300 single-node arm on the new image Drop the vllm-container-deps.sh setup script (the image already has numactl and msgpack), enable FlashInfer autotune on prefill, move the GB300 single-node arm to vllm/vllm-openai:nightly-aarch64-dev-20261012-59e75f4 with its TP4 points at conc 1 and 8, and restore a historical changelog pr-link that the previous commit changed by mistake. 处理评审意见：删除 vllm-container-deps.sh 安装脚本（镜像已自带 numactl 和 msgpack），prefill 开启 FlashInfer autotune；GB300 单节点配置改用 vllm/vllm-openai:nightly-aarch64-dev-20261012-59e75f4，TP4 只跑 conc 1 和 8；并恢复上一个提交误改的一条历史 changelog pr-link。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Move the TP4 points to a Dynamo aggregated key; leave the single-node vLLM arm unchanged Add dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg (one TP4 worker with a native Dynamo sidecar, conc 1 and 8) and restore the GB300 single-node vLLM key and recipe to main. 将 TP4 点移到新的 Dynamo 聚合配置项，单节点 vLLM 配置保持不变：新增 dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg（单个 TP4 worker + 原生 Dynamo sidecar，conc 1 和 8），并将 GB300 单节点 vLLM 配置项和配方恢复为 main 的版本。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add a 10P1D point at conc 6656 to the Dynamo disaggregated key Ten DEP2 prefill instances feed the DEP16 decode at max-num-seqs 132. 在 Dynamo 分离式配置项中新增 10P1D 点（conc 6656）：10 个 DEP2 prefill 实例配 DEP16 decode，max-num-seqs 132。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Dynamo frontend settings for the GB300 DSV4.1-Flash Dynamo + vLLM AgentX recipes (#3950) * Dynamo frontend settings for the GB300 DSV4.1-Flash Dynamo + vLLM AgentX recipes Carry the Dynamo runtime and router settings that held up on the GB300 Dynamo + vLLM P/D frontier (c1 to c640, full-hour AgentX replays) into the disaggregated and aggregated recipes: - Conditional disaggregation off: active-decode-blocks-threshold, active-prefill-tokens-threshold and active-prefill-tokens-threshold-frac set to None, so every request is prefilled on a prefill rank and never on one of the 16 lockstep MegaMoE decode ranks. - Router defaults pinned: DYN_ROUTER_TEMPERATURE=0 and DYN_ROUTER_QUEUE_THRESHOLD None (temperature 1 measured as noise, +1 to 4%). - Dynamo request plane tcp and event plane zmq; the NATS service is dropped (no broker hop; the frontier ran this way at c640). - Disaggregated recipe: etcd on a dedicated infra node and the frontend placed beside it, so the single Rust frontend does not share a node with two DEP2 prefill ranks running Engram CPU offload and 70 GiB host KV each. Adds one CPU-only node to the job; the GPU count of the measurement is unchanged. - vLLM engines run with --reasoning-parser None on every role: the deepseek_v41 tool-call and reasoning parsers are applied once, by the Dynamo frontend (published through the sidecar --dyn-*-parser flags), not a second time inside vllm-rs. - DYN_LOG=warn for measured runs. Validated with srtctl dry-run (5P1D c2560 and TP4 c1) on oci-jhb. * Keep the Dynamo frontend on the head node; etcd stays on infra srtctl's orchestrator accepts only head|first_decode|last_decode for frontend.placement.node (src/srtctl/core/topology.py), even though the schema and `srtctl dry-run` accept `infra`; a 10P1D job with the previous commit's placement failed at launch with "frontend.placement.node='infra' is invalid". Drop the frontend placement and the dedicated infra node for etcd; the frontend runs on the head node as before. Verified: the same 10P1D job launches with this change. * Give each worker step every CPU on its node so prefill numa-bind can auto-detect With cpus-per-task 72 every worker step was pinned to CPUs 0-71 (NUMA 0), including the DEP2 prefill worker on GPUs 2-3, and vLLM refuses automatic GPU-to-NUMA detection under a constrained affinity, so the prefill engines exited. With 144 each step (one task, --overlap) gets CPUs 0-143 and vLLM binds each DEP2 prefill worker to its GPUs' NUMA node (GPUs 0-1 -> node 0, GPUs 2-3 -> node 1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Bump srt-slurm to v2.51.4 Picks up NVIDIA/srt-slurm#602: on the Python sidecar launcher (multi-node hybrid-DP decode) srtctl drops `reasoning-parser: none`, which Python vLLM rejects as an unknown parser name, while single-node `vllm-rs serve` endpoints keep it as Dynamo sidecars require. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Run the Dynamo frontend and the AgentX client on a dedicated node On the head node the frontend and the client grew next to two DEP2 prefill instances (Engram CPU tables plus 70 GiB host KV per rank) until the 8P1D c5120 job was OOM-killed during warmup. Both now share one CPU-only node (frontend and benchmark placement dedicated, colocated), which adds one node per disaggregated point and leaves the GPU count unchanged. The frontend is pinned to NUMA node 0 (frontend.numa_bind) now that worker steps get every CPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Retune the AgentX points and bind workers to NUMA nodes Disaggregated points move to 8P1D c4096, 6P1D c2884 and 4P1D c1228, with 10P1D dropped. Prefill keeps its Engram tables in transparent huge pages and takes 32K batched tokens; each aggregated TP4 worker binds to its GPU's NUMA node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add GB300 DeepSeek-V4.1-Flash Dynamo + vLLM disaggregated AgentX recipes Add 8P1D, 6P1D and 5P1D points (DEP2 prefill, DEP16 decode) behind the Dynamo KV router with native vLLM sidecars. 新增 GB300 DeepSeek-V4.1-Flash Dynamo + vLLM 分离式 AgentX 配方：8P1D、6P1D、5P1D 三个点（DEP2 prefill、DEP16 decode），使用 Dynamo KV 路由与原生 vLLM sidecar。 * Address review: drop setup script, enable prefill autotune; GB300 single-node arm on the new image Drop the vllm-container-deps.sh setup script (the image already has numactl and msgpack), enable FlashInfer autotune on prefill, move the GB300 single-node arm to vllm/vllm-openai:nightly-aarch64-dev-20261012-59e75f4 with its TP4 points at conc 1 and 8, and restore a historical changelog pr-link that the previous commit changed by mistake. 处理评审意见：删除 vllm-container-deps.sh 安装脚本（镜像已自带 numactl 和 msgpack），prefill 开启 FlashInfer autotune；GB300 单节点配置改用 vllm/vllm-openai:nightly-aarch64-dev-20261012-59e75f4，TP4 只跑 conc 1 和 8；并恢复上一个提交误改的一条历史 changelog pr-link。 * Move the TP4 points to a Dynamo aggregated key; leave the single-node vLLM arm unchanged Add dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg (one TP4 worker with a native Dynamo sidecar, conc 1 and 8) and restore the GB300 single-node vLLM key and recipe to main. 将 TP4 点移到新的 Dynamo 聚合配置项，单节点 vLLM 配置保持不变：新增 dsv41flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg（单个 TP4 worker + 原生 Dynamo sidecar，conc 1 和 8），并将 GB300 单节点 vLLM 配置项和配方恢复为 main 的版本。 * Add a 10P1D point at conc 6656 to the Dynamo disaggregated key Ten DEP2 prefill instances feed the DEP16 decode at max-num-seqs 132. 在 Dynamo 分离式配置项中新增 10P1D 点（conc 6656）：10 个 DEP2 prefill 实例配 DEP16 decode，max-num-seqs 132。 * Dynamo frontend settings for the GB300 DSV4.1-Flash Dynamo + vLLM AgentX recipes (#3950) * Dynamo frontend settings for the GB300 DSV4.1-Flash Dynamo + vLLM AgentX recipes Carry the Dynamo runtime and router settings that held up on the GB300 Dynamo + vLLM P/D frontier (c1 to c640, full-hour AgentX replays) into the disaggregated and aggregated recipes: - Conditional disaggregation off: active-decode-blocks-threshold, active-prefill-tokens-threshold and active-prefill-tokens-threshold-frac set to None, so every request is prefilled on a prefill rank and never on one of the 16 lockstep MegaMoE decode ranks. - Router defaults pinned: DYN_ROUTER_TEMPERATURE=0 and DYN_ROUTER_QUEUE_THRESHOLD None (temperature 1 measured as noise, +1 to 4%). - Dynamo request plane tcp and event plane zmq; the NATS service is dropped (no broker hop; the frontier ran this way at c640). - Disaggregated recipe: etcd on a dedicated infra node and the frontend placed beside it, so the single Rust frontend does not share a node with two DEP2 prefill ranks running Engram CPU offload and 70 GiB host KV each. Adds one CPU-only node to the job; the GPU count of the measurement is unchanged. - vLLM engines run with --reasoning-parser None on every role: the deepseek_v41 tool-call and reasoning parsers are applied once, by the Dynamo frontend (published through the sidecar --dyn-*-parser flags), not a second time inside vllm-rs. - DYN_LOG=warn for measured runs. Validated with srtctl dry-run (5P1D c2560 and TP4 c1) on oci-jhb. * Keep the Dynamo frontend on the head node; etcd stays on infra srtctl's orchestrator accepts only head|first_decode|last_decode for frontend.placement.node (src/srtctl/core/topology.py), even though the schema and `srtctl dry-run` accept `infra`; a 10P1D job with the previous commit's placement failed at launch with "frontend.placement.node='infra' is invalid". Drop the frontend placement and the dedicated infra node for etcd; the frontend runs on the head node as before. Verified: the same 10P1D job launches with this change. * Give each worker step every CPU on its node so prefill numa-bind can auto-detect With cpus-per-task 72 every worker step was pinned to CPUs 0-71 (NUMA 0), including the DEP2 prefill worker on GPUs 2-3, and vLLM refuses automatic GPU-to-NUMA detection under a constrained affinity, so the prefill engines exited. With 144 each step (one task, --overlap) gets CPUs 0-143 and vLLM binds each DEP2 prefill worker to its GPUs' NUMA node (GPUs 0-1 -> node 0, GPUs 2-3 -> node 1). * Bump srt-slurm to v2.51.4 Picks up NVIDIA/srt-slurm#602: on the Python sidecar launcher (multi-node hybrid-DP decode) srtctl drops `reasoning-parser: none`, which Python vLLM rejects as an unknown parser name, while single-node `vllm-rs serve` endpoints keep it as Dynamo sidecars require. * Run the Dynamo frontend and the AgentX client on a dedicated node On the head node the frontend and the client grew next to two DEP2 prefill instances (Engram CPU tables plus 70 GiB host KV per rank) until the 8P1D c5120 job was OOM-killed during warmup. Both now share one CPU-only node (frontend and benchmark placement dedicated, colocated), which adds one node per disaggregated point and leaves the GPU count unchanged. The frontend is pinned to NUMA node 0 (frontend.numa_bind) now that worker steps get every CPU. * Retune the AgentX points and bind workers to NUMA nodes Disaggregated points move to 8P1D c4096, 6P1D c2884 and 4P1D c1228, with 10P1D dropped. Prefill keeps its Engram tables in transparent huge pages and takes 32K batched tokens; each aggregated TP4 worker binds to its GPU's NUMA node. * Adapt GB300 Dynamo AgentX recipes to current launch contracts Use master recipe selectors and shared AgentX fragments while preserving the five tuned points and dedicated frontend/client placement. Backport the vLLM sidecar reasoning-parser fix through the existing patch mechanism and retain the shared srt-slurm pin. * Limit GB300 Dynamo AgentX submission to completed points --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Pei Li <peili.dev@gmail.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com>
1 parent 01acaba commit f516173

6 files changed

Lines changed: 538 additions & 1 deletion

File tree

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/vllm/gb300-fp4/agentx/agg-variants.yaml‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
schema: 2
2+
3+
base:
4+
name: dsv41flash-fp4-gb300-dynamo-vllm-agentx-agg
5+
identity:
6+
model: {}
7+
container: {}
8+
frameworks:
9+
dynamo: "1.6.0.dev20261010"
10+
dynamo:
11+
install: true
12+
source:
13+
wheel: "1.6.0.dev20261010"
14+
request_plane: tcp
15+
event_plane: zmq
16+
sidecar: true
17+
sidecar_args:
18+
- --grpc-startup-deadline-secs
19+
- "3600"
20+
- --dyn-tool-call-parser
21+
- deepseek_v41
22+
- --dyn-reasoning-parser
23+
- deepseek_v41
24+
environment:
25+
PYTHONHASHSEED: "0"
26+
slurm:
27+
time_limit: "6:00:00"
28+
health_check:
29+
max_attempts: 2160
30+
interval_seconds: 10
31+
resources:
32+
gpu_type: "gb300"
33+
gpus_per_node: 4
34+
services:
35+
- name: etcd
36+
type: etcd
37+
placement:
38+
node: infra
39+
frontend:
40+
type: dynamo
41+
enable_multiple_frontends: false
42+
args:
43+
router-mode: "kv"
44+
router-session-affinity-ttl-secs: 14400
45+
active-decode-blocks-threshold: None
46+
active-prefill-tokens-threshold: None
47+
active-prefill-tokens-threshold-frac: None
48+
env:
49+
DYN_ROUTER_TEMPERATURE: "0"
50+
DYN_ROUTER_QUEUE_THRESHOLD: None
51+
DYN_LOG: warn
52+
DYN_TOKENIZER: "fastokens"
53+
DYN_TOKENIZER_CACHE: "1"
54+
DYN_TOKENIZER_CACHE_BYTES: "8000000000"
55+
DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS: "3600"
56+
DYN_TCP_CHANNEL_BUFFER: "128"
57+
DYN_TCP_REQUEST_TIMEOUT: "60"
58+
engine:
59+
type: vllm
60+
connector:
61+
dp_launch_mode: per_node
62+
roles:
63+
agg:
64+
nodes: 1
65+
workers: 1
66+
gpus: 4
67+
kv_events: true
68+
env:
69+
HF_HUB_CACHE: "/hf_hub_cache"
70+
HUGGINGFACE_HUB_CACHE: "/hf_hub_cache"
71+
TRANSFORMERS_CACHE: "/hf_hub_cache"
72+
VLLM_USE_V2_MODEL_RUNNER: "1"
73+
VLLM_ENGINE_READY_TIMEOUT_S: "7200"
74+
PYTHONUNBUFFERED: "1"
75+
args:
76+
served-model-name: "deepseek-ai/DeepSeek-V4.1-Flash"
77+
language-model-only: true
78+
tokenizer-mode: "deepseek_v41"
79+
# Parse once in the Dynamo frontend.
80+
reasoning-parser: None
81+
tensor-parallel-size: 4
82+
engram-config: '{"cpu_offload":true,"use_thp":true}'
83+
kernel-config: '{"enable_flashinfer_autotune":true}'
84+
attention-config: '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
85+
kv-cache-dtype: "fp8"
86+
speculative-config: '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'
87+
max-model-len: 1048576
88+
# Full-node CPU affinity lets vLLM detect each GPU's NUMA node.
89+
numa-bind: true
90+
sbatch_directives:
91+
mem: "0"
92+
cpus-per-task: "144"
93+
srun_options:
94+
container-remap-root: ""
95+
benchmark:
96+
env:
97+
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
98+
PORT: "8000"
99+
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
100+
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
101+
102+
override_tp4_c1:
103+
name: "dsv41flash-fp4-gb300-dynamo-vllm-agentx-agg-tp4-c1"
104+
roles:
105+
agg:
106+
args:
107+
max-num-seqs: 8
108+
compilation-config: '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}'
109+
max-cudagraph-capture-size: 4092
110+
max-num-batched-tokens: 4096
111+
112+
override_tp4_c8:
113+
name: "dsv41flash-fp4-gb300-dynamo-vllm-agentx-agg-tp4-c8"
114+
roles:
115+
agg:
116+
args:
117+
max-num-seqs: 16
118+
compilation-config: '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}'
119+
max-cudagraph-capture-size: 8190
120+
max-num-batched-tokens: 8192

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/vllm/gb300-fp4/agentx/disagg-variants.yaml‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
schema: 2
2+
3+
base:
4+
name: dsv41flash-fp4-gb300-dynamo-vllm-agentx-disagg
5+
identity:
6+
model: {}
7+
container: {}
8+
frameworks:
9+
dynamo: "1.6.0.dev20261010"
10+
dynamo:
11+
install: true
12+
source:
13+
wheel: "1.6.0.dev20261010"
14+
request_plane: tcp
15+
event_plane: zmq
16+
sidecar: true
17+
sidecar_args:
18+
- --grpc-startup-deadline-secs
19+
- "3600"
20+
- --dyn-tool-call-parser
21+
- deepseek_v41
22+
- --dyn-reasoning-parser
23+
- deepseek_v41
24+
environment:
25+
PYTHONHASHSEED: "0"
26+
slurm:
27+
time_limit: "6:00:00"
28+
health_check:
29+
max_attempts: 2160
30+
interval_seconds: 10
31+
resources:
32+
gpu_type: "gb300"
33+
gpus_per_node: 4
34+
services:
35+
- name: etcd
36+
type: etcd
37+
placement:
38+
node: infra
39+
frontend:
40+
type: dynamo
41+
enable_multiple_frontends: false
42+
numa_bind: true
43+
placement:
44+
# Sharing prefill hosts exhausted RAM at c5120 with Engram and host KV offload.
45+
node: dedicated
46+
args:
47+
router-mode: "kv"
48+
router-session-affinity-ttl-secs: 14400
49+
# Keep prefill off the lockstep decode ranks.
50+
active-decode-blocks-threshold: None
51+
active-prefill-tokens-threshold: None
52+
active-prefill-tokens-threshold-frac: None
53+
env:
54+
DYN_ROUTER_TEMPERATURE: "0"
55+
DYN_ROUTER_QUEUE_THRESHOLD: None
56+
DYN_LOG: warn
57+
DYN_TOKENIZER: "fastokens"
58+
DYN_TOKENIZER_CACHE: "1"
59+
DYN_TOKENIZER_CACHE_BYTES: "8000000000"
60+
DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS: "3600"
61+
DYN_TCP_CHANNEL_BUFFER: "128"
62+
DYN_TCP_REQUEST_TIMEOUT: "60"
63+
engine:
64+
type: vllm
65+
connector:
66+
dp_launch_mode: per_node
67+
roles:
68+
prefill:
69+
gpus: 2
70+
kv_events: true
71+
env:
72+
HF_HUB_CACHE: "/hf_hub_cache"
73+
HUGGINGFACE_HUB_CACHE: "/hf_hub_cache"
74+
TRANSFORMERS_CACHE: "/hf_hub_cache"
75+
VLLM_USE_V2_MODEL_RUNNER: "1"
76+
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
77+
VLLM_RPC_TIMEOUT: "600000"
78+
VLLM_LOG_STATS_INTERVAL: "1"
79+
VLLM_SERVER_DEV_MODE: "1"
80+
VLLM_ALLREDUCE_USE_SYMM_MEM: "0"
81+
VLLM_RANDOMIZE_DP_DUMMY_INPUTS: "1"
82+
NCCL_CUMEM_ENABLE: "1"
83+
NCCL_MNNVL_ENABLE: "1"
84+
NCCL_NVLS_ENABLE: "1"
85+
UCX_TLS: "cuda_copy,cuda_ipc,tcp"
86+
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
87+
UCX_MEMTYPE_CACHE: "n"
88+
UCX_MEMTYPE_REG_WHOLE: "n"
89+
UCX_RNDV_PIPELINE_ERROR_HANDLING: "y"
90+
UCX_MODULE_DIR: "/usr/local/lib/python3.12/dist-packages/nixl_cu13.libs/ucx"
91+
NIXL_UCX_SGL_ENABLE: "1"
92+
args:
93+
kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_connector_module_path":"vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_as_nixl","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"backends":["UCX"],"enforce_handshake_compat":false,"kv_lease_duration":120}},{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":75161927680,"lazy_offload":true}}]}}'
94+
served-model-name: "deepseek-ai/DeepSeek-V4.1-Flash"
95+
language-model-only: true
96+
tokenizer-mode: "deepseek_v41"
97+
# Parse once in the Dynamo frontend.
98+
reasoning-parser: None
99+
kv-cache-dtype: "fp8"
100+
max-model-len: 1048576
101+
tensor-parallel-size: 1
102+
enable-expert-parallel: true
103+
enable-cumem-allocator: true
104+
enable-ep-weight-filter: true
105+
attention-config: '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
106+
engram-config: '{"cpu_offload":true,"embedding_across_dp":true,"use_thp":true}'
107+
speculative-config: '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","enable_adaptive_verification":false}'
108+
prefill-schedule-interval: 2
109+
gpu-memory-utilization: 0.95
110+
data-parallel-size: 2
111+
kernel-config: '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe","sparse_indexer_topk_backend":"deep_select"}'
112+
compilation-config: '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048],"max_cudagraph_capture_size":2046}'
113+
max-num-seqs: 64
114+
max-num-batched-tokens: 32768
115+
long-prefill-token-threshold: 8192
116+
prefix-cache-retention-interval: 32768
117+
numa-bind: true
118+
decode:
119+
nodes: 4
120+
workers: 1
121+
gpus: 16
122+
env:
123+
HF_HUB_CACHE: "/hf_hub_cache"
124+
HUGGINGFACE_HUB_CACHE: "/hf_hub_cache"
125+
TRANSFORMERS_CACHE: "/hf_hub_cache"
126+
VLLM_USE_V2_MODEL_RUNNER: "1"
127+
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
128+
VLLM_RPC_TIMEOUT: "600000"
129+
VLLM_LOG_STATS_INTERVAL: "1"
130+
VLLM_SERVER_DEV_MODE: "1"
131+
VLLM_ALLREDUCE_USE_SYMM_MEM: "0"
132+
VLLM_RANDOMIZE_DP_DUMMY_INPUTS: "1"
133+
NCCL_CUMEM_ENABLE: "1"
134+
NCCL_MNNVL_ENABLE: "1"
135+
NCCL_NVLS_ENABLE: "1"
136+
UCX_TLS: "cuda_copy,cuda_ipc,tcp"
137+
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
138+
UCX_MEMTYPE_CACHE: "n"
139+
UCX_MEMTYPE_REG_WHOLE: "n"
140+
UCX_RNDV_PIPELINE_ERROR_HANDLING: "y"
141+
UCX_MODULE_DIR: "/usr/local/lib/python3.12/dist-packages/nixl_cu13.libs/ucx"
142+
NIXL_UCX_SGL_ENABLE: "1"
143+
VLLM_MAX_ASYNC_KV_ADMITS_PER_STEP: "4"
144+
args:
145+
kv-transfer-config: '{"kv_connector":"NixlConnector","kv_connector_module_path":"vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_as_nixl","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"backends":["UCX"],"enforce_handshake_compat":false,"kv_lease_duration":120}}'
146+
served-model-name: "deepseek-ai/DeepSeek-V4.1-Flash"
147+
language-model-only: true
148+
tokenizer-mode: "deepseek_v41"
149+
# Parse once in the Dynamo frontend.
150+
reasoning-parser: None
151+
kv-cache-dtype: "fp8"
152+
max-model-len: 1048576
153+
tensor-parallel-size: 1
154+
enable-expert-parallel: true
155+
enable-cumem-allocator: true
156+
enable-ep-weight-filter: true
157+
attention-config: '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
158+
engram-config: '{"cpu_offload":true,"embedding_across_dp":true,"use_thp":false}'
159+
speculative-config: '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","enable_adaptive_verification":false}'
160+
prefill-schedule-interval: 2
161+
gpu-memory-utilization: 0.95
162+
data-parallel-size: 16
163+
kernel-config: '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe","sparse_indexer_topk_backend":"deep_select"}'
164+
enable-eplb: true
165+
eplb-config: '{"communicator":"symm_mem","num_redundant_experts":16,"window_size":1000,"step_interval":3000,"use_async":true,"log_balancedness":true,"log_balancedness_interval":3000}'
166+
no-enable-prefix-caching: true
167+
numa-bind: true
168+
numa-bind-nodes: [0, 0, 1, 1]
169+
sbatch_directives:
170+
mem: "0"
171+
cpus-per-task: "144"
172+
srun_options:
173+
container-remap-root: ""
174+
benchmark:
175+
placement:
176+
node: dedicated
177+
env:
178+
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
179+
PORT: "8000"
180+
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
181+
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
182+
183+
override_6p1d_dep2_dep16_c2884:
184+
name: "dsv41flash-fp4-gb300-dynamo-vllm-agentx-disagg-6p1d-dep2-dep16-c2884"
185+
roles:
186+
prefill:
187+
nodes: 3
188+
workers: 6
189+
decode:
190+
args:
191+
max-num-seqs: 40
192+
max-num-batched-tokens: 400
193+
compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":240,"cudagraph_capture_sizes":[6,12,18,24,30,36,42,48,54,60,66,72,78,84,90,96,102,108,114,120,126,132,138,144,150,156,162,168,174,180,186,192,198,204,210,216,222,228,234,240]}'
194+
195+
override_4p1d_dep2_dep16_c1228:
196+
name: "dsv41flash-fp4-gb300-dynamo-vllm-agentx-disagg-4p1d-dep2-dep16-c1228"
197+
roles:
198+
prefill:
199+
nodes: 2
200+
workers: 4
201+
decode:
202+
args:
203+
max-num-seqs: 18
204+
max-num-batched-tokens: 180
205+
compilation-config: '{"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":108,"cudagraph_capture_sizes":[6,12,18,24,30,36,42,48,54,60,66,72,78,84,90,96,102,108]}'

0 commit comments

Comments
 (0)