Skip to content

Commit 2d27bdf

Browse files
hshrivastava-droidclaude
hshrivastava-droid
and
claude
committed
[NVIDIA][AgentX] Add Qwen3.5 NVFP4 B300 TensorRT-LLM MTP AgentX recipe / [NVIDIA][AgentX] 新增 Qwen3.5 NVFP4 B300 TensorRT-LLM MTP AgentX 配置
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent f85e1b9 commit 2d27bdf

3 files changed

Lines changed: 247 additions & 0 deletions

File tree

‎inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/qwen3.5/trtllm/b300-fp4-mtp/agentic.yaml‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# Qwen3.5-397B-A17B NVFP4 AgentX on B300 with TensorRT-LLM: one aggregated
2+
# TEP8 worker (TP8, MoE EP8) with the checkpoint's native MTP head at draft
3+
# length 6. The AgentX connector sets forced acceptance from the golden curve
4+
# (qwen3.5 MTP, thinking_on, draft 6: AL 4.80), so this recipe does not set it.
5+
#
6+
# Every point uses piecewise CUDA graphs for steps that carry context tokens.
7+
# Three per-concurrency settings sit on top of that base:
8+
# c4, c8: TRTLLM-gen MoE and two torch.compile streams
9+
# c16: the base alone (CUTEDSL MoE, no context batching)
10+
# c24-c64: TRTLLM-gen MoE and context batching (batch_wait 0.9 / 24)
11+
# Context batching lets several prompts share one context step. At c16 and
12+
# below it adds more first-token wait than it saves end to end.
13+
#
14+
# kv_cache_config.host_cache_size is TRT-LLM's native host KV tier: 128 GiB per
15+
# rank, 1,100 GB per node at TP8. The master config declares it as
16+
# kv-offloading dram with the native backend.
17+
base:
18+
schema: 2
19+
name: qwen3.5-fp4-b300-trtllm-agentic
20+
model:
21+
path: hf:nvidia/Qwen3.5-397B-A17B-NVFP4
22+
container: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc24
23+
precision: fp4
24+
resources:
25+
gpu_type: b300
26+
gpus_per_node: 8
27+
frontend:
28+
type: trtllm_serve
29+
enable_multiple_frontends: false
30+
observability:
31+
enabled: false
32+
tachometer:
33+
enabled: false
34+
engine:
35+
type: trtllm
36+
served_model_name: nvidia/Qwen3.5-397B-A17B-NVFP4
37+
roles:
38+
agg:
39+
nodes: 1
40+
workers: 1
41+
gpus: 8
42+
args:
43+
tensor_parallel_size: 8
44+
moe_expert_parallel_size: 8
45+
pipeline_parallel_size: 1
46+
enable_attention_dp: false
47+
enable_lm_head_tp_in_adp: false
48+
max_batch_size: 128
49+
max_num_tokens: 16384
50+
max_seq_len: 262144
51+
enable_chunked_prefill: true
52+
disable_overlap_scheduler: false
53+
cuda_graph_config:
54+
enable_padding: true
55+
batch_sizes: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128]
56+
# Piecewise graphs cover steps with context tokens; decode keeps full graphs.
57+
torch_compile_config:
58+
enable_fullgraph: true
59+
enable_inductor: false
60+
enable_piecewise_cuda_graph: true
61+
capture_num_tokens: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384]
62+
moe_config:
63+
backend: CUTEDSL
64+
use_low_precision_moe_combine: true
65+
nvfp4_gemm_config:
66+
allowed_backends: [cutlass, cublaslt, cutedsl, cuda_core]
67+
kv_cache_config:
68+
dtype: fp8
69+
free_gpu_memory_fraction: 0.8
70+
tokens_per_block: 64
71+
use_kv_cache_manager_v2: true
72+
enable_block_reuse: true
73+
block_reuse_config:
74+
policy: per_conversation
75+
max_num_turns: 3
76+
event_buffer_max_size: 0
77+
host_cache_size: 137438953472
78+
pool_ratio: [0.8, 0.2]
79+
mamba_state_config:
80+
periodic_snapshot_interval: 0
81+
additional_snapshot_offsets_from_end: [2]
82+
scheduler_config:
83+
capacity_scheduler_policy: GUARANTEED_NO_EVICT
84+
speculative_config:
85+
decoding_type: MTP
86+
max_draft_len: 6
87+
stream_interval: 20
88+
num_postprocess_workers: 4
89+
enable_iter_perf_stats: false
90+
enable_iter_req_stats: false
91+
print_iter_log: true
92+
trust_remote_code: true
93+
env:
94+
CUDA_SCALE_LAUNCH_QUEUES: 4x
95+
MIMALLOC_ARENA_RESERVE: '0'
96+
MIMALLOC_PURGE_DELAY: ''
97+
NCCL_GRAPH_MIXING_SUPPORT: '0'
98+
PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
99+
TLLM_LOG_LEVEL: INFO
100+
TRTLLM_ENABLE_PDL: '1'
101+
TRTLLM_PINNED_WEIGHT_STAGING: '1'
102+
TRTLLM_SERVER_DISABLE_GC: '1'
103+
TRTLLM_WORKER_DISABLE_GC: '1'
104+
# Piecewise capture to 16384 tokens plus MTP warmup at TP8.
105+
health_check:
106+
max_attempts: 540
107+
interval_seconds: 10
108+
benchmark:
109+
type: custom
110+
command: bash /infmax-workspace/benchmarks/srt_agentic.sh
111+
env:
112+
MODEL: nvidia/Qwen3.5-397B-A17B-NVFP4
113+
# trtllm-serve exposes Prometheus text at /prometheus/metrics.
114+
AIPERF_SERVER_METRICS_URLS: http://localhost:8000/prometheus/metrics
115+
# Request-level trtllm_* series come from return_perf_metrics (the
116+
# trtllm-serve default); per-iteration gauges stay off as measured.
117+
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: trtllm_
118+
# trtllm-serve closes idle connections after 5 s (uvicorn keep-alive,
119+
# fixed in this release); keep the client pool below that so it never
120+
# reuses a connection the server already closed (ECONNRESET in warmup).
121+
AIPERF_HTTP_KEEPALIVE_TIMEOUT: '4'
122+
# Thinking on with Qwen3.5's published sampling, as the golden curve uses.
123+
AIPERF_EXTRA_INPUTS: '{"chat_template_kwargs":{"enable_thinking":true},"presence_penalty":0,"temperature":0.6,"top_k":20,"top_p":0.95}'
124+
# 256k-capped corpus, within this model's context.
125+
WEKA_LOADER_OVERRIDE: semianalysis_cc_traces_weka_062126_256k
126+
127+
# One variant per point.
128+
# Low concurrency: TRTLLM-gen MoE, and two torch.compile streams.
129+
override_c4:
130+
roles:
131+
agg:
132+
args:
133+
moe_config:
134+
backend: TRTLLM
135+
torch_compile_config:
136+
max_num_streams: 2
137+
benchmark:
138+
env:
139+
CONC: '4'
140+
KV_OFFLOADING: dram
141+
TOTAL_CPU_DRAM_GB: '1100'
142+
143+
override_c8:
144+
roles:
145+
agg:
146+
args:
147+
moe_config:
148+
backend: TRTLLM
149+
torch_compile_config:
150+
max_num_streams: 2
151+
benchmark:
152+
env:
153+
CONC: '8'
154+
KV_OFFLOADING: dram
155+
TOTAL_CPU_DRAM_GB: '1100'
156+
157+
override_c16:
158+
benchmark:
159+
env:
160+
CONC: '16'
161+
KV_OFFLOADING: dram
162+
TOTAL_CPU_DRAM_GB: '1100'
163+
164+
# Throughput end: TRTLLM-gen MoE and context batching. A context step waits up
165+
# to 24 iterations, or until 90% of max_num_tokens is queued.
166+
override_c24:
167+
roles:
168+
agg:
169+
args:
170+
moe_config:
171+
backend: TRTLLM
172+
batch_wait_max_tokens_ratio: 0.9
173+
batch_wait_timeout_iters: 24
174+
benchmark:
175+
env:
176+
CONC: '24'
177+
KV_OFFLOADING: dram
178+
TOTAL_CPU_DRAM_GB: '1100'
179+
180+
override_c32:
181+
roles:
182+
agg:
183+
args:
184+
moe_config:
185+
backend: TRTLLM
186+
batch_wait_max_tokens_ratio: 0.9
187+
batch_wait_timeout_iters: 24
188+
benchmark:
189+
env:
190+
CONC: '32'
191+
KV_OFFLOADING: dram
192+
TOTAL_CPU_DRAM_GB: '1100'
193+
194+
override_c52:
195+
roles:
196+
agg:
197+
args:
198+
moe_config:
199+
backend: TRTLLM
200+
batch_wait_max_tokens_ratio: 0.9
201+
batch_wait_timeout_iters: 24
202+
benchmark:
203+
env:
204+
CONC: '52'
205+
KV_OFFLOADING: dram
206+
TOTAL_CPU_DRAM_GB: '1100'
207+
208+
override_c64:
209+
roles:
210+
agg:
211+
args:
212+
moe_config:
213+
backend: TRTLLM
214+
batch_wait_max_tokens_ratio: 0.9
215+
batch_wait_timeout_iters: 24
216+
benchmark:
217+
env:
218+
CONC: '64'
219+
KV_OFFLOADING: dram
220+
TOTAL_CPU_DRAM_GB: '1100'

‎inferencex-e2e/configs/nvidia-master.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5248,6 +5248,24 @@ qwen3.5-fp4-b300-sglang-agentic-mtp:
52485248
- { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20, 24, 28, 32], srt-recipe: benchmarks/single_node/srt-slurm-recipes/qwen3.5/sglang/b300-fp4-mtp/agentic.yaml }
52495249
- { tp: 2, ep: 2, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [36, 44, 52], srt-recipe: benchmarks/single_node/srt-slurm-recipes/qwen3.5/sglang/b300-fp4-mtp/agentic.yaml }
52505250

5251+
# Qwen3.5-397B-A17B NVFP4 AgentX on B300 with TensorRT-LLM: one aggregated
5252+
# TEP8 worker (TP8, MoE EP8) with native MTP at draft length 6 (golden AL 4.80,
5253+
# thinking on). Every point keeps TRT-LLM's native 128 GiB/rank host KV tier,
5254+
# 1,100 GB per node at TP8 (dram-utilization 0.367 of the AgentX DRAM ceiling).
5255+
qwen3.5-fp4-b300-trtllm-agentic-mtp:
5256+
image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc24
5257+
model: nvidia/Qwen3.5-397B-A17B-NVFP4
5258+
model-prefix: qwen3.5
5259+
runner: cluster:b300-dsxe
5260+
precision: fp4
5261+
framework: trt
5262+
multinode: false
5263+
scenarios:
5264+
agentic-coding:
5265+
- dram-utilization: 0.367
5266+
search-space:
5267+
- { tp: 8, ep: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: native, version: "1.3.0rc24" }, conc-list: [4, 8, 16, 24, 32, 52, 64], srt-recipe: benchmarks/single_node/srt-slurm-recipes/qwen3.5/trtllm/b300-fp4-mtp/agentic.yaml }
5268+
52515269
# Qwen3.8-Flash-Next NVFP4 AgentX on B300 via SGLang with native NEXTN MTP.
52525270
# Day-zero recipe; mirrors the B200 arm. TP1 per the cookbook's verified
52535271
# single-node command: 126 GiB of NVFP4 weights fit on one B300.

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9646,3 +9646,12 @@
96469646
- "ATOM's mono decode path (on by default) now takes any decode step of up to 48 rows, i.e. concurrency 1-8 with five DSpark drafts, not only DSpark verify steps. Steps of 12 or more rows that are a multiple of TP run its sublayers token-parallel, with reduce-scatters instead of all-reduces. The draft runner never runs token-parallel."
96479647
- "No data-type or precision change to the DSpark draft. The checkpoint's bundled mtp.0-2 stages are stored like the target's layers: FP8 E4M3 weights with 32x32 UE8M0 block scales for the attention projections, shared experts and main_proj; MXFP4 routed experts; BF16 norms, router gate, Markov head and confidence head; F32 attention sinks and hc mixers. ATOM builds the draft with the target's native_quant_config() and make_v4_quant_config(), with no online quantization in this recipe, so it runs as shipped: FP8 activations on the FP8 GEMMs and the target's BF16 KV cache. Between 73defd9b8 and a0527c9db only the rows the draft reads under bounded replay change."
96489648
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3928
9649+
9650+
- config-keys:
9651+
- qwen3.5-fp4-b300-trtllm-agentic-mtp
9652+
scenario-type:
9653+
- agentic-coding
9654+
description:
9655+
- "Add Qwen3.5-397B-A17B NVFP4 B300 AgentX on TensorRT-LLM release:1.3.0rc24: one aggregated TEP8 worker (TP8, MoE EP8) with native MTP at draft length 6, at concurrency 4, 8, 16, 24, 32, 52 and 64."
9656+
- "FP8 KV cache with KV cache manager v2, per-conversation block reuse, and TRT-LLM's native 128 GiB/rank host KV tier (dram-utilization 0.367)."
9657+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3967

0 commit comments

Comments
 (0)