Skip to content

Commit 552b356

Browse files
richardhuo-nvRohitNagrajclaudecquil11
authored
glm5.2 gb300 trt agentx: rc28 + agent bounce + Dynamo --publish-metrics / glm5.2 gb300 trt agentx:rc28 + agent bounce + Dynamo --publish-metrics (#3707)
* feat(config): update GLM-5.2 GB300 TRT AgentX recipes Update the image, Dynamo wheel, bounce-buffer settings, and metric publication across all six existing points. Remove the redundant served-model-name patch. 更新 GLM-5.2 GB300 TRT AgentX 六个现有点的镜像、Dynamo wheel、bounce buffer 设置及指标发布方式,删除重复的 served-model-name 补丁。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Cameron Quilici <60715037+cquil11@users.noreply.github.com> * fix: pin GLM-5.2 GB300 Dynamo routed address selection 修复:固定 GLM-5.2 GB300 Dynamo 版本以选择可路由的节点地址。 * glm5.2 gb300 trt agentx: move Dynamo to the ai-dynamo 1.6.0.dev20261007 wheel The nightly is built after ai-dynamo/dynamo#15628 (ee07d6b6, routed node address selection), so it replaces the source-build pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(config): align GLM-5.2 KV offload backend version with rc28 将 GLM-5.2 GB300 的六个 KV 卸载后端版本标签更新为 rc28,与服务镜像版本保持一致。 * chore(config): bump GLM-5.2 GB300 TRT AgentX Dynamo wheel to 1.6.0.dev20261009 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(config): bump GLM-5.2 GB300 TRT AgentX Dynamo wheel to 1.6.0.dev20261010 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Rohit Pujar Nagraj <rpujarnagraj@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com> Co-authored-by: cquil11 <cjquilici@gmail.com>
1 parent 271b3ad commit 552b356

3 files changed

Lines changed: 47 additions & 51 deletions

File tree

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/glm5.2/trtllm/gb300-fp4/agentx/disagg-variants.yaml‎

Lines changed: 29 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ schema: 2
77
base:
88
model:
99
path: nvidia/GLM-5.2-NVFP4
10-
container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc26.dev202609040000
10+
container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc28.dev202609210001
1111
precision: fp4
1212
dynamo:
1313
install: true
@@ -20,7 +20,11 @@ base:
2020
gpus_per_node: 4
2121
engine:
2222
type: trtllm
23-
publish_events_and_metrics: false
23+
# Publish engine metrics without KV events. The pinned Dynamo wheel also
24+
# selects the routed node address for metadata and transport advertisement
25+
# (ai-dynamo/dynamo#15628).
26+
# srtctl advertises the worker leaders' /metrics to AIPerf.
27+
publish_metrics: true
2428
roles:
2529
prefill:
2630
gpus: 4
@@ -57,7 +61,13 @@ base:
5761
cache_transceiver_config:
5862
backend: NIXL
5963
transceiver_runtime: PYTHON
60-
kv_cache_bounce_size_mb: 5120
64+
# C++ transfer-agent bounce: every prefill->decode copy is staged in fixed pieces instead of
65+
# first-touch writes into the remote pool.
66+
agent_bounce_buffer_enable: true
67+
agent_bounce_params:
68+
max_average_descriptor_size: 4MB
69+
min_descriptor_count: '1'
70+
kv_cache_bounce_size_mb: 1024
6171
max_tokens_in_buffer: 1048576
6272
kv_transfer_timeout_ms: 600000
6373
cuda_graph_config:
@@ -80,6 +90,7 @@ base:
8090
moe_expert_parallel_size: 4
8191
num_postprocess_workers: 8
8292
pipeline_parallel_size: 1
93+
enable_iter_perf_stats: true
8394
print_iter_log: true
8495
return_perf_metrics: false
8596
scheduler_config:
@@ -122,7 +133,13 @@ base:
122133
cache_transceiver_config:
123134
backend: NIXL
124135
transceiver_runtime: PYTHON
125-
kv_cache_bounce_size_mb: 5120
136+
# C++ transfer-agent bounce: every prefill->decode copy is staged in fixed pieces instead of
137+
# first-touch writes into the remote pool.
138+
agent_bounce_buffer_enable: true
139+
agent_bounce_params:
140+
max_average_descriptor_size: 4MB
141+
min_descriptor_count: '1'
142+
kv_cache_bounce_size_mb: 1024
126143
max_tokens_in_buffer: 1048576
127144
kv_transfer_timeout_ms: 600000
128145
cuda_graph_config:
@@ -142,6 +159,7 @@ base:
142159
use_low_precision_moe_combine: true
143160
num_postprocess_workers: 4
144161
pipeline_parallel_size: 1
162+
enable_iter_perf_stats: true
145163
print_iter_log: true
146164
return_perf_metrics: false
147165
sparse_attention_config:
@@ -178,6 +196,7 @@ base:
178196
command: bash /infmax-workspace/benchmarks/srt_agentic.sh
179197
env:
180198
INFMAX_CONTAINER_WORKSPACE: /infmax-workspace
199+
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: trtllm_kv_cache_utilization
181200
RESULT_DIR: /logs/agentic
182201
PORT: '8000'
183202
IS_MULTINODE: 'true'
@@ -199,7 +218,7 @@ override_1p1d_tp8_c1_b1_mtp5:
199218
name: dynamo-disagg-gb300-1p1d-tp8-c1-b1-mtp5
200219
dynamo:
201220
source:
202-
wheel: 1.4.0.dev20260807
221+
wheel: 1.6.0.dev20261010
203222
roles:
204223
prefill:
205224
nodes: 1
@@ -225,14 +244,11 @@ override_1p1d_tep8_c20_b5_mtp5:
225244
name: dynamo-disagg-gb300-1p1d-tep8-c20-b5-mtp5
226245
dynamo:
227246
source:
228-
git: https://github.com/cquil11/dynamo.git
229-
rev: 2cbbdc863c336c2cfffa2a3668e2a6661801b259
247+
wheel: 1.6.0.dev20261010
230248
roles:
231249
prefill:
232250
nodes: 1
233251
workers: 1
234-
args:
235-
enable_iter_perf_stats: true
236252
decode:
237253
nodes: 2
238254
gpus: 8
@@ -245,26 +261,19 @@ override_1p1d_tep8_c20_b5_mtp5:
245261
- 5
246262
max_batch_size: 5
247263
moe_expert_parallel_size: 8
248-
enable_iter_perf_stats: true
249264
sparse_attention_config:
250265
use_cute_dsl_topk: true
251266
tensor_parallel_size: 8
252-
benchmark:
253-
env:
254-
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: trtllm_kv_cache_utilization
255267

256268
override_1p4d_tep4_c30_b2_mtp5:
257269
name: dynamo-disagg-gb300-1p4d-tep4-c30-b2-mtp5
258270
dynamo:
259271
source:
260-
git: https://github.com/cquil11/dynamo.git
261-
rev: 2cbbdc863c336c2cfffa2a3668e2a6661801b259
272+
wheel: 1.6.0.dev20261010
262273
roles:
263274
prefill:
264275
nodes: 1
265276
workers: 1
266-
args:
267-
enable_iter_perf_stats: true
268277
decode:
269278
nodes: 4
270279
workers: 4
@@ -276,26 +285,19 @@ override_1p4d_tep4_c30_b2_mtp5:
276285
- 2
277286
max_batch_size: 2
278287
moe_expert_parallel_size: 4
279-
enable_iter_perf_stats: true
280288
sparse_attention_config:
281289
use_cute_dsl_topk: true
282290
tensor_parallel_size: 4
283-
benchmark:
284-
env:
285-
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: trtllm_kv_cache_utilization
286291

287292
override_3p4d_tep4_c60_b5_mtp5:
288293
name: dynamo-disagg-gb300-3p4d-tep4-c60-b5-mtp5
289294
dynamo:
290295
source:
291-
git: https://github.com/cquil11/dynamo.git
292-
rev: 2cbbdc863c336c2cfffa2a3668e2a6661801b259
296+
wheel: 1.6.0.dev20261010
293297
roles:
294298
prefill:
295299
nodes: 3
296300
workers: 3
297-
args:
298-
enable_iter_perf_stats: true
299301
decode:
300302
nodes: 4
301303
workers: 4
@@ -309,26 +311,20 @@ override_3p4d_tep4_c60_b5_mtp5:
309311
- 5
310312
max_batch_size: 5
311313
moe_expert_parallel_size: 4
312-
enable_iter_perf_stats: true
313314
sparse_attention_config:
314315
use_cute_dsl_topk: true
315316
tensor_parallel_size: 4
316-
benchmark:
317-
env:
318-
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: trtllm_kv_cache_utilization
319317

320318
override_4p1d_dep8_c227_b16_mtp3:
321319
name: dynamo-disagg-gb300-4p1d-dep8-c227-b16-mtp3
322320
dynamo:
323321
source:
324-
git: https://github.com/cquil11/dynamo.git
325-
rev: 2cbbdc863c336c2cfffa2a3668e2a6661801b259
322+
wheel: 1.6.0.dev20261010
326323
roles:
327324
prefill:
328325
nodes: 4
329326
workers: 4
330327
args:
331-
enable_iter_perf_stats: true
332328
speculative_config:
333329
max_draft_len: 3
334330
decode:
@@ -351,26 +347,20 @@ override_4p1d_dep8_c227_b16_mtp3:
351347
moe_config:
352348
backend: CUTEDSL
353349
moe_expert_parallel_size: 8
354-
enable_iter_perf_stats: true
355350
speculative_config:
356351
max_draft_len: 3
357352
tensor_parallel_size: 8
358-
benchmark:
359-
env:
360-
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: trtllm_kv_cache_utilization
361353

362354
override_5p1d_dep16_c260_b16_mtp3:
363355
name: dynamo-disagg-gb300-5p1d-dep16-c260-b16-mtp3
364356
dynamo:
365357
source:
366-
git: https://github.com/cquil11/dynamo.git
367-
rev: 2cbbdc863c336c2cfffa2a3668e2a6661801b259
358+
wheel: 1.6.0.dev20261010
368359
roles:
369360
prefill:
370361
nodes: 5
371362
workers: 5
372363
args:
373-
enable_iter_perf_stats: true
374364
speculative_config:
375365
max_draft_len: 3
376366
decode:
@@ -390,12 +380,8 @@ override_5p1d_dep16_c260_b16_mtp3:
390380
free_gpu_memory_fraction: 0.9
391381
max_batch_size: 16
392382
moe_expert_parallel_size: 16
393-
enable_iter_perf_stats: true
394383
sparse_attention_config:
395384
use_cute_dsl_topk: true
396385
speculative_config:
397386
max_draft_len: 3
398387
tensor_parallel_size: 16
399-
benchmark:
400-
env:
401-
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: trtllm_kv_cache_utilization

‎inferencex-e2e/configs/nvidia-master.yaml‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7885,13 +7885,13 @@ glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg:
78857885
# GLM-5.2 NVFP4 TensorRT-LLM AgentX on GB300 with Dynamo disaggregated serving.
78867886
# Six topology variants use NIXL KV transfer and MTP3/MTP5 decoding.
78877887
glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp:
7888-
image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc26.dev202609040000
7888+
image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc28.dev202609210001
78897889
model: nvidia/GLM-5.2-NVFP4
78907890
model-prefix: glm5.2
78917891
runner: cluster:gb300-nv
78927892
precision: fp4
78937893
framework: dynamo-trt
7894-
router: { name: dynamo-router, version: "1.4.0.dev20260807" }
7894+
router: { name: dynamo-router, version: "1.6.0.dev20261010" }
78957895
kv-p2p-transfer: nixl
78967896
multinode: true
78977897
disagg: true
@@ -7902,7 +7902,7 @@ glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp:
79027902
- spec-decoding: mtp
79037903
conc-list: [1]
79047904
kv-offloading: dram
7905-
kv-offload-backend: { name: native, version: "1.3.0rc26.dev202609040000" }
7905+
kv-offload-backend: { name: native, version: "1.3.0rc28.dev202609210001" }
79067906
prefill:
79077907
num-worker: 1
79087908
tp: 4
@@ -7919,7 +7919,7 @@ glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp:
79197919
- spec-decoding: mtp
79207920
conc-list: [30]
79217921
kv-offloading: dram
7922-
kv-offload-backend: { name: native, version: "1.3.0rc26.dev202609040000" }
7922+
kv-offload-backend: { name: native, version: "1.3.0rc28.dev202609210001" }
79237923
prefill:
79247924
num-worker: 1
79257925
tp: 4
@@ -7936,7 +7936,7 @@ glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp:
79367936
- spec-decoding: mtp
79377937
conc-list: [20]
79387938
kv-offloading: dram
7939-
kv-offload-backend: { name: native, version: "1.3.0rc26.dev202609040000" }
7939+
kv-offload-backend: { name: native, version: "1.3.0rc28.dev202609210001" }
79407940
prefill:
79417941
num-worker: 1
79427942
tp: 4
@@ -7953,7 +7953,7 @@ glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp:
79537953
- spec-decoding: mtp
79547954
conc-list: [60]
79557955
kv-offloading: dram
7956-
kv-offload-backend: { name: native, version: "1.3.0rc26.dev202609040000" }
7956+
kv-offload-backend: { name: native, version: "1.3.0rc28.dev202609210001" }
79577957
prefill:
79587958
num-worker: 3
79597959
tp: 4
@@ -7970,7 +7970,7 @@ glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp:
79707970
- spec-decoding: mtp
79717971
conc-list: [260]
79727972
kv-offloading: dram
7973-
kv-offload-backend: { name: native, version: "1.3.0rc26.dev202609040000" }
7973+
kv-offload-backend: { name: native, version: "1.3.0rc28.dev202609210001" }
79747974
prefill:
79757975
num-worker: 5
79767976
tp: 4
@@ -7987,7 +7987,7 @@ glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp:
79877987
- spec-decoding: mtp
79887988
conc-list: [227]
79897989
kv-offloading: dram
7990-
kv-offload-backend: { name: native, version: "1.3.0rc26.dev202609040000" }
7990+
kv-offload-backend: { name: native, version: "1.3.0rc28.dev202609210001" }
79917991
prefill:
79927992
num-worker: 4
79937993
tp: 4

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9611,3 +9611,13 @@
96119611
- "Concurrency 1 (dcp1 and dcp8) adds quantization-config linear fp8_per_channel: vLLM quantizes the target's linear layers to FP8 W8A8 at load, except kv_b_proj, in_proj_qkvgfab, f_b_proj, fused_qkv_a_proj, self_attn.g_proj and layer 0 mlp.down_proj, which stay BF16. The routed experts stay MXFP4."
96129612
- "No change to the Inferact/Kimi-K3-DSpark draft's precision: its model config takes the speculative config's own quantization (unset) and never receives quantization-config, and get_draft_quant_config() returns None for K3DSparkModel, so its weights stay BF16 and its KV cache stays fp8."
96139613
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3833
9614+
9615+
- config-keys:
9616+
- glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp
9617+
scenario-type:
9618+
- agentic-coding
9619+
description:
9620+
- "Update the GLM-5.2 NVFP4 GB300 TensorRT-LLM AgentX image from 1.3.0rc26.dev202609040000 to 1.3.0rc28.dev202609210001 on all six points."
9621+
- "Switch NIXL KV transfer to the C++ transfer-agent bounce buffer (max_average_descriptor_size 4MB, min_descriptor_count 1) and lower kv_cache_bounce_size_mb from 5120 to 1024."
9622+
- "Move Dynamo on all six points to the ai-dynamo 1.6.0.dev20261010 wheel, which includes ai-dynamo/dynamo#15628 (routed node address selection for metadata and transport advertisement)."
9623+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3707

0 commit comments

Comments
 (0)