perf(mi355x): cap DSv4.1-Flash prefill chunk at 8192 on every point

Replace --max-num-batched-tokens 16384 with 8192 on the thirteen TP2/TP4
points that still carried it; TP2 c64 was already 8192. The chunk is now
uniform across the arm, matching the B300 arm and the B200 TP4 arm from
concurrency 8 up.

A prefill chunk scheduled into the same engine step as a decode batch
stalls every decoding sequence for the length of that chunk, so the
stall scales with the chunk. On the two TP4 ep1 arms, the p90-minus-p50
ITL spread is 10.81 ms at c64 on MI355X at 16384 against 1.74 ms on B200
at 8192.

Also restore two perf-changelog.yaml entries that the merge commit of
PR #3690 dropped: the config-keys header of the GLM-5.2 #3616 entry and
the whole Qwen3.5 GB300 #3231 entry.

将仍为 16384 的十三个 TP2/TP4 数据点的 --max-num-batched-tokens 改为 8192；
TP2 c64 原本已是 8192。该分块现在在整个臂上统一，与 B300 臂以及并发 8 及以上
的 B200 TP4 臂一致。

与 decode 批次调度进同一个引擎步的 prefill 分块，会让所有正在解码的序列停顿
该分块的时长，因此停顿时间随分块大小增长。在两个 TP4 ep1 臂上，c64 的 ITL
p90 减 p50 差值在 16384 的 MI355X 上为 10.81 ms，而在 8192 的 B200 上为 1.74 ms。

同时恢复 PR #3690 的合并提交丢失的两条 perf-changelog.yaml 记录：GLM-5.2

Co-authored-by: Cursor <cursoragent@cursor.com>
#3616 条目的 config-keys 头，以及完整的 Qwen3.5 GB300 #3231 条目。

Author: Fangzhou-Ai

inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml

@@ -89,9 +89,12 @@ base:
 
 # One variant per point (c128 excluded for both TP2 and TP4). Engram tables
 # move to pinned host memory at TP2, where resident tables leave too little
-# KV. Graph capture covers 2x CONC x (1 + 5 drafts). The prefill chunk shrinks
-# at TP2 from c64 to buy KV room, and where it drops below 1024 x 6 slots the
-# scheduler is capped at the captured shape.
+# KV. Graph capture covers 2x CONC x (1 + 5 drafts). Every point caps the
+# prefill chunk at 8192, the value the B300 arm uses throughout and the B200
+# arm uses from c8 up: a chunk scheduled alongside a decode step stalls every
+# decoding sequence for the length of that chunk, so the cap bounds the stall
+# rather than buying KV room. It stays above 1024 x 6 slots, so the scheduler
+# is never capped at the captured shape.
 override_tp4_c1:
   roles:
     agg:
@@ -100,7 +103,7 @@ override_tp4_c1:
         tensor-parallel-size: 4
         engram-config: '{"cpu_offload":false}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '1'
@@ -113,7 +116,7 @@ override_tp4_c2:
         tensor-parallel-size: 4
         engram-config: '{"cpu_offload":false}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '2'
@@ -126,7 +129,7 @@ override_tp4_c4:
         tensor-parallel-size: 4
         engram-config: '{"cpu_offload":false}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '4'
@@ -139,7 +142,7 @@ override_tp4_c8:
         tensor-parallel-size: 4
         engram-config: '{"cpu_offload":false}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '8'
@@ -152,7 +155,7 @@ override_tp4_c16:
         tensor-parallel-size: 4
         engram-config: '{"cpu_offload":false}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '16'
@@ -165,7 +168,7 @@ override_tp4_c32:
         tensor-parallel-size: 4
         engram-config: '{"cpu_offload":false}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '32'
@@ -178,7 +181,7 @@ override_tp4_c64:
         tensor-parallel-size: 4
         engram-config: '{"cpu_offload":false}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '64'
@@ -191,7 +194,7 @@ override_tp2_c1:
         tensor-parallel-size: 2
         engram-config: '{"cpu_offload":true}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '1'
@@ -204,7 +207,7 @@ override_tp2_c2:
         tensor-parallel-size: 2
         engram-config: '{"cpu_offload":true}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '2'
@@ -217,7 +220,7 @@ override_tp2_c4:
         tensor-parallel-size: 2
         engram-config: '{"cpu_offload":true}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '4'
@@ -230,7 +233,7 @@ override_tp2_c8:
         tensor-parallel-size: 2
         engram-config: '{"cpu_offload":true}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '8'
@@ -243,7 +246,7 @@ override_tp2_c16:
         tensor-parallel-size: 2
         engram-config: '{"cpu_offload":true}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '16'
@@ -256,7 +259,7 @@ override_tp2_c32:
         tensor-parallel-size: 2
         engram-config: '{"cpu_offload":true}'
         max-cudagraph-capture-size: 1024
-        max-num-batched-tokens: 16384
+        max-num-batched-tokens: 8192
   benchmark:
     env:
       CONC: '32'

inferencex-e2e/configs/amd-master.yaml

@@ -1507,11 +1507,13 @@ dsv41flash-fp4-mi355x-vllm-agentic-dspark:
     agentic-coding:
     - dram-utilization: 0.60
       search-space:
-      # The recipe only changes the memory split where a point ran short of KV:
-      # the prefill chunk shrinks at TP=2 c64. c128 is excluded for both TP=2
-      # and TP=4. Every point is measured here rather than combined from the
-      # earlier run: the image move leaves concurrency 1-32 measured only on
-      # the superseded nightly-eed1f3d0 pin.
+      # The prefill chunk is a uniform 8192 across every point, bounding the
+      # decode stall a chunk causes when it shares an engine step with a decode
+      # batch. The recipe otherwise changes the memory split only where a point
+      # ran short of KV. c128 is excluded for both TP=2 and TP=4. Every point is
+      # measured here rather than combined from the earlier run: the image move
+      # leaves concurrency 1-32 measured only on the superseded nightly-eed1f3d0
+      # pin, and the chunk change invalidates the 16384 points.
       - { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
       - { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32, 64], srt-recipe: benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml }
 

inferencex-e2e/docs/configuration-procedures.md

@@ -697,7 +697,7 @@ A configuration is ready for sweep only when the executable files agree, the exa
 
 The `dsv41flash-fp4-mi355x-vllm-agentic-dspark` recipe extends [#2958](https://github.com/SemiAnalysisAI/InferenceX/pull/2958) to MI355X AgentX: TP4 and TP2, concurrency 1–64, native five-token DSpark. Throughput uses the [committed golden AL](../infx/golden_al_distribution/dsv41flash_dspark.yaml) of 3.51 for thinking on and five draft tokens, with synthetic rejection sampling and adaptive verification disabled. Accuracy evals retain real block rejection but, unlike the CUDA arms, also keep adaptive verification disabled: it trims verification requests on device, which the ROCm `DeepseekV4IndexerBackend` does not support, and the engine refused to start with it enabled ([run 34651830283](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34651830283)). FP4 describes the MXFP4 experts; the checkpoint also contains MXFP8 weights.
 
-Follow the AMD overrides in the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968): `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, and `--moe-backend aiter`. The generic AITER selector uses CK experts; `VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1` opts into the DeepSeek-V4.1 a4w4 path from [vllm-project/vllm#58819](https://github.com/vllm-project/vllm/pull/58819). This is a process-wide opt-in for MXFP4 MoE experts, including DSpark layers that use the same implementation. `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` additionally routes the tensor-parallel all-reduce through the INT4 quick-reduce codec, as in [upstream recipe #1061](https://github.com/vllm-project/recipes/pull/1061) and the merged MI355X DSV4-Pro and Kimi-K3 vLLM recipes. The recipe pins `semianalysis_cc_traces_weka_062126` (the unfiltered corpus) via `WEKA_LOADER_OVERRIDE`. KV stays GPU-resident. Engram stayed on GPU under the upstream AMD defaults until [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) widened the two `is_cuda()` gates to `is_cuda_alike()`. From that commit on, ROCm resolves an `EngramConfig` and `cpu_offload` defaults to on through `VLLM_PLE_CPU_OFFLOAD`, so the recipe sets `--engram-config` explicitly rather than leaning on that default. TP=2 always offloads, since the tables need 94.4 GiB per rank there; TP=4 keeps them resident, since its KV pool is not the constraint through concurrency 64. The recipe likewise trims `--max-num-batched-tokens` only above concurrency 32, to 8192 at TP=2 c64, because the sparse-attention indexer and its companion per-rank buffers grow at roughly 4.4 MiB per batched token. Where that chunk falls below six times the API-server default of 1024 sequences, `--max-num-seqs` is capped at the graph-capture shape: DSpark verifies 1+5 tokens per sequence, and at 4096 against 1024 sequences the engram projection faults during profiling. The rule in every case is to spend device memory on KV only at the concurrencies that ran short of it, leaving the validated low-concurrency settings alone. Images built before that merge still reject the option on ROCm. The MI355X launcher uses the shared HF cache and mounts this model's repository at `/ix`, and exports `INFMAX_CONTAINER_WORKSPACE=/ix` so AgentX dependencies and outputs resolve inside that mount.
+Follow the AMD overrides in the merged [upstream recipe #968](https://github.com/vllm-project/recipes/pull/968): `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_MOE=1`, and `--moe-backend aiter`. The generic AITER selector uses CK experts; `VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4=1` opts into the DeepSeek-V4.1 a4w4 path from [vllm-project/vllm#58819](https://github.com/vllm-project/vllm/pull/58819). This is a process-wide opt-in for MXFP4 MoE experts, including DSpark layers that use the same implementation. `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` additionally routes the tensor-parallel all-reduce through the INT4 quick-reduce codec, as in [upstream recipe #1061](https://github.com/vllm-project/recipes/pull/1061) and the merged MI355X DSV4-Pro and Kimi-K3 vLLM recipes. The recipe pins `semianalysis_cc_traces_weka_062126` (the unfiltered corpus) via `WEKA_LOADER_OVERRIDE`. KV stays GPU-resident. Engram stayed on GPU under the upstream AMD defaults until [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491) widened the two `is_cuda()` gates to `is_cuda_alike()`. From that commit on, ROCm resolves an `EngramConfig` and `cpu_offload` defaults to on through `VLLM_PLE_CPU_OFFLOAD`, so the recipe sets `--engram-config` explicitly rather than leaning on that default. TP=2 always offloads, since the tables need 94.4 GiB per rank there; TP=4 keeps them resident, since its KV pool is not the constraint through concurrency 64. `--max-num-batched-tokens` is 8192 at every point, which is also what the B300 arm uses throughout and the B200 TP4 arm uses from concurrency 8 up. This cap is for decode latency, not memory: a prefill chunk scheduled into the same engine step as a decode batch stalls every decoding sequence for the length of that chunk, so the stall scales with the chunk, and the p90-minus-p50 ITL spread at 16384 reached 10.81 ms at TP=4 c64 against 1.74 ms on the B200 arm at 8192. Memory is a second reason to keep it there, since the sparse-attention indexer and its companion per-rank buffers grow at roughly 4.4 MiB per batched token. Do not drop it below six times the API-server default of 1024 sequences, or `--max-num-seqs` is capped at the graph-capture shape: DSpark verifies 1+5 tokens per sequence, and at 4096 against 1024 sequences the engram projection faults during profiling. Engram placement still follows the older rule of spending device memory on KV only at the concurrencies that ran short of it. Images built before that merge still reject the option on ROCm. The MI355X launcher uses the shared HF cache and mounts this model's repository at `/ix`, and exports `INFMAX_CONTAINER_WORKSPACE=/ix` so AgentX dependencies and outputs resolve inside that mount.
 
 The current candidate pins `vllm/vllm-openai-rocm:nightly-rocm100-0cbac6cd1305f710e12193596b27488397bcb205` with a4w4 and INT4 quick reduce enabled. Its GPU sweep and accuracy evaluation must qualify this image/flag combination; the evidence below describes the preceding a8w4, unquantized-collective baseline.
 
@@ -719,10 +719,10 @@ rejects every candidate.
 
 Both arms run **TP8**, not the MI355X TP4: a 192 GB (MI300X) or 256 GB (MI325X) card must
 hold its share of the 511 GB checkpoint plus the GPU-resident Engram tables (upstream AMD
-defaults; no CPU offload) and still leave a 1M-context KV pool. MI300X additionally caps
-`--max-num-batched-tokens` at 8192 because the sparse-attention indexer allocates a
-`[batched-tokens, max-model-len]` fp8 logits buffer at startup (16 GiB at 8192, 32 GiB at
-the MI355X arm's 16384). Concurrency is 1–32 on both. Both entries also carry smaller
+defaults; no CPU offload) and still leave a 1M-context KV pool. MI300X caps
+`--max-num-batched-tokens` at 8192, as the MI355X arm now does at every point, because the
+sparse-attention indexer allocates a `[batched-tokens, max-model-len]` fp8 logits buffer at
+startup (16 GiB at 8192, 32 GiB at the upstream default of 16384). Concurrency is 1–32 on both. Both entries also carry smaller
 layouts (MI300X TP4; MI325X TP2 and TP4) that move the Engram tables to host memory with
 `--engram-config '{"cpu_offload":true}'`.