Repository navigation
Conversation
…DRAM Port of #3615 onto the current srt driver. Every single-node srun step gets --mem = available-cpu-dram-mib / gpus-per-node * GPU_COUNT through SRT_SRUN_OPTIONS, so a point using half the node can no longer take more than half its host DRAM. The sbatch job requests the whole available-cpu-dram-mib only when slurm.srt-slurm.single-node-exclusive is true; on shared nodes it requests the same GPU share, so it leaves the rest of the node to other jobs. Clusters without available-cpu-dram-mib keep their previous behavior. Co-authored-by: Cam Quilici <cjquilici@gmail.com>
Three single-node AgentX DRAM arms pinned host KV pools at or within a few GiB of their TP4 srun step's new --mem share of node DRAM: - minimaxm3-fp4-b200-vllm-agentic-mtp TP4 SimpleCPU: dram-utilization 1.0 -> 0.96, cpu_bytes_to_use and TOTAL_CPU_DRAM_GB 1082 -> 1038 GB (headroom 0.08 -> 41.1 GiB under 1,031,960 MiB). - minimaxm3-fp4-b200-trtllm-agentic-mtp TP4: per-rank host_cache_size 250 -> 240 GiB (headroom 7.8 -> 47.8 GiB under 1,031,960 MiB). - qwen3.5-fp8-mi355x-sglang-agentic-mtp TP4 conc 64-80: per-rank hicache-size 400 -> 395 GB (headroom 21.5 -> 40.1 GiB under 1,547,890 MiB). Also trims the MiniMax-M3 B200 vLLM master-config comment to its rationale.
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
This was referenced Oct 8, 2026
Draft
Slurm's RealMemory on b200-nscale and b300-dsxe is below available-cpu-dram-mib, so sbatch rejected the whole-node --mem. Exclusive jobs now take all the memory Slurm has, and a step that spans the whole node uses the job's memory instead of a share cap.
Measured on each cluster's GPU nodes as the lower of RealMemory - MemSpecLimit and MemTotal less 16 GiB: h100-dgxc and b300-dsxe reserve 128 GiB through MemSpecLimit, b200-nscale configures RealMemory 15.5 GiB under MemTotal, and the AMD clusters set RealMemory to MemTotal.
The lower available-cpu-dram-mib on h100-dgxc, b200-nscale and mi300x-amd lowers the matrix total-cpu-dram-gb of their single-node DRAM offload points, and the single-node binder rejected every recipe variant still naming the old TOTAL_CPU_DRAM_GB. Each variant now names the new budget, and host pools sized from it follow their recipe's rule: - DSv4 B200 vLLM: SimpleCPU bytes per rank = budget / 8, 243.375 -> 241.5 GB. - MiniMax-M3 B200 vLLM: cpu_bytes_to_use = budget, 1038 -> 1030 GB. - MiniMax-M3 H100 vLLM: Mooncake segment = (budget GiB - 414) / 8 - 30, 119 -> 105 GB. - MiniMax-M3 MI300X vLLM: LMCache L1 = 8 x budget / 8, 1296 -> 1280 GB. - Qwen3.5 B200 SGLang: hicache-size = (budget - TP) * 15 / TP / 31, 104 -> 103. - Qwen3.5 H100 SGLang: hicache-size = budget / TP8 / 2 pools, 108 -> 100. GLM-5.2 B200's 169 GB/rank pool (~1,662 GB with the DSA indexer) and the MiniMax-M3 B200 TRT-LLM per-rank pools, which are not derived from the budget, still fit. Every pool stays 32+ GiB under its srun step's share of node DRAM.
Validation OOM-killed the Qwen3.5 MI355X FP8 TP4 CONC 64 HiCache point inside its srun step cap: 4 x 395 GB per-rank pools also exceeded the 3 TB policy share of four GPUs (1500 GB). The MiniMax-M3 B300 TRT-LLM TP2/TP4 host caches (388.6 GB per rank) exceeded theirs too (750/1500 GB). - Qwen3.5 MI355X FP8: dram-utilization 0.80 -> 1.0, hicache-size at CONC 64-80 395 -> 374 GB per rank (budget 1199 -> 1499 GB). - MiniMax-M3 B300 TRT-LLM: dram-utilization 0.8 -> 1.0, TP2/TP4 host_cache_size 388554555392 -> 374500000000/374750000000 bytes; TOTAL_CPU_DRAM_GB 599/1199/2399 -> 749/1499/2999. The budgets of two formula-sized pools now name the pool itself, as the refactor stack's '@dram' references expect: MiniMax-M3 MI300X 0.80 -> 0.7972 (TOTAL_CPU_DRAM_GB 1284 -> 1280, the LMCache L1) and MiniMax-M3 H100 0.80 -> 0.449 (1607 -> 902, eight 105 GiB Mooncake segments).
A step capped at its GPUs' share of node DRAM leaves no room for the
engine and page cache once its host pools fill the share:
qwen3.5-fp8-mi355x-sglang-agentic-mtp TP4 HiCache failed inside the step
cgroup ("Requesting 150.27 GB but only have 15.04 GB free") with its
pools at the DRAM policy limit.
Exclusive single-node jobs keep requesting --mem=0 and no longer cap
their srun steps, so pools must fit the node. On shared nodes the job and
its step still get the serving GPUs' share, which pools must fit.
…nother cluster 8c9da40 took h100-dgxc's 1,915,904 MiB from login-h100, which fronts the DSXE H100 cluster (partitions batch_1/batch_all, /data), not the DGX Cloud cluster runners.yaml calls h100-dgxc (partition hpc-gpu-1, /mnt/nfs). The 128 GiB MemSpecLimit that commit's message attributes to h100-dgxc is the DSXE cluster's; h100-dgxc was not measured. Restore 2,063,837 MiB and undo the H100 values derived from the wrong one: - Qwen3.5 H100 SGLang: hicache-size 100 -> 108 GB, TOTAL_CPU_DRAM_GB 1607 -> 1731. - MiniMax-M3 H100 vLLM: dram-utilization 0.449 -> 0.80, Mooncake segment 105 -> 119 GB, TOTAL_CPU_DRAM_GB 902 -> 1731. Every h100-dgxc and h100 matrix row and H100 recipe again matches main. CONFIGS.md gives the Slurm rule as how to measure the field, not as what every recorded value is. runners.yaml h100-dgxc has no registered runners; the DSXE H100 runners (cluster:h100-dsxe) have no runners.yaml entry.
Condense the driver's DRAM cap comment, the available-cpu-dram-mib doc and the MiniMax-M3 B200 dram-utilization comment; drop the test comment that repeated the driver's and name the test for all three of its cases; fix the MI300X LMCache recipe comment, whose L1 is now the whole 1280 GB budget rather than shares of 1284 GB. 精简驱动中的 DRAM 上限注释、available-cpu-dram-mib 文档和 MiniMax-M3 B200 的 dram-utilization 注释；删除重复驱动注释的测试注释，并按三种情形重命名该 测试；修正 MI300X LMCache 配方注释：其 L1 现为完整的 1280 GB 预算，而非 1284 GB 的分摊份额。
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks #3615 (Cam) on current
main. Single-node srt-slurm jobs make an explicit Slurm memory request based onrunners.yaml.srt-slurm.single-node-exclusiveis true (the default), the sbatch job requests--mem=0, all the memory Slurm has on the node, and its srun steps get no memory cap, so host pools must fit the node.available-cpu-dram-mib * GPU_COUNT / gpus-per-node, and each srun step is capped at it throughSRT_SRUN_OPTIONS, so jobs on shared nodes (h100-cw, h200-cw) no longer take the whole node's memory and their host pools must fit the share.qwen3.5-fp8-mi355x-sglang-agentic-mtpTP4 HiCache failed inside one (Requesting 150.27 GB but only have 15.04 GB free) with its pools at the DRAM policy limit.available-cpu-dram-mibnow records what Slurm can allocate: the lower ofRealMemory - MemSpecLimitandMemTotalless 16 GiB, measured on b200-nscale, b300-dsxe, mi300x, mi325x and mi355x. The AgentX DRAM budgets that changed (b200-nscale, mi300x) are updated in the single-node recipes'TOTAL_CPU_DRAM_GB, with derived pools (SimpleCPU bytes, LMCache L1, HiCache size) recomputed by each recipe's own rule. b300-dsxe and mi355x are above the 3 TB cap, so their budgets are unchanged.h100-dgxc(hpc-gpu-1,/mnt/nfs) currently has no registered runners, and the new DSXE H100 runners (cluster:h100-dsxe) have no runners.yaml entry.Headroom fixes
I audited all 223 single-node DRAM-offload points against their GPUs' share of node DRAM. Three configs had less than 32 GiB of it left for engine processes after the KV pool:
minimaxm3-fp4-b200-vllm-agentic-mtpTP4 (b200-nscale)dram-utilization1.0 → 0.96; pool 1082 → 1038 GBminimaxm3-fp4-b200-trt-agentic-mtpTP4host_cache_size250 → 240 GiB per rankqwen3.5-fp8-mi355x-sglang-agentic-mtpTP4 c64-80hicache-size400 → 395 GB per rankEvery other point has at least 129 GiB of headroom.
These three changes affect results. There is no perf-changelog entry yet; add one when we want to rerun them.
Verification
GPU_COUNT=4) covers: job mem0and no step cap when exclusive; step and job mem4000Mwhen shared; and no mem when the field is absent.Risk
Cluster smokes showed b200-nscale and b300-dsxe set
available-cpu-dram-mibabove Slurm's RealMemory: requesting the whole value made sbatch fail with "Memory specification can not be satisfied". That is why exclusive jobs now request--mem=0. On shared nodes the job still requests the share, so there the value must stay at or below what Slurm can allocate.