Skip to content

Commit c453dd6

Browse files
cquil11csahithiclaude[bot]
authored
[NVIDIA][no perf changes] chore: add more NVIDIA runners (H100, H200, B300, GB200, GB300) (#778)
* initial commit for multiple runners * tail logs * add cleanup to this cluster * set x for logs * testing new logging * update * enroot import instead of srun * cleanup * initial commit for multiple nv runners (h100) * initial commit for multiple nv runners (h100) pt 2 * add config name override * cleanup override in single node benchmark tmpl * cleanup override in single node benchmark tmpl * set +x * exclude broken nodes TEMP FIX * exclude broken nodes TEMP FIX pt 2 * fix single node stuff * fix single node stuff add job name runner name * fix single node stuff add job name runner name * single node: --no-container-mount-home * wait for server ready also has to wait for log file to be ready * enable multiple runners for b200-multinode-slurm (#771) Co-authored-by: Cameron Quilici <cjquilici@gmail.com> * cleanup, add b200 stuff * test perf changelog * go * go 2 * trigger test check * change slurm account * enropot import * enropot import * add b300 * add b300 * add b300 test * add b300 test * add b300 test * b300 cleanup * chore: add more gb200 runners (#767) * initial commit for multiple runners * tail logs * add cleanup to this cluster * set x for logs * testing new logging * update * enroot import instead of srun * cleanup * cleanup override in single node benchmark tmpl * add gb300 * remove test perf changelog * add gb300 cleanup * add gb300 jhob name dynamo trt * gb300 text file busy fix * result filename fix * add enroot root fs * add h200 multiple runners * add h200 multiple runners pt 2 * add nfs cleanup gb300 * flock exp * enroot import on login node * enroot import on compute node * fix model paths * remove torchao installation from gb300 dynamo sglang that accidentally snuck in there * Update .github/workflows/benchmark-multinode-tmpl.yml Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * Update runners/launch_gb200-nv.sh Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> --------- Co-authored-by: Sahithi Chigurupati <58884509+csahithi@users.noreply.github.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
1 parent 85940fc commit c453dd6

11 files changed

Lines changed: 702 additions & 428 deletions

‎.github/configs/nvidia-master.yaml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1987,7 +1987,7 @@ dsr1-fp8-h200-dynamo-trt:
19871987
image: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.8.1.post1
19881988
model: deepseek-ai/DeepSeek-R1-0528
19891989
model-prefix: dsr1
1990-
runner: h200-multinode-slurm
1990+
runner: h200-multinode
19911991
precision: fp8
19921992
framework: dynamo-trt
19931993
multinode: true
@@ -2530,7 +2530,7 @@ dsr1-fp8-h100-dynamo-trt:
25302530
image: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.8.1.post3
25312531
model: deepseek-ai/DeepSeek-R1-0528
25322532
model-prefix: dsr1
2533-
runner: h100-multinode-slurm
2533+
runner: h100-multinode
25342534
precision: fp8
25352535
framework: dynamo-trt
25362536
multinode: true
@@ -3069,7 +3069,7 @@ dsr1-fp8-h100-dynamo-sglang:
30693069
image: lmsysorg/sglang:v0.5.8-cu130
30703070
model: deepseek-ai/DeepSeek-R1-0528
30713071
model-prefix: dsr1
3072-
runner: h100-multinode-slurm
3072+
runner: h100-multinode
30733073
precision: fp8
30743074
framework: dynamo-sglang
30753075
multinode: true
@@ -6409,7 +6409,7 @@ dsr1-fp8-h200-dynamo-sglang:
64096409
image: lmsysorg/sglang:v0.5.8.post1-cu130
64106410
model: deepseek-ai/DeepSeek-R1-0528
64116411
model-prefix: dsr1
6412-
runner: h200-multinode-slurm
6412+
runner: h200-multinode
64136413
precision: fp8
64146414
framework: dynamo-sglang
64156415
multinode: true

‎.github/configs/runners.yaml‎

Lines changed: 40 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,45 @@ h100:
33
- 'h100-cr_1'
44
- 'h100-cw_0'
55
- 'h100-cw_1'
6+
- 'h100-dgxc-slurm_0'
7+
- 'h100-dgxc-slurm_1'
8+
- 'h100-dgxc-slurm_2'
9+
- 'h100-dgxc-slurm_3'
10+
- 'h100-dgxc-slurm_4'
11+
- 'h100-dgxc-slurm_5'
12+
- 'h100-dgxc-slurm_6'
13+
- 'h100-dgxc-slurm_7'
14+
- 'h100-dgxc-slurm_8'
15+
- 'h100-dgxc-slurm_9'
16+
- 'h100-dgxc-slurm_10'
17+
- 'h100-dgxc-slurm_11'
18+
h100-multinode:
19+
- 'h100-dgxc-slurm_9'
20+
- 'h100-dgxc-slurm_10'
21+
- 'h100-dgxc-slurm_11'
622
h200:
723
- 'h200-cw_0'
824
- 'h200-cw_1'
925
- 'h200-nb_0'
1026
- 'h200-nb_1'
11-
- 'h200-nb_2'
12-
- 'h200-nb_3'
13-
- 'h200-nv_0'
14-
- 'h200-nv_1'
15-
- 'h200-nv_2'
16-
- 'h200-nv_3'
27+
- 'h200-dgxc-slurm_0'
28+
- 'h200-dgxc-slurm_1'
29+
- 'h200-dgxc-slurm_2'
30+
- 'h200-dgxc-slurm_3'
31+
- 'h200-dgxc-slurm_4'
32+
- 'h200-dgxc-slurm_5'
33+
- 'h200-dgxc-slurm_6'
34+
- 'h200-dgxc-slurm_7'
35+
- 'h200-dgxc-slurm_8'
36+
- 'h200-dgxc-slurm_9'
37+
- 'h200-dgxc-slurm_10'
38+
- 'h200-dgxc-slurm_11'
39+
- 'h200-dgxc-slurm_12'
40+
- 'h200-dgxc-slurm_13'
41+
h200-multinode:
42+
- 'h200-dgxc-slurm_11'
43+
- 'h200-dgxc-slurm_12'
44+
- 'h200-dgxc-slurm_13'
1745
b200-trt:
1846
- 'b200-nb_0'
1947
- 'b200-nb_1'
@@ -22,8 +50,6 @@ b200:
2250
- 'b200-dgxc_1'
2351
- 'b200-dgxc_2'
2452
# Slurm nodes
25-
- 'b200-nv_0'
26-
- 'b200-nv_1'
2753
- 'b200-nb_0'
2854
- 'b200-nb_1'
2955
- 'b200-dgxc-slurm_0'
@@ -71,9 +97,13 @@ mi355x-disagg:
7197
- 'mi355x-amds_2'
7298
gb200:
7399
- gb200-nv_0
100+
- gb200-nv_1
101+
- gb200-nv_2
74102
b300:
75103
- 'b300-nv_0'
104+
- 'b300-nv_1'
105+
- 'b300-nv_2'
76106
gb300:
77107
- 'gb300-nv_0'
78-
h100-multinode-slurm:
79-
- 'h100-dgxc-slurm_0'
108+
- 'gb300-nv_1'
109+
- 'gb300-nv_2'

‎.github/workflows/benchmark-multinode-tmpl.yml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@ jobs:
120120
- name: Slurm cleanup (pre-run)
121121
run: &slurm-cleanup |
122122
if command -v squeue >/dev/null 2>&1; then
123-
if [[ "${{ runner.name }}" == mi355x-amds* || "${{ runner.name }}" == b200-dgxc-slurm* ]]; then
123+
if [[ "${{ runner.name }}" == mi355x-amds* || "${{ runner.name }}" == gb200-nv* || "${{ runner.name }}" == gb300-nv* || "${{ runner.name }}" == h100-dgxc-slurm* || "${{ runner.name }}" == h200-dgxc-slurm* || "${{ runner.name }}" == b200-dgxc-slurm* || "${{ runner.name }}" == b300-nv* ]]; then
124124
echo "[Slurm] Cleaning up jobs with name: ${{ runner.name }} ..."
125125
scancel --name="${{ runner.name }}" || true
126126
while [ -n "$(squeue --name='${{ runner.name }}' --noheader --format='%i')" ]; do
@@ -150,6 +150,9 @@ jobs:
150150
RESULT_FILENAME: ${{ env.EXP_NAME }}_${{ env.PRECISION }}_${{ env.FRAMEWORK }}_prefill-tp${{ env.PREFILL_TP }}-ep${{ env.PREFILL_EP }}-dp${{ env.PREFILL_DP_ATTN }}-nw${{ env.PREFILL_NUM_WORKERS }}_decode-tp${{ env.DECODE_TP }}-ep${{ env.DECODE_EP }}-dp${{ env.DECODE_DP_ATTN }}-nw${{ env.DECODE_NUM_WORKERS }}_disagg-${{ env.DISAGG }}_spec-${{ env.SPEC_DECODING }}_conc${{ join(fromJson(inputs.conc-list), 'x') }}_${{ runner.name }}
151151
run: |
152152
set -x
153+
# Export RESULT_FILENAME early so it's available for artifact uploads even if cancelled
154+
echo "RESULT_FILENAME=${RESULT_FILENAME}" >> $GITHUB_ENV
155+
153156
export ${{ join(fromJson(inputs.prefill-additional-settings), ' ') }} ${{ join(fromJson(inputs.decode-additional-settings), ' ') }}
154157
export IS_MULTINODE=true
155158
bash ./runners/launch_${RUNNER_NAME%%_*}.sh
@@ -188,6 +191,14 @@ jobs:
188191
name: bmk_${{ env.RESULT_FILENAME }}
189192
path: agg_${{ env.RESULT_FILENAME }}_*.json
190193

194+
- name: Upload logs
195+
if: always()
196+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
197+
with:
198+
name: multinode_server_logs_${{ env.RESULT_FILENAME }}
199+
path: multinode_server_logs.tar.gz
200+
if-no-files-found: ignore
201+
191202
- name: Slurm cleanup (post-run)
192203
if: always()
193204
run: *slurm-cleanup

‎.github/workflows/benchmark-tmpl.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ jobs:
108108

109109
# Cleanup SLURM resources
110110
if command -v squeue >/dev/null 2>&1; then
111-
if [[ "${{ runner.name }}" == mi355x-amds* || "${{ runner.name }}" == b200-dgxc-slurm* ]]; then
111+
if [[ "${{ runner.name }}" == mi355x-amds* || "${{ runner.name }}" == gb200-nv* || "${{ runner.name }}" == gb300-nv* || "${{ runner.name }}" == h100-dgxc-slurm* || "${{ runner.name }}" == h200-dgxc-slurm* || "${{ runner.name }}" == b200-dgxc-slurm* ]]; then
112112
echo "[Slurm] Cleaning up jobs with name: ${{ runner.name }} ..."
113113
scancel --name="${{ runner.name }}" || true
114114
while [ -n "$(squeue --name='${{ runner.name }}' --noheader --format='%i')" ]; do

‎benchmarks/benchmark_lib.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,15 @@ wait_for_server_ready() {
7777
return 1
7878
fi
7979

80+
# Wait for server log file to be created (container startup may delay this)
81+
while [ ! -f "$server_log" ]; do
82+
if ! kill -0 "$server_pid" 2>/dev/null; then
83+
echo "Server died before creating log file. Exiting."
84+
exit 1
85+
fi
86+
sleep 1
87+
done
88+
8089
# Show logs until server is ready
8190
tail -f -n +1 "$server_log" &
8291
local TAIL_PID=$!

‎benchmarks/multi_node/gptoss_fp4_gb200_dynamo-trt.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@ check_env_vars \
2424
DECODE_MAX_BATCH_SIZE \
2525
DECODE_GPU_MEM_FRACTION \
2626
MODEL_PATH \
27-
SERVED_MODEL_NAME
27+
SERVED_MODEL_NAME \
28+
RUNNER_NAME
2829

2930
if [[ "$SPEC_DECODING" == "mtp" ]]; then
3031
check_env_vars DECODE_MTP_SIZE
@@ -65,6 +66,7 @@ decode_eplb_num_slots=0
6566
sbatch --nodes=${total_nodes} \
6667
--ntasks=${total_tasks} \
6768
--ntasks-per-node=${ntasks_per_node} \
69+
--job-name="${RUNNER_NAME}" \
6870
--segment=${total_nodes} ${additional_slurm_args} \
6971
benchmark_disagg.slurm \
7072
${PREFILL_NUM_WORKERS} ${PREFILL_TP} \

0 commit comments

Comments
 (0)