Skip to content

Commit ec0e63d

Browse files
nvpohanhnvpohanh
committed
fix(dsxe): preserve Pyxis RDMA libraries for B300 AgentX
跳过 EFA 安装器的 rdma-core 升级，避免覆盖 Pyxis 挂载的 libibverbs，并确保安装失败时立即终止初始化。排除 TP8 c1 性能异常的 gpu-11 节点，追加性能变更记录。
1 parent 2236fa5 commit ec0e63d

7 files changed

Lines changed: 31 additions & 11 deletions

File tree

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/configs/install-mooncake-efa-cu13.sh‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,13 @@ set -eo pipefail
33

44
export EFA_VERSION=1.50.0
55
curl --retry 3 --retry-delay 2 -fsSL -o aws-efa-installer-${EFA_VERSION}.tar.gz \
6-
https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_VERSION}.tar.gz && \
7-
tar -xf aws-efa-installer-${EFA_VERSION}.tar.gz && \
8-
cd aws-efa-installer && \
9-
apt-get update && \
10-
./efa_installer.sh -y --skip-kmod --skip-limit-conf --no-verify && \
11-
cd .. && rm -rf aws-efa-installer* && \
6+
https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_VERSION}.tar.gz
7+
tar -xf aws-efa-installer-${EFA_VERSION}.tar.gz
8+
cd aws-efa-installer
9+
apt-get update
10+
./efa_installer.sh -y --skip-kmod --skip-limit-conf --no-verify --skip-rdma-core
11+
cd ..
12+
rm -rf aws-efa-installer*
1213
ldconfig
1314

1415
python3 -m pip uninstall --break-system-packages -y mooncake-transfer-engine-cuda13 mooncake-transfer-engine-efa-cuda13

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/b300-fp4/agentx/agg-tp4-c4-mtp.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ roles:
104104
sbatch_directives:
105105
mem: "0"
106106
cpus-per-task: "192"
107-
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-16"
107+
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-11,dsxe-sa-b300-prd0-gpu-16"
108108

109109
srun_options:
110110
mem: "0"

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/b300-fp4/agentx/agg-tp4-c8-mtp.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ roles:
104104
sbatch_directives:
105105
mem: "0"
106106
cpus-per-task: "192"
107-
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-16"
107+
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-11,dsxe-sa-b300-prd0-gpu-16"
108108

109109
srun_options:
110110
mem: "0"

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/b300-fp4/agentx/agg-tp8-c1-mtp.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ roles:
104104
sbatch_directives:
105105
mem: "0"
106106
cpus-per-task: "192"
107-
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-16"
107+
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-11,dsxe-sa-b300-prd0-gpu-16"
108108

109109
srun_options:
110110
mem: "0"

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/b300-fp4/agentx/disagg-1p1d-dep8-dep8-c240-mtp-kvoffload.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -208,7 +208,7 @@ roles:
208208
sbatch_directives:
209209
mem: "0"
210210
cpus-per-task: "192"
211-
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-16"
211+
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-11,dsxe-sa-b300-prd0-gpu-16"
212212

213213
srun_options:
214214
mem: "0"

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/b300-fp4/agentx/disagg-2p1d-dep8-dep8-c480-mtp-kvoffload.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -210,7 +210,7 @@ roles:
210210
sbatch_directives:
211211
mem: "0"
212212
cpus-per-task: "192"
213-
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-16"
213+
exclude: "dsxe-sa-b300-prd0-gpu-00,dsxe-sa-b300-prd0-gpu-01,dsxe-sa-b300-prd0-gpu-04,dsxe-sa-b300-prd0-gpu-06,dsxe-sa-b300-prd0-gpu-11,dsxe-sa-b300-prd0-gpu-16"
214214

215215
srun_options:
216216
mem: "0"

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9035,3 +9035,22 @@
90359035
- "Set stable, separate Prometheus multiprocess directories for the B300 prefill and decode roles, creating them in container setup before SGLang starts. This avoids the pinned SGLang build's temporary-directory lifetime race during scheduler metrics initialization."
90369036
- "为 B300 prefill 和 decode 角色设置稳定且分离的 Prometheus 多进程目录，并在容器初始化时预先创建，避免所用 SGLang 版本在调度器指标初始化期间出现临时目录生命周期竞争。"
90379037
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3190
9038+
9039+
- config-keys:
9040+
- dsv4-fp4-b300-dynamo-sglang-agentic-agg
9041+
- dsv4-fp4-b300-dynamo-sglang-agentic-disagg
9042+
scenario-type:
9043+
- agentic-coding
9044+
description:
9045+
- "Exclude DSXE gpu-11 from all B300 Dynamo+SGLang AgentX recipes after an otherwise identical TP8 c1 canary measured 175.69 P90 tokens/s/user there versus 291.12 on gpu-17."
9046+
- "在其他条件相同的 TP8 c1 预检中，DSXE gpu-11 的 P90 速度为每用户 175.69 token/s，而 gpu-17 为 291.12；因此在所有 B300 Dynamo+SGLang AgentX 配方中排除 gpu-11。"
9047+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3190
9048+
9049+
- config-keys:
9050+
- dsv4-fp4-b300-dynamo-sglang-agentic-disagg
9051+
scenario-type:
9052+
- agentic-coding
9053+
description:
9054+
- "Skip the EFA installer's rdma-core upgrade on DSXE so the Pyxis-provided libibverbs and provider files remain intact; fail setup immediately if EFA installation fails."
9055+
- "在 DSXE 上跳过 EFA 安装器对 rdma-core 的升级，保留 Pyxis 提供的 libibverbs 和 provider 文件；若 EFA 安装失败则立即终止初始化。"
9056+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3190

0 commit comments

Comments
 (0)