Skip to content

refactor(srt): take cluster fabric values from runners.yaml via @fabric refs #3820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
adibarra wants to merge 11 commits into from
+597 −222
Draft

refactor(srt): take cluster fabric values from runners.yaml via @fabric refs #3820

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
50e840b
refactor(srt): take cluster fabric facts from runners.yaml
adibarra Oct 8, 2026
55e4d2e
feat(srt): check recipe fabric references at matrix generation
adibarra Oct 8, 2026
deb473d
chore: reconcile srt-fabric-refs with lower branch after linearizing
adibarra Oct 8, 2026
3087b9a
fix(srt): keep the fabric reference constant importable on Python 3.10
adibarra Oct 8, 2026
d659b80
refactor(srt): merge RDMA NIC fabric fields into rdma-devices
adibarra Oct 8, 2026
2c15d3d
refactor(srt): take the GB300 DSv4 Mooncake RDMA devices from the fabric
adibarra Oct 9, 2026
3855f49
test(srt): drop fabric assertions other tests already make
adibarra Oct 9, 2026
1383b0c
refactor(srt): fail fabric lookups for clusters without a record
adibarra Oct 9, 2026
2f9a85e
docs: condense fabric reference guidance
adibarra Oct 9, 2026
ec139db
refactor(srt): take the GB200 DSv4 Mooncake RDMA devices from the fabric
adibarra Oct 12, 2026
daf9f02
refactor(srt): resolve every fabric reference with resolve_fabric
adibarra Oct 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/RECIPES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,15 @@ Recipes do not hardcode host DRAM sizes either. A DRAM point's budget, the matri

References resolve to integers, or to decimal strings in env values and argument lists (e.g. after `--l1-size-gb`), including whole values inside JSON object strings such as `kv-transfer-config`. Per-rank pools take the `per-gpu` sizes, and Mooncake `global_segment_size` takes bytes because it reads `GB` as GiB. Unknown names, embedded references and references on a point without a DRAM budget fail. SimpleCPU, LMCache CPU and `--l1-size-gb` sizes must be references; measured HiCache, TRT-LLM host-cache and Mooncake sizes may stay literal if they fit the node's share. When a backend allocates several pools from the budget, such as the KV and Mamba HiCache pools of a hybrid model, the master's `dram-utilization` sizes one of them.

Recipes do not hardcode cluster hardware facts: a value `'@fabric.<name>'`, anywhere in the recipe, becomes the job cluster's `srt-slurm.fabric` field after binding (lists comma-joined; fields in [CONFIGS.md](../../../configs/CONFIGS.md#runners)). Matrix generation fails a selected variant whose reference is embedded in a longer string, names no field, or names one that a cluster its runner label reaches does not set. The recipe fingerprint hashes references as written, so it stays cluster-independent. A value a recipe chooses on purpose, such as a permuted rail order or a subset of the devices, stays literal.

Inspect what the launcher submits without a cluster:

```bash
uv run --extra recipes infx generate --config-key 'dsr1-fp8-h200-*' --output-dir /tmp/recipes
```

It writes one bound recipe per fixed-sequence or AgentX point, validated by the pinned srtctl, plus a `manifest.json` that maps each file to its matrix point and to the one cluster its runner label schedules on (`null` when the label spans several). Multi-node points take their cluster's binder inputs (DCGM exporter port, AgentX client paths, GPUs per node); a point that needs them fails unless its runner label names one cluster and the lane accepts the row's `power`. Launch-time edits such as the job name, health-check floor and runtime `--set` values are not applied.
It writes one bound recipe per fixed-sequence or AgentX point, validated by the pinned srtctl, plus a `manifest.json` that maps each file to its matrix point and to the one cluster its runner label schedules on (`null` when the label spans several). Multi-node points take their cluster's binder inputs (DCGM exporter port, AgentX client paths, GPUs per node); a point that needs them fails unless its runner label names one cluster and the lane accepts the row's `power`. Fabric references take that cluster's facts; when the label spans several clusters they stay references. Launch-time edits such as the job name, health-check floor and runtime `--set` values are not applied.

## Recipe fingerprints

Expand Down
20 changes: 10 additions & 10 deletions .../benchmarks/multi_node/srt-slurm-recipes/dsr1/sglang/mi355x-fp4/8k1k/disagg-variants.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,11 @@ base:
HF_HUB_CACHE: /hf_hub_cache/hub
HUGGINGFACE_HUB_CACHE: /hf_hub_cache/hub
PYTHONUNBUFFERED: '1'
IBDEVICES: rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7
GLOO_SOCKET_IFNAME: eno0
NCCL_SOCKET_IFNAME: eno0
MORI_RDMA_TC: '104'
MORI_IO_TC: '104'
IBDEVICES: '@fabric.rdma-devices'
GLOO_SOCKET_IFNAME: '@fabric.socket-ifname'
NCCL_SOCKET_IFNAME: '@fabric.socket-ifname'
MORI_RDMA_TC: '@fabric.mori-rdma-tc'
MORI_IO_TC: '@fabric.mori-io-tc'
MORI_IO_SQ_BACKOFF_TIMEOUT_US: '50000'
MORI_IO_QP_MAX_SEND_WR: '16384'
MORI_IO_QP_MAX_CQE: '32768'
Expand Down Expand Up @@ -89,11 +89,11 @@ base:
HF_HUB_CACHE: /hf_hub_cache/hub
HUGGINGFACE_HUB_CACHE: /hf_hub_cache/hub
PYTHONUNBUFFERED: '1'
IBDEVICES: rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7
GLOO_SOCKET_IFNAME: eno0
NCCL_SOCKET_IFNAME: eno0
MORI_RDMA_TC: '104'
MORI_IO_TC: '104'
IBDEVICES: '@fabric.rdma-devices'
GLOO_SOCKET_IFNAME: '@fabric.socket-ifname'
NCCL_SOCKET_IFNAME: '@fabric.socket-ifname'
MORI_RDMA_TC: '@fabric.mori-rdma-tc'
MORI_IO_TC: '@fabric.mori-io-tc'
MORI_IO_SQ_BACKOFF_TIMEOUT_US: '50000'
MORI_IO_QP_MAX_SEND_WR: '16384'
MORI_IO_QP_MAX_CQE: '32768'
Expand Down
20 changes: 10 additions & 10 deletions .../benchmarks/multi_node/srt-slurm-recipes/dsr1/sglang/mi355x-fp8/8k1k/disagg-variants.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,11 @@ base:
HF_HUB_CACHE: /hf_hub_cache/hub
HUGGINGFACE_HUB_CACHE: /hf_hub_cache/hub
PYTHONUNBUFFERED: '1'
IBDEVICES: rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7
GLOO_SOCKET_IFNAME: eno0
NCCL_SOCKET_IFNAME: eno0
MORI_RDMA_TC: '104'
MORI_IO_TC: '104'
IBDEVICES: '@fabric.rdma-devices'
GLOO_SOCKET_IFNAME: '@fabric.socket-ifname'
NCCL_SOCKET_IFNAME: '@fabric.socket-ifname'
MORI_RDMA_TC: '@fabric.mori-rdma-tc'
MORI_IO_TC: '@fabric.mori-io-tc'
MORI_IO_SQ_BACKOFF_TIMEOUT_US: '50000'
MORI_IO_QP_MAX_SEND_WR: '16384'
MORI_IO_QP_MAX_CQE: '32768'
Expand Down Expand Up @@ -84,11 +84,11 @@ base:
HF_HUB_CACHE: /hf_hub_cache/hub
HUGGINGFACE_HUB_CACHE: /hf_hub_cache/hub
PYTHONUNBUFFERED: '1'
IBDEVICES: rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7
GLOO_SOCKET_IFNAME: eno0
NCCL_SOCKET_IFNAME: eno0
MORI_RDMA_TC: '104'
MORI_IO_TC: '104'
IBDEVICES: '@fabric.rdma-devices'
GLOO_SOCKET_IFNAME: '@fabric.socket-ifname'
NCCL_SOCKET_IFNAME: '@fabric.socket-ifname'
MORI_RDMA_TC: '@fabric.mori-rdma-tc'
MORI_IO_TC: '@fabric.mori-io-tc'
MORI_IO_SQ_BACKOFF_TIMEOUT_US: '50000'
MORI_IO_QP_MAX_SEND_WR: '16384'
MORI_IO_QP_MAX_CQE: '32768'
Expand Down
4 changes: 2 additions & 2 deletions ...i_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-10p1d-dep8-b256-eplb0-mtp1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -72,7 +72,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...lti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-1p3d-tep4-b32-eplb0-mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -71,7 +71,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...ulti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-1p3d-tep8-b1-eplb0-mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -70,7 +70,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...lti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-1p3d-tep8-b16-eplb0-mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -70,7 +70,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...ulti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-1p4d-tep4-b2-eplb0-mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -70,7 +70,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...ulti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-1p4d-tep4-b8-eplb0-mtp3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -73,7 +73,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...ulti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-1p4d-tep8-b1-eplb0-mtp3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -73,7 +73,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...ulti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-1p4d-tep8-b4-eplb0-mtp3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -73,7 +73,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...lti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-3p1d-dep8-b16-eplb0-mtp3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -72,7 +72,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...lti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-5p2d-dep8-b32-eplb0-mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -69,7 +69,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...ti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-6p1d-dep8-b128-eplb0-mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -69,7 +69,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...ti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-8p1d-dep8-b256-eplb0-mtp0.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -69,7 +69,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...ti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp4/8k1k/disagg-9p1d-dep8-b128-eplb0-mtp1.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
max_batch_size: 2
Expand Down Expand Up @@ -72,7 +72,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"


args:
Expand Down
4 changes: 2 additions & 2 deletions ...i_node/srt-slurm-recipes/dsr1/trtllm/b300-fp8/8k1k/disagg-1p1d-dp8-b8-eplb0-mtp3-c72.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
allreduce_strategy: AUTO
Expand Down Expand Up @@ -78,7 +78,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
allreduce_strategy: AUTO
Expand Down
4 changes: 2 additions & 2 deletions ..._node/srt-slurm-recipes/dsr1/trtllm/b300-fp8/8k1k/disagg-1p2d-tp8-b16-eplb0-mtp3-c40.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
allreduce_strategy: AUTO
Expand Down Expand Up @@ -79,7 +79,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
allreduce_strategy: AUTO
Expand Down
4 changes: 2 additions & 2 deletions ...ti_node/srt-slurm-recipes/dsr1/trtllm/b300-fp8/8k1k/disagg-1p4d-tp8-b1-eplb0-mtp3-c8.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
allreduce_strategy: AUTO
Expand Down Expand Up @@ -79,7 +79,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
allreduce_strategy: AUTO
Expand Down
4 changes: 2 additions & 2 deletions ..._node/srt-slurm-recipes/dsr1/trtllm/b300-fp8/8k1k/disagg-1p4d-tp8-b16-eplb0-mtp0-c64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
allreduce_strategy: AUTO
Expand Down Expand Up @@ -76,7 +76,7 @@ roles:
UCX_RNDV_SCHEME: "put_zcopy"
OMPI_MCA_btl: "tcp,self"
OMPI_MCA_pml: "ob1"
TRTLLM_UCX_INTERFACE: "mlx5_0:1,mlx5_1:1,mlx5_10:1,mlx5_11:1,mlx5_16:1,mlx5_17:1,mlx5_20:1,mlx5_21:1,mlx5_22:1,mlx5_23:1,mlx5_4:1,mlx5_5:1,mlx5_8:1,mlx5_9:1,mlx5_2:1,mlx5_3:1"
TRTLLM_UCX_INTERFACE: "@fabric.ucx-net-devices"

args:
allreduce_strategy: AUTO
Expand Down
Loading
Loading