Repository navigation
Conversation
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
adibarra
changed the base branch from
refactor/srt-agentx-compose
to
refactor/composed-fingerprint
October 8, 2026 06:15
This was referenced Oct 8, 2026
Draft
adibarra
added this pull request to stack #3808
October 8, 2026 14:35
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
2 times, most recently
from
October 8, 2026 16:25
93acce6 to
e38f0b7
Compare
functionstackx
requested changes
Oct 8, 2026
| local_buffer_size: "4GB" | ||
| protocol: "rdma" | ||
| device_name: "mlx5_0,mlx5_1,mlx5_2,mlx5_3" | ||
| device_name: "@fabric.mooncake-devices" |
Collaborator
There was a problem hiding this comment.
should there be an dedicated mooncake-devices, isnt this just same values as nccl-hca
i.e. it shouldnt even be @fabric.nccl-hca
maybe just @fabric.nic_devices or soemthing generic
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
from
October 8, 2026 23:04
99bdc28 to
b8b092f
Compare
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
from
October 9, 2026 11:39
b8b092f to
9643753
Compare
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
from
October 9, 2026 17:47
9643753 to
1ac14ae
Compare
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
from
October 9, 2026 18:20
1ac14ae to
95311bd
Compare
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
from
October 9, 2026 18:48
95311bd to
9215a18
Compare
Comment on lines
+613
to
+615
| ucx-net-devices: [mlx5_0:1, mlx5_1:1, mlx5_2:1, mlx5_3:1, mlx5_4:1, mlx5_5:1, mlx5_10:1, | ||
| mlx5_11:1] | ||
| rdma-devices: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_4, mlx5_5, mlx5_10, mlx5_11] |
Collaborator
There was a problem hiding this comment.
isnt ucx device an rdma device too
i guess ucx device speciifs the port # too on the device
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
from
October 9, 2026 23:43
9215a18 to
aff8b96
Compare
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
from
October 10, 2026 07:51
aff8b96 to
528d948
Compare
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
2 times, most recently
from
October 12, 2026 08:02
6077e30 to
5cb2a51
Compare
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
2 times, most recently
from
October 12, 2026 16:01
a50e3a6 to
80dca82
Compare
Recipes stop hardcoding the cluster's network hardware. Each srt-slurm cluster profile gains a typed fabric record: ucx-net-devices, nccl-ib-hca, ib-devices, socket-ifname and mooncake-devices are lists, and mooncake-gid-index, mori-rdma-tc and mori-io-tc are numbers. A recipe value '@fabric.<name>' takes the job cluster's field after binding, lists comma-joined and numbers as decimal strings. The launcher hands both binders the cluster's rendered fabric; an unknown name, a reference inside a longer string, or a field the cluster does not set fails before submission. Nothing resolves references at plan time. Literals equal to their cluster's fact become references in 72 active recipes on gb200-nv, gb300-nv, b200-nscale, b300-dsxe, h200-dgxc and mi355x-amds. Deliberate per-recipe values stay literal: the MI355X UMBP recipe's permuted rail order, B200 Mooncake's four of eight HCAs, and empty Mooncake device lists that leave discovery to Mooncake. infx generate resolves references for a point whose runner label names one cluster and records that cluster in the manifest; otherwise references stay and the manifest's cluster is null.
Matrix generation now fails a row whose selected recipe variant (srt-recipe or eval-srt-recipe) has an '@fabric.<name>' value naming no Fabric field, or a field that a cluster the row's runner label reaches does not set. The launcher still resolves references at bind time and fingerprints hash them as written. Cluster env and srt-slurm.host-setup.env values may reference the cluster's own fabric the same way and resolve when the record loads; env keeps rejecting commas after resolution. mi355x-amds takes MORI_RDMA_TC and IBDEVICES from its fabric instead of repeating them.
The binder runs in srtctl's Python 3.10 venv, where importing infx.clusters fails on typing.Self.
nccl-ib-hca, mooncake-devices and ib-devices held the same RDMA HCA list on every cluster that set them. One Fabric field, rdma-devices, now carries that list for NCCL_IB_HCA, the Mooncake device_name, SGLang disaggregation-ib-device and IBDEVICES, and each cluster lists its HCAs once. Bound recipes are unchanged. Rows whose recipes reference the field get new fingerprints, since references are hashed as written.
prefill role. Like the other recipes' NIC lists, both read '@fabric.rdma-devices', which renders the same mlx5_0..mlx5_3 list on gb300-nv.
The workload tests already prove the multi-node binder resolves --fabric, the single-node driver case proves the launcher passes the job cluster's rendering, and the matrix tests already cover an unknown fabric name.
infx generate now raises the binder inputs' missing srt-slurm settings error instead of binding an empty fabric, and the matrix check reads the record a cluster label names instead of treating a missing one as an empty fabric.
#3701 lists the GB200 DSv4-Pro AgentX disagg Mooncake NICs literally, in the decode stores and the prefill role. Like the GB300 recipes, both read '@fabric.rdma-devices', which renders the same mlx5_0..mlx5_3 list on gb200-nv.
adibarra
force-pushed
the
refactor/srt-fabric-refs
branch
from
October 12, 2026 16:34
80dca82 to
1bc0646
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recipes no longer hardcode cluster NIC and fabric facts.
slurm.srt-slurm.fabricrecord per cluster inrunners.yaml, with lists instead of comma strings. Fields:ucx-net-devices,rdma-devices(the RDMA NIC list that NCCL, Mooncake and MORI all read),socket-ifname,mooncake-gid-index,mori-rdma-tc,mori-io-tc.UCX_NET_DEVICES: '@fabric.ucx-net-devices', valid in env, args, orservices[i]...device_name.infx generate: resolves references when the point's runner label maps to exactly one cluster.CONFIGS.mdandRECIPES.mdupdated.Left as recipe literals on purpose
host_setupcommand.device_name(4 of the 8 HCAs). Nothing shows it is the cluster fact.device_name: ''on b300 and h100 single-node recipes.network-interfacestays""(autodetect) while its recipes pineth0. Merging them would changesrtslurm.yaml.Verification
python -m infx.launch. Of the 3620 compared, all have 0 differences in the srtctl-resolved recipe, argv, env,srtslurm.yamland the AgentX client plan. 448 of those launches use references, across 6 clusters. The 2 launches not compared are the pre-existing no-selector gb200 row from refactor(srt): compose fixed-sequence recipes from fragments and bind the matrix point #3809, which fails the same way on the base.infx generatewith a label on one cluster and on two.Open questions
env.MORI_RDMA_TCandhost-setup.env.IBDEVICESduplicate the new fabric values. Should cluster env accept references?infx generate, not at plan time.@fabric.<name>in a selected variant must name a Fabric field that every reachable cluster sets.envandsrt-slurm.host-setup.envaccept@fabric.<name>; mi355x-amds uses it forMORI_RDMA_TC/IBDEVICES.