You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(rccl): add JAX/MaxText Llama-3.1-405B on 4 nodes to the madengine gate - #13182
The MADEngine gate runs PyTorch/Megatron-LM only — every workload comes out of MAD's scripts/primus_megatron-lm on a rocm/primus base. JAX/MaxText drives RCCL differently and is a shipping training path, but nothing in CI runs it.
This adds llama-3.1-405b-maxtext at 4 nodes (32 GPUs) as the first JAX workload, through the same same-node A/B as the rest. The enabling work is ROCm/MAD#228, which validated it at 4N. Stacked on #12899 and #13180, based on the latter's branch; draft until both land.
Technical Details
A family key dispatches the three places the families differ: install_madengine() copies scripts/jax-maxtext and checks Primus out, generate_manifest() emits --config_path plus a distributed.primus block instead of --model_repo, and build_rccl_overlay_image() takes a different recipe. It defaults to megatron, so the existing workloads keep their path.
build_maxtext_baseline_image() is new. The Megatron baseline is the stock registry tag, which cannot work here: rocm/jax-training:maxtext-* ships MaxText but not Primus, so run.sh would not find primus-cli and the baseline phase would die before step 0. The baseline is instead base + Primus payload with stock RCCL, and the candidate is that same Dockerfile plus the swap — a literal prefix, so the payload layers are cache hits and the pair differs by the RCCL swap alone. Its manifest advertises its own Dockerfile; left at the default a node would rebuild the candidate recipe under the baseline tag and the A/B would compare the CI RCCL against itself.
The swap is discovery-based — find, per-target RUNPATH preservation including an empty one, and an unresolved-soname set gate — ported from MAD#228, because this base has none of the Primus SDK venv paths the existing branches write to. The libamd_smi bridge runs over every discovered directory, after the copy and before the post-swap measurement.
Three workload settings are failure modes from MAD#228, not taste: XLA_GPU_AUTOTUNE_LEVEL=0 (level 4 presents as a dead hang at 32 ranks, and the MAD-native XLA_AUTOTUNE_LEVEL is a no-op), per_device_batch_size=1 (OOM at 241.25 GiB otherwise), --ulimit nofile=1048576 (silent stall at Initialize clique).
metric_aliases accepts the MaxText extractor's tok_per_s_per_gpu for this workload only — Megatron emits that name per iteration and _avg for the average, so widening the global table would score llama-3.1-70b on its final iteration. required_nodes is checked before anything allocates, since the workflow's nodes input defaults to 2. The baseline stays local_image under --registry, having nothing to push to. timeout_minutes is an input again, 420 here: the script's budget covers queueing, and 5 h plus the 2-node workloads' 90 minutes of slack is what keeps a 4-node allocation from being killed while still queued.
Render every manifest and diff the Megatron ones against the base branch; build both generated Dockerfiles on a ruby gfx950 node and inspect the images; dispatch this workflow at 4 nodes.
Test Result
Both images build on rocm/jax-training:maxtext-v26.6; the base carries MaxText and no Primus, as assumed. The swap reports SWAP GATE PASS: 2 librccl replaced, preserving the empty-entry RUNPATH verbatim and leaving _rocm_sdk_devel's six pre-existing unresolved sonames unchanged — an absolute gate would have rejected that build. Both base copies need libamd_smi.so.26, so the bridge is load-bearing if the artifact needs .27.
Both Megatron workloads' A/B phase manifests — four in total — are byte-identical to the base branch.
Open until a run lands: whether TheRock's kpack librccl loads on this base, and whether JAX prints RCCL version : into the node log — verify_rccl_replacement() fails the job without that banner.
patchelf --set-rpath writes RUNPATH, so a target that carried RPATH changes tag. Resolution is unaffected and this matches MAD#228; left alone pending evidence that the loaded libamd_smi moved.
The reason will be displayed to describe this comment to others. Learn more.
The family split is in good shape. Megatron stays on its existing path, the baseline image exists because rocm/jax-training:maxtext-* ships MaxText and no Primus, the baseline manifest advertises its own Dockerfile, and the model name no longer contains a slash. The autotune hang, the 4-node OOM, and the nofile stall are each called out as the failure they came from.
Not ready. The gate cannot score this workload, and the 5-hour allocation does not fit the timeout that kills the job. Both are inline, along with the 2-node dispatch default and the --registry path.
patchelf --set-rpath writes a RUNPATH, so the _rocm_sdk_libraries copy that started as RPATH (0x0f) is RUNPATH (0x1d) afterwards. The unresolved-soname gate does not see that, because resolution still succeeds. Leaving it, matching MAD#228, is reasonable until a run shows the loaded libamd_smi changed.
The in-flight run still has to show that a TheRock kpack librccl loads on this base, and that JAX prints RCCL version : into the node log. verify_rccl_replacement() fails the job when that banner is missing, so a good training run still goes red.
The comment explaining the previous fix contained a literal empty
expression. The run block is scanned for expressions, comment lines
included, so that alone rejected the file:
(Line: 206, Col: 14): An expression was expected
Reworded, and the hazard is now stated in the comment itself.
Five fixes. The first would have failed the real run; the second doubled
the cluster cost of the weekly job.
kpack destination. The archives went to `$d/.kpack`, beside the library.
TheRock's packer puts them at the artifact root and embeds `../.kpack` for a
library under `lib/`, so a copy at `<x>/lib/librccl.so.1` looks in
`<x>/.kpack`. Both hardcoded paths elsewhere in this file agree -- they name
`_rocm_sdk_libraries/.kpack` for a library in `_rocm_sdk_libraries/lib/` --
as does the artifact-side search, which reads `rccl_lib.parent.parent`. One
`dirname` was missing, and nothing would have failed until the library was
loaded. The hardware check missed it because it fed the base image's own
non-kpack librccl back as the candidate, so the block never ran.
Saturday double-run. `therock-rccl-ci.yml` carries a daily and a Saturday
cron, both at '17 6', and derived the scope from `date -u +%u` rather than
from which schedule fired. On Saturdays both runs matched and each requested
four exclusive nodes. Keyed on `github.event.schedule` instead. This was
merely wasteful while every `test_scope == 'all'` job was small.
TFLOPS companion rows. The same widening as the throughput metric, missed
one filter over: `_TFLOPS_METRICS` accepts only `TFLOPS_per_gpu_avg`, so
every MaxText result recorded a null and the report omitted TFLOPS.
`tflops_aliases` is per workload, like `metric_aliases`.
Baseline tag identity. `{base}-primus-{ref}` was fixed across runs while the
candidate is keyed by RCCL commit, so a long-lived runner could pair a
candidate against a baseline built from an older payload -- breaking the one
invariant this pair exists to hold. The tag now carries a digest of the
recipe, and the build pulls so a mutable base tag cannot pin it to an older
layer than the candidate.
Datastore provenance. The baseline row recorded `workload_config["base_image"]`,
the stock tag, though maxtext runs the locally built Primus image.
Verified: the kpack pipeline resolves to the two `_rocm_sdk_*` roots, which
is where the existing hardcoded path points; TFLOPS now lands on the 405B
row and still does not on Megatron's; the tag changes when the recipe does;
each cron maps to one scope; both Megatron workloads' manifests and scoring
are unchanged against the base branch.
Run 38404457804 failed the swap gate on the first real artifact:
INFO: RCCL needs: libamd_smi.so.27
unresolved sonames before=6 after=6 .../_rocm_sdk_devel/lib/librccl.so.1.0
GATE FAIL: the swap introduced unresolved sonames not present before:
libamd_smi.so.27
The artifact needs libamd_smi.so.27 and the base ships .26. _soname_bridge()
drops a .27 symlink next to the library, but that copy's RUNPATH is a row of
empty entries, so the loader never searches its own directory and the symlink
is inert -- which is also why six sonames were already unresolved there before
the swap, .26 among them.
So .26 was unresolved before and .27 is unresolved after: the same library,
equally absent, under a different version. The count was identical and the
exact-name set comparison read the rename as new breakage.
The gate's contract is that the swap must not make linkage worse than it found
it, and this does not. A newly unresolved soname is now tolerated when some
version of the same library was already unresolved on that target, and logged
when it is. A library that resolved before and does not now still fails.
Verified against the four cases, with the ldd line lifted out of the loop so a
synthetic post-swap set survives: the run's own .26-to-.27 rename passes; a
previously-fine library family going missing still fails; no change passes; and
a swap that resolves something previously broken passes. Both swap forms parse
under bash -n, and the Megatron manifests are unchanged.
The first attempt at this test regenerated post.$n from ldd on a path that does
not exist, so every case reported after=0 and a regression scored PASS. The
numbers above are from the corrected harness.
…ly hold
Three review findings, and all three are the same mistake: a fix verified
where it was written rather than end to end.
Job timeout. `timeout_minutes` raised only the step. GitHub caps a job at 360
minutes unless the job says otherwise, so a 420-minute step was cut short at
the job limit and the 405B got less slack than the 2-node workloads, not more.
The knob is now `job_timeout_minutes`, applied at the job, and the step
inherits it rather than carrying a second number that could disagree. The
script is given 60 less -- about 20 minutes of observed setup runs before it,
and it needs room afterwards to report. 390 by default and 450 for the 405B
keeps every workload at the same 90 minutes of queue slack, with the 2-node
budget unchanged at 330.
Schedule. Keying the scope on `github.event.schedule` distinguished the two
crons but not the two runs: both were due at 06:17 on a Saturday, and the
concurrency group falls back to the commit for a schedule event, so with
cancel-in-progress whichever started second cancelled the other. If the
nightly won, the Saturday 'all' run -- and the weekly 405B gate -- never
happened. The nightly cron is now '17 6 * * 0-5'.
Baseline tag. The digest covered the Dockerfile text but not the mutable base
tag or the Primus tree, and on a tag hit `docker image inspect` skips the
build, so the `--pull` added to defend against a stale base never ran in the
case it was added for. The baseline now carries the same run token as the
candidate and is built unconditionally; with the payload layers shared, a
fully cached rebuild measured 2 seconds on ruby.
Verified: the job carries the timeout and no step overrides it; the budget
chain gives all three workloads 90 minutes of slack with the 2-node numbers
unchanged; the crons no longer share a day; the baseline tag carries the
candidate's token; no workflow expression is empty or arithmetic; both
Megatron workloads' manifests are unchanged.
Run 38480610532 built both images and trained the baseline -- 200.7 tok/s/GPU
at 4 nodes, the first end-to-end 405B result through this harness -- then the
candidate died on every node before step 0:
OSError: libamd_smi.so.27: cannot open shared object file
rocm_sdk/__init__.py:114 in preload_libraries
_soname_bridge() links only inside the directories that received a librccl,
which is where the Primus image keeps both and why it works there. On this
base it is the wrong set. Checked in the image rather than assumed:
_rocm_sdk_core/lib/libamd_smi.so.26 <- no librccl here
_rocm_sdk_devel/lib/libamd_smi.so{,.26,.26.5.0}
The copy that actually runs, _rocm_sdk_libraries/lib/librccl.so.1, resolves
through an RPATH reaching _rocm_sdk_core/lib -- a directory holding
libamd_smi and no librccl, so the bridge never touched it. The artifact needs
.27, nothing in the image provided it, and the preload failed.
_soname_bridge_discovered() derives the directories from the library being
bridged instead: every directory already holding some version of it, which is
exactly the set an RPATH or RUNPATH pointing at that library can reach.
Verified against the layout above: .27 now appears in _rocm_sdk_core/lib and
_rocm_sdk_devel/lib. The Megatron path keeps _soname_bridge() untouched.
The gate could not have caught this. It reported before=0 after=0 for that
copy while the runtime could not load it, so `ldd` told it nothing useful. It
now records whether ldd could read each copy at all and says so when it could
not, rather than reporting counts that prove nothing.
Both new shell blocks use if/then rather than a trailing `[ ... ] && cmd`,
which returns false when the test fails and would abort the whole RUN under
set -e -- the existing bridge has that shape and survives only because its
directories always match.
Verified: the bridge covers _rocm_sdk_core/lib against the image's real
layout; both swap forms parse; no `[ ... ] &&` statement can abort the layer;
the gate still passes a version rename, fails a genuinely new missing
library, and passes an unchanged or improved swap; Megatron manifests are
unchanged.
The earlier standalone shell checks were wrong twice: once stripping the
line continuations Docker joins, once treating a no-op bridge's false exit as
a syntax error. A control run of the production bridge failing identically is
what showed the harness, not the code, was at fault.
The baseline rebuilt with --pull every run while the candidate skipped the
pull entirely, so on a persistent runner the baseline could refresh to a newer
rocm/jax-training:maxtext-v26.6 while the candidate kept an older cached one.
The ratio would then be measuring the base image as well as RCCL.
Adding --pull to the candidate would only narrow that window: the two are
separate `docker build` invocations minutes apart, and a push in between still
splits the pair. resolve_base_digest() pulls once and resolves the tag to a
digest, and both Dockerfiles are written with that digest as their FROM, so
there is nothing left to drift. Neither build needs --pull afterwards, because
a digest cannot move.
Falls back to the tag when the base has no RepoDigest -- a locally built base
has nothing to pin to and nothing remote that could move underneath it either
-- and says so rather than pretending the pair is pinned.
Verified: both generated Dockerfiles carry the identical FROM line and the
candidate is still an exact prefix-extension of the baseline, so the payload
layers stay cache hits; the two Megatron overlay recipes are byte-identical to
the base branch and still use _soname_bridge with FROM {base_image}; both
Megatron workloads' manifests are unchanged.
The base added max-parallel: 1 so the two matrix legs stop competing for
exclusive allocations, since queue time counts against each leg's own budget.
That cannot reach this job: it sits outside the matrix, so on a Saturday it
would still queue alongside a leg while asking for four nodes instead of two,
and whichever lost would spend its budget waiting and report a timeout for
both phases -- exactly the failure max-parallel was added to prevent.
`needs` the matrix job for ordering only. `!cancelled()` keeps a failed or
skipped leg from skipping this one in turn, and the condition deliberately
does not test the matrix result: the workloads are independent, and a 70B
failure must not hide a 405B result or the reverse.
The stack conflicts when it lands in order, and this comment is why. #12899
reworded it to a one-liner after #13180 branched, so #13180 -- and this
branch on top of it -- still carry the old two-line form. Landing #12899
first and then this branch makes both sides changed in the same region.
Removing the `timeout-minutes` line from the step is substantive and stays.
The explanation of why it is gone does not need to live in that hunk, so it
moves up to the job-level timeout it is actually about, and the pre-existing
comment is left exactly as #13180 has it. Only one side then touches those
lines and the merge resolves on its own.
The real staleness is that #13180 sits on an older ci/madengine-same-node-ab;
rebasing it is that PR's call, not this one's. This just stops this branch
from adding to the collision.
Run 38602462914 lost both phases and five minutes of a four-node allocation
because two of the four nodes had no Docker daemon:
dial unix /var/run/docker.sock: connect: no such file or directory
Nothing in the log said so where anyone would look. The two healthy nodes
started their containers and sat in the launcher's rendezvous until it gave
up, so their logs -- and the job summary -- blamed the distributed runtime:
F client.h:80] Terminating process because the JAX distributed service
detected fatal errors ... DEADLINE_EXCEEDED
The real message was in a third node's log. The verdict that followed, "no
'RCCL version :' found", was equally misleading: RCCL never got the chance.
The allocation script now starts the daemon and checks it on every node
before either phase runs, prints one line per node, and exits 3 on the first
bad one so the allocation is released instead of burning its time limit.
A healthy allocation pays one srun.
This is not specific to the 405B, but it is worst there: the wider the job,
the likelier it catches a bad node. Two of four here.
Verified on the cluster with the script this code generates -- not a
re-typed copy, to avoid the quoting mistakes that invalidated two earlier
harnesses -- allocating cv350-rck-g03-c13-18, one of the nodes that broke the
run, plus a healthy one. Both report ok: the preflight does not merely detect
the daemon being down, it starts it.
Two review findings.
The timeout input was pasted straight into the run block, which executes it
as shell, in the step that carries MAD_SECRETS_HFTOKEN. The job-level
fromJSON() is not a sanitizer: it parses the leading JSON value, so a string
beginning with digits satisfies the timeout and still reaches bash as code.
It now arrives through the environment, where it is data rather than script
text, and is rejected unless it is a plain integer. Checked against
"450 ; echo PWNED", "450$(id)", "4 50", "abc" and "": only 450 is accepted.
Both MaxText image tags now carry a digest of the resolved base. The run id
alone identified them, and a workflow re-run keeps GITHUB_RUN_ID: on a second
attempt resolve_base_digest() could pick up a moved base tag, the baseline
would rebuild on it, and the candidate's `docker image inspect` would hit the
first attempt's image and skip its build. The pair would have straddled two
base images while every Dockerfile prefix still matched, and compute nodes,
which match on the tag too, would have reused the stale one. Simulated a
re-run across a moved base: all four tags change.
Megatron manifests are unchanged.
Noting, not fixing: `notify_email`, `teams_webhook`, `workload` and `nodes`
are still interpolated into the same run block, and command substitution
expands inside double quotes, so they are the same class of exposure. They
predate this branch and belong to the base PR; worth the same treatment
there.
MaxText inherits NCCL_DEBUG=WARN, but projects/rccl/src/init.cc:3778-3780 prints the version at that level only for communicator rank 0. With a communicator spanning all 32 GPUs, nodes that never host rank 0 can have no version banner. verify_rccl_replacement() rejects any node log without that banner, so successful training can still fail the gate. Set NCCL_DEBUG=INFO in this workload's environment so every participating process can print the version; this overrides the manifest's WARN default without changing Megatron.
verify_rccl_replacement() requires the "RCCL version :" banner in every
node's log and fails the job without it. At WARN only global rank 0 prints
it. From src/init.cc, around the showVersion call_once:
if ((levelMask & (1u << NCCL_LOG_INFO)) ||
((levelMask & (1u << NCCL_LOG_VERSION)) && myrank == 0))
MaxText runs one communicator across all 32 GPUs, so rank 0 sits on exactly
one of the four nodes and the other three emit nothing. The check would then
reject a run that trained perfectly -- and the message it prints, "RCCL may
not have initialized on this node", would send the next person looking in
entirely the wrong place.
Megatron does not hit this because PyTorch builds several process groups, so
some communicator's rank 0 lands on each node. That is luck rather than
design, which is why this is scoped to the workload rather than changed
globally: the two Megatron workloads keep WARN, verified by rendering all
three manifests.
SUBSYS=INIT,NET bounds the extra volume to the sections holding the banner
and the NET/IB confirmation. Both values match ROCm/MAD#228's template.
Found by review. Its line reference pointed at 3778-3780, which is config
validation; the guard is at the showVersion call sites. The substance was
right.
Run 38773329260 never reached the workload. It died in "Create Python
environment":
[Errno 122] Disk quota exceeded
while /apps still had 7.2 TB free, so this is the runner account's quota
rather than the filesystem. The cleanup step removed madengine, MAD, conda,
env, overlay_ctx and docker_builds but left `ab`, which is where the per-node
slurm_output lands -- the only part that grows with the run. 23 work
directories were sitting there; one from 12 October held 2.1 GB by itself,
almost all of it a single node log repeating
NCCL WARN Please note that ncclAllToAllv is deprecated ...
777600 times, once per all-to-all in an MoE workload.
The upload step runs immediately before cleanup and already keeps the perf
files, the manifests and every slurm_output .out/.err as an artifact with
30-day retention, so what stayed on NFS was that data a second time and
permanently. Removing `ab` loses nothing that a failure post-mortem needs;
it just moves the reading to the artifact, which is where someone without
cluster access has to look anyway.
Checked while diagnosing: the 186 MB node logs come from that deprecation
warning, which is WARN level and always on, not from NCCL_DEBUG. This
workload's INFO setting adds init-time output on four nodes, not per-collective
output, so it does not reproduce that volume.
Leaving the existing 23 directories alone -- they belong to the runner
account and predate this branch.
The phase manifests uploaded today name a local-only overlay image whose
build context lives in WORK_DIR, which the cleanup step deletes, so a night
could not be rerun by hand from its artifacts (rerunning run 38523607643 took
re-fetching RCCL from TheRock's bucket and rebuilding the context).
Write WORK_DIR/repro before the run and upload it as its own artifact:
overlay_ctx/ without the byte-identical librccl copies (37 MB for the 70B
workload), the model scripts, manifest.repro.json and
manifest.baseline.repro.json with dockerfile and dockercontext relative to the
bundle, repro.env with the refs, run id, base digest, TheRock S3 prefix and
SLURM_TMPDIR, and REPRO.md. An explicit dockercontext also keeps madengine
from falling back to a MAD checkout's ./docker.
The CI manifests and the run itself are unchanged, and a failure while
writing the bundle is logged rather than failing the gate.
upload-artifact skips dot-paths unless include-hidden-files is set, so the
repro artifact lost overlay_ctx/rccl_libs/.kpack -- the GPU kernels of a kpack
librccl -- and a rebuild from it would install a library with no kernels.
Set the flag, and pin it with a test that also checks the bundle really
holds a hidden path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
github actions
coketaste
left a comment
Motivation
The MADEngine gate runs PyTorch/Megatron-LM only — every workload comes out of MAD's
scripts/primus_megatron-lmon arocm/primusbase. JAX/MaxText drives RCCL differently and is a shipping training path, but nothing in CI runs it.This adds
llama-3.1-405b-maxtextat 4 nodes (32 GPUs) as the first JAX workload, through the same same-node A/B as the rest. The enabling work is ROCm/MAD#228, which validated it at 4N. Stacked on #12899 and #13180, based on the latter's branch; draft until both land.Technical Details
A
familykey dispatches the three places the families differ:install_madengine()copiesscripts/jax-maxtextand checks Primus out,generate_manifest()emits--config_pathplus adistributed.primusblock instead of--model_repo, andbuild_rccl_overlay_image()takes a different recipe. It defaults tomegatron, so the existing workloads keep their path.build_maxtext_baseline_image()is new. The Megatron baseline is the stock registry tag, which cannot work here:rocm/jax-training:maxtext-*ships MaxText but not Primus, sorun.shwould not findprimus-cliand the baseline phase would die before step 0. The baseline is instead base + Primus payload with stock RCCL, and the candidate is that same Dockerfile plus the swap — a literal prefix, so the payload layers are cache hits and the pair differs by the RCCL swap alone. Its manifest advertises its own Dockerfile; left at the default a node would rebuild the candidate recipe under the baseline tag and the A/B would compare the CI RCCL against itself.The swap is discovery-based —
find, per-target RUNPATH preservation including an empty one, and an unresolved-soname set gate — ported from MAD#228, because this base has none of the Primus SDK venv paths the existing branches write to. Thelibamd_smibridge runs over every discovered directory, after the copy and before the post-swap measurement.Three workload settings are failure modes from MAD#228, not taste:
XLA_GPU_AUTOTUNE_LEVEL=0(level 4 presents as a dead hang at 32 ranks, and the MAD-nativeXLA_AUTOTUNE_LEVELis a no-op),per_device_batch_size=1(OOM at 241.25 GiB otherwise),--ulimit nofile=1048576(silent stall atInitialize clique).metric_aliasesaccepts the MaxText extractor'stok_per_s_per_gpufor this workload only — Megatron emits that name per iteration and_avgfor the average, so widening the global table would score llama-3.1-70b on its final iteration.required_nodesis checked before anything allocates, since the workflow'snodesinput defaults to 2. The baseline stayslocal_imageunder--registry, having nothing to push to.timeout_minutesis an input again, 420 here: the script's budget covers queueing, and 5 h plus the 2-node workloads' 90 minutes of slack is what keeps a 4-node allocation from being killed while still queued.MAD_REF→7a9a25e, picking up fix(jax): point fetch_primus.sh at a ref that still exists MAD#279 and [Feature Request]: Support for Treating Device Functions As hsa_executable_symbol_t's #280; nothing in that range touchesscripts/primus_megatron-lmorscripts/jax-maxtext.Issue Tracking
JIRA ID: AICOMNET-426
Test Plan
Render every manifest and diff the Megatron ones against the base branch; build both generated Dockerfiles on a ruby gfx950 node and inspect the images; dispatch this workflow at 4 nodes.
Test Result
Both images build on
rocm/jax-training:maxtext-v26.6; the base carries MaxText and no Primus, as assumed. The swap reportsSWAP GATE PASS: 2 librccl replaced, preserving the empty-entry RUNPATH verbatim and leaving_rocm_sdk_devel's six pre-existing unresolved sonames unchanged — an absolute gate would have rejected that build. Both base copies needlibamd_smi.so.26, so the bridge is load-bearing if the artifact needs.27.Both Megatron workloads' A/B phase manifests — four in total — are byte-identical to the base branch.
Open until a run lands: whether TheRock's kpack
librcclloads on this base, and whether JAX printsRCCL version :into the node log —verify_rccl_replacement()fails the job without that banner.patchelf --set-rpathwrites RUNPATH, so a target that carried RPATH changes tag. Resolution is unaffected and this matches MAD#228; left alone pending evidence that the loadedlibamd_smimoved.Submission Checklist