feat: [AMD Support via SemiAnalysis][3/4] ATOM and AToMesh support

[cquil11]

Summary

Add native ROCm ATOM workers and the official AToMesh router for aggregate and prefill/decode deployments, using srt-slurm's existing endpoint allocation, Slurm launch, and health/lifecycle machinery.

• Launch atom.entrypoints.openai_server with native CLI spelling, model identity, and role-aware Mooncake transfer configuration.
• Launch atomesh with the ATOM backend, static worker topology, and worker readiness gating. Infera is not part of this layer.
• Restrict managed ATOM logical endpoints to one Slurm node each; P/D deployments can use multiple nodes and endpoints.

Unrelated fixes were extracted into standalone PRs outside this stack: vLLM hybrid-DP routing #496 and SGLang interface selection #497.

Review the stack

The upstream PRs target NVIDIA main and are manually stacked, so later upstream diffs include preceding layers. The downstream fork PRs form an actual branch-based stack (main → #1 → #11 → #7 → #14); each fork PR's Files changed tab isolates its layer for easier review.

• AMD hardware and cluster runtime: upstream #431 ↔ fork #1.
• vLLM/MoRI: upstream #432 ↔ fork #11.
• ATOM/AToMesh: upstream #433 ↔ fork #7.
• Optional observation and diagnostics: upstream #476 ↔ fork #14.

Merge in that order, then rebase the remaining layers. ATOM uses Mooncake, not MoRI; its stack position is review ordering, not an engine dependency. The core runtime does not require the observation layer, but the InferenceX launcher uses its srtctl wait command.

E2E validation is maintained only in the base PR: upstream #431 / fork #1.

---

Note

Medium Risk
Introduces a new distributed inference launch path (Mooncake KV transfer, router pairing, and orchestration branches) on top of existing Slurm machinery; mistakes could affect job startup or P/D wiring, but scope is isolated to the new engine/frontend pair.

Overview
Adds AMD ATOM as engine: atom and the AToMesh static router as frontend.type: atomesh, wired through schema v2 (atom_config, polymorphic backend loading) and the usual Slurm launch/allocator paths.

Workers run atom.entrypoints.openai_server with srtctl-owned model, HTTP port, tensor parallel size, and Mooncake --kv-transfer-config (handshake ports from topology); role args cannot override those flags. P/D and aggregate layouts are supported, but each logical endpoint must fit on one Slurm node; Dynamo sidecars are rejected. AToMesh launches atomesh launch with static prefill/decode URLs, forces --backend atom, and skips worker bootstrap/metrics scraping. served_model_name is aligned with the worker --model value (including hf: and staging). ATOM/AToMesh runs skip head etcd/NATS unlike Dynamo.

Ships an example recipe, docs/schema updates, and contract tests for commands, validation, and orchestrator behavior.

^{Reviewed by Cursor Bugbot for commit 8081c5b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cquil11]

Validated the writable lm-eval runtime fix on MI355X using the exact serving image lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260817 on healthy node mia1-p01-g16.

The smoke created a job-local Python 3.10 venv with --system-site-packages, verified the venv was the active sys.prefix, installed lm-eval successfully, and imported it from the writable venv path. Repository validation: focused lm-eval tests 7/7 pass; the full benchmark test file has 71 pass / 3 known baseline SA-bench fixture failures, identical on pristine head 10a58d2. Bash syntax and diff checks pass.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f1537b3. Configure here.}

[cquil11]

ATOM port parity fix pushed in 265dbed359ef13f92c206d214978560738a97589:

• Preserve native ATOM CLI spelling. Its original parser uses --kv_cache_dtype and --no-enable_prefix_caching, not their hyphen-normalized forms. The adapter now passes recipe keys through literally; reserved srt-slurm-owned argument checks still canonicalize aliases.
• Preserve the original worker-local proxy_ip in Mooncake KV-transfer configuration.
• Extend the existing command-contract test rather than add a new test file. All 39 ATOM/dry-run tests pass, and all four MI355X ATOM recipe worker commands parse with the original ATOM parser.

Downstream validation: SemiAnalysisAI/InferenceX#2627. This retains the upstream June image, default RDMA transport and original engine/workload settings. Hardware results are pending.

[cquil11]

Fixed the native ATOMesh startup failure observed in InferenceX run 33665959215 (all four Slurm allocations failed before worker launch because the orchestrator attempted Dynamo-only NATS/etcd startup). Commit 527525d selects that infrastructure only for the Dynamo frontend. No engine arguments, image, workload, or KV transport settings changed.

The new regression test failed before the fix for ATOMesh and passes afterward; it also confirms SGLang still skips the infrastructure and Dynamo still starts it. The relevant suite has 217 passing tests and 2 skipped. Three unrelated sa-bench shell-test failures and 19 existing test lint findings reproduce unchanged in a clean checkout of the pre-fix commit, so they are not claimed as passing.

InferenceX PR #2627 now pins this runtime through shared PR #2542. Previously validated InferenceX PRs #2555 and #2630 retain their validated heads; fresh ATOM hardware validation is next.

[cquil11]

Fixed a confirmed ATOM model-identity mismatch in fa99473. The pinned ATOM server advertises its literal --model argument, but srt-slurm supplied the path basename to lm-eval. InferenceX run 33720270808 therefore requested DeepSeek-V4-Pro while AToMesh registered deepseek-ai/DeepSeek-V4-Pro, producing HTTP 503 before worker generation. The fix preserves the HF ID or container-visible local/staged model path for ATOM only; SGLang/vLLM naming is unchanged. Three new parameterized cases in the existing ATOM test file failed before the fix and pass afterward. All 420 focused configuration, evaluation, ATOM, frontend, Slurm, dry-run, health, and accelerator tests pass. Hardware revalidation remains pending; this is not a claim of a green full sweep.

[cquil11]

Rebased onto PR #1 at 93b854c; new head 81d4627. All existing ATOM/AToMesh fixes are retained, with the host-repair hook removed and native job waiting inherited from the base.

396 focused runtime/backend/router/config/CLI tests and 76 benchmark tests passed. The three known macOS profiling.sh shell-test failures reproduce before this rebase. No serving flags, model identity, KV transport, or benchmark workload were intentionally changed by this cleanup.

Shared InferenceX adapter/entrypoint cleanup: SemiAnalysisAI/InferenceX#2542

Current ATOM hardware run remains untouched on old head 1fb90fea and runtime fa99473:
https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33782782708
A subsequent full native PR sweep is still required for the cleaned-up launcher/runtime.

[cquil11]

Validation update for this stack:

• ATOM/AToMesh: InferenceX PR port MI355X DeepSeek V4 ATOM disagg to srt-slurm InferenceX#2627 has a green native full PR sweep https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33825304569 and successfully staged results. That validated recipe remains pinned to 81d4627; its ATOM implementation was not changed by the later vLLM port fix.
• vLLM/MoRI: InferenceX PR Validate archived MI355X MiniMax M3 vLLM disaggregation with srt-slurm InferenceX#2824 now has a complete green end-to-end run https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33837838363 at this PR head 33a3d81. All nine original 8k/1k points passed (5,116 measured requests), and the full 1,319-question GSM8K eval scored 94.69% strict / 94.62% flexible with zero empty responses and no integration error.

The only runtime delta from 81d4627 to 33a3d81 is the scoped MoRI port-allocation fix from PR #1 and its regression coverage. 112 focused ATOM/vLLM/router/accelerator/dry-run tests passed. No engine patches, custom images, or graph-mode/config workarounds were used. The archived vLLM compatibility PR remains draft because of InferenceX's existing active-key changelog policy; no staging/reuse eligibility is claimed for that manual run.

[cquil11]

Closing because ATOM and AToMesh support was merged upstream in NVIDIA#433.