feat: [AMD Support via SemiAnalysis][1/4] AMD hardware and required cluster runtime

[cquil11]

Scope

Enable AMD hardware and make native srt-slurm jobs work on our Slurm clusters. This combines the former hardware and required-runtime PRs; functionality needed to launch the deployment is no longer hidden in a separate prerequisite layer.

• Cluster configuration owns GPU visibility, GPU-exporter defaults, network interface, mounts, model paths, and Slurm options. NVIDIA defaults are preserved. There is no vendor enum, accelerator dispatch module, or new GRES convention.
• Resolve GPU-exporter defaults once at config load, with explicit recipe overrides taking precedence.
• Apply the configured visibility variable to allocated GPU subsets; preserve full-node and backend device-binding behavior. vLLM has one set_visible_devices option.
• Respect the configured interface for control-plane and SGLang worker addresses, and forward recipe Slurm options to native routers.
• Write worker configuration inside the existing container log mount.
• Static routers that need ready HTTP workers opt into the existing health polling before launch. SGLang enables it so slow model loading cannot exhaust initial registration. Worker URLs are resolved once and reused.

Upstream scheduling escape hatches handle GRES: use_gpus_per_node_directive: false and default_sbatch_directives: {gres: "gpu:8"}. Cluster host setup uses the existing lifecycle. No engine patch, new deployment system, automatic network fallback reordering, or Dynamo installation workaround is added.

Review map

• core/schema.py, core/config.py, core/runtime.py: cluster defaults and runtime context.
• cli/mixins/worker_stage.py, backends/vllm.py: GPU binding and container-visible config.
• frontends/, services/implicit.py: configured addresses, Slurm options, and native startup.
• Example profile, dry-run, docs, and focused behavioral tests accompany those changes.

Optional deployment fork #16, Dynamo installation fork #15, and Infera fork #3 are outside this validation.

Review the stack

The upstream PRs target NVIDIA main and are manually stacked, so later upstream diffs include preceding layers. The downstream fork PRs form an actual branch-based stack (main → #1 → #11 → #7 → #14); each fork PR's Files changed tab isolates its layer for easier review.

• AMD hardware and cluster runtime: upstream #431 ↔ fork #1.
• vLLM/MoRI: upstream #432 ↔ fork #11.
• ATOM/AToMesh: upstream #433 ↔ fork #7.
• Optional observation and diagnostics: upstream #476 ↔ fork #14.

Merge in that order, then rebase the remaining layers. ATOM uses Mooncake, not MoRI; its stack position is review ordering, not an engine dependency. The core runtime does not require the observation layer, but the InferenceX launcher uses its srtctl wait command.

E2E validation

• AMD hardware/runtime #431: MI300X aggregate smoke — passed; Qwen3-0.6B, 16/16 requests. Tests a second AMD generation/cluster and direct SGLang serving without disaggregation or a router.
• AMD hardware/runtime #431: MI355X DSpark SGLang/MoRI AgentX c4 — passed; 190 measured requests, zero errors, one end-of-window cancellation. Tests separate prefill/decode nodes, native SGLang routing, and MoRI transfer.
• AMD hardware/runtime #431: R1 2P1D c8 — passed; 80/80 requests. Tests registration and routing across two prefill replicas and one decode replica with SGLang/MoRI.
• AMD hardware/runtime #431: R1 1P2D c8 — passed; 80/80 requests. Tests one prefill replica and two TP8 decode replicas with SGLang/MoRI; both decode endpoints served requests, including the reprovisioned g50 node.
• AMD hardware/runtime #431: DSpark UMBP c128 smoke — passed; five-minute measurement, 795 successful requests, zero request errors, 12 cutoff cancellations (cancellation grace expired). Tests managed UMBP startup/teardown, GPU IPC registration, DP-aware routing across eight prefill and eight decode ranks, and host-cache reuse. Uses a 64 GiB smoke cache; does not validate the production 1.5 TB cache setting.
• vLLM/MoRI #432: Colocated MiniMax M3 c4 — passed; 40/40 requests. Tests TP4 prefill and TP4 decode on one node, GPU separation, distinct listener ports, and vLLM routing with MoRI-IO.
• ATOM/AToMesh #433: DeepSeek V4 ATOM + AToMesh/Mooncake c4 — passed; 40/40 requests. Tests the ATOM engine, AToMesh router, and Mooncake KV transfer.
• All runs above use fork stack 7a7ca652 and exercise job observation #476. These are functional serving smoke tests, not accuracy, power-telemetry, or full-sweep validation.

[ishandhanani]

The net diff is small (9 files, ~180 lines) and the scope discipline is good: GPU scheduling stays on use_gpus_per_node_directive / default_sbatch_directives, and the sidecar special case in worker_stage.py was only moved, not added. The concern is that the design center, accelerator_vendor, is a closed enum whose two effects (pick a visibility env var, suppress the DCGM default) are both hardcoded in Python, when each already has a cluster-config pattern in this repo. srtslurm is meant to be portable across clusters by editing srtslurm.yaml, not by adding vendor branches. Inline comments have the specifics.

Suggested shape instead of the enum:

• visible_devices_env: str = "CUDA_VISIBLE_DEVICES" on ClusterConfig, read once into RuntimeContext, same as network_interface. An AMD profile sets ROCR_VISIBLE_DEVICES. No accelerator.py, no ValueError path, no Literal to keep in sync across three files.
• default_gpu_exporter: TelemetryExporterConfig | None on ClusterConfig, following default_health_check / default_host_setup. An AMD profile points it at AMD's device-metrics-exporter once; NVIDIA clusters keep the built-in DCGM default; observability.tachometer.dcgm_exporter in a recipe still wins. The schema property then stops reading global config.

Not in the diff, but worth noting for a PR titled base AMD support:

• src/srtctl/core/fingerprint.py:330 and :917 shell out to nvidia-smi; :353-363 call nvcc and torch.cuda.nccl.version(). On AMD the fingerprint and lockfile silently record GPU, CUDA, and NCCL as unavailable.
• srtslurm.yaml.example is not updated here (#432 adds the line).
• The CLAUDE.md hunk is a trailing-newline removal only.

Housekeeping:

• The first commit touched ~30 files and the second reverted nearly all of them. Please squash so the history matches what lands.
• CI conventional-title is red on the [AMD Support][1/3] prefix.

[ishandhanani]

bet

[ishandhanani]

Merged main into this branch (5b61ec6) so it lands cleanly on top of #486, #487, and #489. Three files conflicted and were resolved by hand; the rest of the branch is unchanged.

What moved, and why:

• should_set_visible_devices() is now a BackendProtocol member. refactor: ports through NodePortAllocator, backends through BackendProtocol #489 made every backend question go through the protocol (no getattr(backend, ...) in src/, enforced by tests/test_backend_protocol.py). The rename from this PR stays; SGLang, TRT-LLM, and the mocker answer True, vLLM answers from set_visible_devices.
• _visible_device_environment reads self.backend.should_set_visible_devices() and self.config.dynamo.sidecar directly. Same behavior as before, without the getattr defaults. Both worker launch paths build the mask from runtime.visible_devices_env through this one helper.
• Static router pre-start probe reads config.health_check and passes runtime.srun_options directly. The MockConfig test fake gained health_check: None, which is how tests opt out of the probe; the probe itself is covered by test_sglang_router_starts_only_after_workers_are_healthy.
• SGLangRouterFrontend.resolve_worker_host passes the network interface. The base class already did after refactor(frontends): the frontend class owns registration, validation, worker shape, and endpoints #487; the override now matches.

Validation on the merged head: full suite 3034 passed on Python 3.13 and 3.10, make lint scope clean, schema docs in sync, and srtctl dry-run on all 26 example recipes differs from main only by the GPU subset visibility variable line this PR adds.

[cquil11]

Passed: MI355X DeepSeek V4 DSpark smoke, SGLang + MoRI 1P1D TP4, c4, on fork stack 7a7ca652. 190 measured requests succeeded, zero request errors; one end-of-window cancellation. Throughput smoke only, no accuracy eval.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@9dbbf43). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #431   +/-   ##
=======================================
  Coverage        ?   82.06%           
=======================================
  Files           ?      139           
  Lines           ?    20623           
  Branches        ?        0           
=======================================
  Hits            ?    16925           
  Misses          ?     3698           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.