Skip to content

feat: run Docker launcher jobs across multiple nodes - #46

Open
cquil11 wants to merge 1 commit into
feat/docker-launcher-single-nodefrom
feat/docker-launcher
Open

cquil11 wants to merge 1 commit into
feat/docker-launcher-single-nodefrom
feat/docker-launcher

Conversation

@cquil11

@cquil11 cquil11 commented Oct 6, 2026 •

Copy link
Copy Markdown

Stacked on #47 (single-node Docker launcher). This PR adds only the multi-node part: DockerLauncher runs a job's processes across several Docker hosts.

Design

  • Nodes. docker_hosts in srtslurm.yaml lists the job's nodes and defaults to this machine. The orchestrator runs where srtctl apply runs and carves docker_hosts into roles with the same Nodes.from_nodelist a Slurm nodelist uses, so aggregated, multi-node aggregated and disaggregated recipes need no changes. validate() now checks the recipe's node count against docker_hosts; the single-node checks on dedicated frontend, benchmark or infra nodes and on service pools are removed, since with several hosts those are just more nodes.
  • ssh fan-out. A LaunchSpec runs one task on each node in its nodelist. Each task is a docker run --gpus all --network host --ipc host container, started directly on this machine and through ssh -n -o BatchMode=yes <node> on any other node. A launch that targets several nodes (the exporters) starts one task per node in parallel, and %N in the log path becomes the node name, as with srun.
  • Per-node IP lookup. With one host, every address is 127.0.0.1. With several, node_ip runs the existing ip_utils/get_node_ip.sh lookup on the target node over ssh, so network_interface behaves as it does under Slurm.
  • Per-host kill and cleanup. A named step is stopped with docker kill --signal=... on its node, because killing the local ssh or docker run client does not stop the container. When the job script exits, any container labelled srtctl.job=<job_id> that is still running is removed on every host.
  • Remote host processes. A container-less step on another node (a host-native exporter) runs under ssh -tt, so it gets SIGHUP when its ssh client is signalled; host steps are stopped by signalling their local process group. Without this, the remote process kept running after the job.
  • Requirements. This machine needs passwordless ssh to every other host. Every path the job mounts (the model, outputs/, the srtctl checkout) must exist at the same path on every host, as on a Slurm cluster's shared filesystem.

Validation

Run on h200, on two nodes held with salloc (worker-12 and worker-13). Each node had Docker 29.2.1 and NVIDIA H200 GPUs. srtctl apply ran on worker-12 with launcher: docker. Model: Qwen3-0.6B (bf16). Image: lmsysorg/sglang:v0.5.20. Benchmark: sa-bench at ISL/OSL 128/128, concurrencies 4 and 8. All three runs used tree 51a61e2b, which is identical to this branch's tree after the merge of #47.

Cluster Scenario docker_hosts Outcome Notes
h200 Single-node agg (TP1) worker-12 Pass 40/40 and 80/80 requests completed; no containers left
h200 2-node agg (one TP2 engine spanning both nodes) worker-12, worker-13 Pass --dist-init-addr set to worker-12's node IP; 40/40 and 80/80 requests completed; no containers or host exporters left on either node
h200 1P1D disagg (prefill on worker-12, decode on worker-13, sglang-router) worker-12, worker-13 Pass Mooncake KV transfer between the nodes; 40/40 and 80/80 requests completed; no containers left

A failed run (the prefill worker exited on a startup flag) stopped the job and removed both nodes' containers.

@cquil11
cquil11 marked this pull request as ready for review October 6, 2026 15:58
cursor[bot]

This comment was marked as outdated.

@cquil11 cquil11 changed the title feat: add a multi-node Docker launcher feat: run Docker launcher jobs across multiple nodes Oct 6, 2026
@cquil11
cquil11 changed the base branch from main to feat/docker-launcher-single-node October 6, 2026 16:20
@cquil11
cquil11 added this pull request to stack #48 October 6, 2026 16:29
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
cquil11 force-pushed the feat/docker-launcher branch from b2b401e to bd2b94d Compare October 12, 2026 16:37
@github-actions

Copy link
Copy Markdown

src/srtctl diff: +151 -85

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant