Repository navigation
chore: sync NVIDIA upstream main through 9d65c0ae - #54
Merged
Merged
Conversation
…IA#583) * fix(trtllm): pin MPI rank order on uniform multi-node endpoints A multi-node TRT-LLM endpoint spanning full nodes passed Slurm a plain --nodelist, which names the nodes but not the task order: Slurm may lay ranks out by its own topology, putting rank 0 on a node other than the leader whose IP is MASTER_ADDR, and the rendezvous then stalls at startup. Uneven layouts already used --distribution=arbitrary with one host per task; this extends that to every multi-node endpoint (e.g. --ntasks=8 --ntasks-per-node=4 --nodelist=A,A,A,A,B,B,B,B, no --nodes). Single-node endpoints are unchanged. Tests cover the rendered srun and the relaunch path; docs/engines.md describes the layout. Signed-off-by: Pei Li <peili@nvidia.com> * fix(trtllm): review follow-ups: num_nodes condition, warn on replaced distribution, two-node example and snapshot Use num_nodes > 1 for the multi-node check; log a warning naming a cluster- or role-configured srun_options.distribution that the explicit per-rank layout replaces, with a test; assert MASTER_ADDR in the rendered command; add examples/trtllm/dynamo-disagg-multinode.yaml with its launch snapshot and README row so the layout is visible in tests/snapshots/launch; docs show the flags as rendered. Signed-off-by: Pei Li <peili@nvidia.com> * fix(trtllm): make the two-node example an H100 example The suite treats every examples/{sglang,vllm,trtllm}/*.yaml as an H100 recipe (8 GPUs per node) and pins their count. The two-node example becomes TP8 prefill on one node and TP16 decode across two 8-GPU nodes; the snapshot shows sixteen ranks, eight hosts each; the curated count is 17. Signed-off-by: Pei Li <peili@nvidia.com> --------- Signed-off-by: Pei Li <peili@nvidia.com>
…#580) * feat(tokenspeed): add a native TokenSpeed backend Add `engine: tokenspeed`, which launches Dynamo's TokenSpeed backend (`python3 -m dynamo.tokenspeed`) behind the Dynamo frontend, aggregated or prefill/decode with Mooncake. srtctl sets the model, served model name, host, multi-node rendezvous and P/D flags. `--port` comes from a new per-node TOKENSPEED_PORTS kind spaced 1024 apart, because TokenSpeed scans for its NCCL port above it, and `--dist-init-addr` from a DIST_INIT_PORTS block sized for the control-plane cluster TokenSpeed binds on the leader. Adds examples, launch snapshots, docs/tokenspeed.md and tests. Signed-off-by: Cam Quilici <cjquilici@gmail.com> * docs(tokenspeed): name the first Dynamo nightly with TokenSpeed prefill/decode Signed-off-by: Cam Quilici <cjquilici@gmail.com> * docs(tokenspeed): link the examples by URL for the strict docs build Signed-off-by: Cam Quilici <cjquilici@gmail.com> * feat(tokenspeed): run TokenSpeed gRPC engines behind SMG Signed-off-by: Cam Quilici <cjquilici@gmail.com> * docs: link the SMG and TokenSpeed examples by URL for the strict docs build Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(tokenspeed): inherit the Prometheus metrics path from Backend Signed-off-by: Cam Quilici <cjquilici@gmail.com> * chore(tokenspeed): use the shared WorkerMode and the NVIDIA copyright header Signed-off-by: Cam Quilici <cjquilici@gmail.com> * feat(tokenspeed): point the Mooncake Store L3 cache at a mooncake-master service Signed-off-by: Cam Quilici <cjquilici@gmail.com> * fix(tokenspeed): skip the gRPC engine warmup on prefill and decode workers Signed-off-by: Cam Quilici <cjquilici@gmail.com> * docs(tokenspeed): document DEP and TEP, KV cache offload and the P/D warmup skip Signed-off-by: Cam Quilici <cjquilici@gmail.com> * test(tokenspeed): update launch snapshots for the P/D warmup skip Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(backends): share the Mooncake worker environment through Backend Signed-off-by: Cam Quilici <cjquilici@gmail.com> * refactor(tokenspeed): match existing worker launch selection * fix(tokenspeed): pass mapping-valued args as JSON Signed-off-by: Cam Quilici <cjquilici@gmail.com> --------- Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> Co-authored-by: Yuewei Na <nv-yna@users.noreply.github.com>
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com> Co-authored-by: Yuewei Na <nv-yna@users.noreply.github.com>
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge NVIDIA/srt-slurm main through
9d65c0ae, including TokenSpeed support and the TRT-LLM MPI rank-order fix. Preserve the fork-specific changes.