chore: sync NVIDIA upstream main through 9d65c0ae

README.md

@@ -1,6 +1,6 @@
 # srtctl
 
-Command-line tool for distributed LLM inference benchmarks on SLURM clusters using SGLang, vLLM, TensorRT LLM, TileRT and AMD's ATOM. Replace complex shell scripts and 50+ CLI flags with a declarative `schema: 2` YAML recipe: `engine:` names the engine, `roles:` describes each worker role, and `services:` covers everything launched next to the workers.
+Command-line tool for distributed LLM inference benchmarks on SLURM clusters using SGLang, vLLM, TensorRT LLM, TileRT, TokenSpeed and AMD's ATOM. Replace complex shell scripts and 50+ CLI flags with a declarative `schema: 2` YAML recipe: `engine:` names the engine, `roles:` describes each worker role, and `services:` covers everything launched next to the workers.
 
 **Documentation:** https://nvidia.github.io/srt-slurm/ (source in [docs/](docs/README.md); agents: [llms.txt](https://nvidia.github.io/srt-slurm/llms.txt))
 

docs/engines.md

@@ -228,8 +228,27 @@ TRT-LLM endpoints also set `MASTER_ADDR` to the rank-zero node and use a distinc
 `MASTER_PORT` per endpoint. This overrides container hooks that infer rank zero
 from Slurm's sorted node list. Explicit recipe environment values take precedence.
 
+Every multi-node endpoint, uniform (two full nodes) or uneven, is launched with
+`--distribution=arbitrary` and one host per task in srt-slurm's node order, for
+example `--ntasks 8 --ntasks-per-node=4 --nodelist A,A,A,A,B,B,B,B` (the
+`--nodes` flag is omitted, as Slurm requires for this distribution). A plain
+`--nodelist` only names the nodes; Slurm may still order tasks by its own
+topology, which can put rank 0 on a node other than the `MASTER_ADDR` leader and
+stall the rendezvous. Single-node endpoints are unchanged. A `distribution` set in
+the cluster's or the role's `srun_options` is replaced for such an endpoint, with a
+warning in the job log.
+
 **Other TRT-LLM launch facts**: TRT-LLM supports prefill, decode, and aggregated roles, uses MPI-style launching (one srun per endpoint with all of its nodes) through `trtllm-llmapi-launch`, and sets `TRTLLM_EPLB_SHM_NAME` to a unique UUID per endpoint.
 
+## TokenSpeed with Dynamo
+
+Use `engine: tokenspeed` with `frontend.type: dynamo` to launch
+`python3 -m dynamo.tokenspeed` workers, aggregated or prefill/decode. TokenSpeed flags
+go under `roles.<role>.args`; srtctl sets the model, host, ports and multi-node
+rendezvous. See [TokenSpeed](tokenspeed.md) and the
+[aggregated](https://github.com/NVIDIA/srt-slurm/blob/main/examples/tokenspeed/dynamo-agg.yaml) and
+[prefill/decode](https://github.com/NVIDIA/srt-slurm/blob/main/examples/tokenspeed/dynamo-disagg.yaml) recipes.
+
 ## ATOM with AToMesh
 
 Use `engine: atom` with `frontend.type: atomesh` to launch native

docs/schema-reference.md

@@ -15,7 +15,7 @@ Top-level keys of a recipe YAML.
 | `resources` | [ResourceConfig](#resourceconfig) | required | GPU type, GPUs per node, and allocation knobs. The worker topology is `roles`. |
 | `schema` | one of `2` | required | Recipe schema version. Every recipe declares `schema: 2`; a recipe without it is the pre-2.0 layout and does not load (see [legacy-v1.md](legacy-v1.md) and `srtctl migrate`). |
 | `slurm` | [SlurmConfig](#slurmconfig) | `SlurmConfig()` | Slurm account, partition, and time limit; unset values come from srtslurm.yaml. |
-| `engine` | str \| mapping | optional when every role sets `engine` | The engine type (`atom`, `sglang`, `tilert`, `trtllm`, `vllm`, `mocker`) as a string, or a mapping with `type` plus the engine-wide knobs listed under [Engine types](#engine-types). |
+| `engine` | str \| mapping | optional when every role sets `engine` | The engine type (`atom`, `sglang`, `tilert`, `tokenspeed`, `trtllm`, `vllm`, `mocker`) as a string, or a mapping with `type` plus the engine-wide knobs listed under [Engine types](#engine-types). |
 | `roles` | dict[str, [RoleConfig](#roleconfig)] | `{}` | One block per worker role (`prefill`, `decode`, `agg`): nodes, workers, GPUs, env, engine args. |
 | `frontend` | [FrontendConfig](#frontendconfig) | `FrontendConfig()` | The HTTP entry point in front of the workers (Dynamo frontend, router, nginx) and where it runs. |
 | `dynamo` | [DynamoConfig](#dynamoconfig) | `DynamoConfig()` | Which Dynamo to install, its request/event planes, and native sidecar mode. |
@@ -44,7 +44,7 @@ Three vocabularies carry the engine, the topology, and the placement. `engine` a
 
 ### engine
 
-`engine: <type>` or `engine: {type: <type>, ...}`. `type` is one of `atom`, `sglang`, `tilert`, `trtllm`, `vllm`, `mocker`; the remaining keys are that engine's knobs, listed under [Engine types](#engine-types).
+`engine: <type>` or `engine: {type: <type>, ...}`. `type` is one of `atom`, `sglang`, `tilert`, `tokenspeed`, `trtllm`, `vllm`, `mocker`; the remaining keys are that engine's knobs, listed under [Engine types](#engine-types).
 Use either one top-level engine or an explicit engine on every role, never both. Role engines do not inherit options from each other.
 
 ### roles
@@ -700,6 +700,16 @@ Launch TileRT's decode server with recipe-owned model and transport settings.
 | `type` | one of `'tilert'` | `'tilert'` | Engine type discriminator. |
 | `served_model_name` | str \| None | `None` | Model name the server reports to clients; unset uses the default served name. |
 
+### TokenSpeedBackend
+
+`engine.type: tokenspeed`
+
+Launch TokenSpeed workers behind the Dynamo frontend or a static router such as SMG.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `type` | one of `'tokenspeed'` | `'tokenspeed'` | Engine type discriminator. |
+
 ### TRTLLMBackend
 
 `engine.type: trtllm`

docs/schema/recipe.schema.json

@@ -1810,6 +1810,7 @@
                 "atom",
                 "sglang",
                 "tilert",
+                "tokenspeed",
                 "trtllm",
                 "vllm",
                 "mocker"
@@ -1824,6 +1825,9 @@
             {
               "$ref": "#/$defs/TileRTBackend"
             },
+            {
+              "$ref": "#/$defs/TokenSpeedBackend"
+            },
             {
               "$ref": "#/$defs/TRTLLMBackend"
             },
@@ -1837,7 +1841,7 @@
               "type": "null"
             }
           ],
-          "description": "The engine type (`atom`, `sglang`, `tilert`, `trtllm`, `vllm`, `mocker`) as a string, or a mapping with `type` plus the engine-wide knobs listed under [Engine types](#engine-types).",
+          "description": "The engine type (`atom`, `sglang`, `tilert`, `tokenspeed`, `trtllm`, `vllm`, `mocker`) as a string, or a mapping with `type` plus the engine-wide knobs listed under [Engine types](#engine-types).",
           "default": null
         },
         "roles": {
@@ -2026,6 +2030,7 @@
                 "atom",
                 "sglang",
                 "tilert",
+                "tokenspeed",
                 "trtllm",
                 "vllm",
                 "mocker"
@@ -2040,6 +2045,9 @@
             {
               "$ref": "#/$defs/TileRTBackend"
             },
+            {
+              "$ref": "#/$defs/TokenSpeedBackend"
+            },
             {
               "$ref": "#/$defs/TRTLLMBackend"
             },
@@ -2053,7 +2061,7 @@
               "type": "null"
             }
           ],
-          "description": "The engine type (`atom`, `sglang`, `tilert`, `trtllm`, `vllm`, `mocker`) as a string, or a mapping with `type` plus the engine-wide knobs listed under [Engine types](#engine-types).",
+          "description": "The engine type (`atom`, `sglang`, `tilert`, `tokenspeed`, `trtllm`, `vllm`, `mocker`) as a string, or a mapping with `type` plus the engine-wide knobs listed under [Engine types](#engine-types).",
           "default": null
         },
         "roles": {
@@ -2457,6 +2465,7 @@
                 "atom",
                 "sglang",
                 "tilert",
+                "tokenspeed",
                 "trtllm",
                 "vllm",
                 "mocker"
@@ -2471,6 +2480,9 @@
             {
               "$ref": "#/$defs/TileRTBackend"
             },
+            {
+              "$ref": "#/$defs/TokenSpeedBackend"
+            },
             {
               "$ref": "#/$defs/TRTLLMBackend"
             },
@@ -3539,6 +3551,20 @@
       },
       "description": "Launch TileRT's decode server with recipe-owned model and transport settings."
     },
+    "TokenSpeedBackend": {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "type": {
+          "enum": [
+            "tokenspeed"
+          ],
+          "description": "Engine type discriminator.",
+          "default": "tokenspeed"
+        }
+      },
+      "description": "Launch TokenSpeed workers behind the Dynamo frontend or a static router such as SMG."
+    },
     "VLLMBackend": {
       "type": "object",
       "additionalProperties": false,