nv-yna · Oct 10, 2026
@@ -111,6 +111,7 @@ One worker role of the recipe: `roles.prefill`, `roles.decode`, or `roles.agg`.
 | `engine` | str \| mapping | `None` | Engine type or mapping with engine options. Set on every role when no top-level `engine` is declared; the two forms cannot be mixed, and role engines do not inherit options from each other. |
 | `container` | str \| None | `None` | Optional role image; accepts cluster container aliases. Defaults to `model.container`. |
 | `kv_events` | bool \| dict[str, Any] \| None | `None` | `true` for the default ZMQ publisher, or a mapping with `publisher` / `topic`. |
+| `mooncake_store_config` | dict[str, Any] | `{}` | TRT-LLM only: Mooncake client config keys for this role's workers (`role`, `global_segment_size`, ...), layered over the `mooncake-master` service's `options.store_config`. Pool-wide keys stay there. |
 | `sidecar` | bool \| None | `None` | Run the native engine with a Dynamo sidecar (turns on `dynamo.sidecar`); every role must agree. |
 | `critical` | bool | `True` | A worker of this role exiting fails the run. `false` keeps the run alive for probes that kill workers. |
 | `restart` | [RestartPolicy](#restartpolicy) | `RestartPolicy()` | Relaunch exited workers in place: `never`, `on-failure`, `always`, or a mapping with `policy`, `max_restarts`, `backoff_seconds`, and `max_backoff_seconds`. |
@@ -345,7 +346,7 @@ One entry of the top-level ``services:`` list.
 | `build_timeout_seconds` | int | `1800` | Kill ``build_command`` after this many seconds. |
 | `enabled` | bool | `True` | ``false`` drops the service, including an implicit one (``etcd`` / ``nats`` under the Dynamo frontend, the default exporters) declared here by name. |
 | `external` | str \| None | `None` | For discovery-plane kinds (``etcd``, ``nats``, ``mooncake-master``): use this already-running endpoint and launch nothing; the URL is what the job's processes are pointed at. |
-| `options` | dict[str, Any] | `{}` | Kind-specific settings (``nats``: ``max_payload_mb``; ``mooncake-master``: ``store_config`` and ``device_names_by_gpu`` for vLLM). Unknown keys are rejected by the kind. |
+| `options` | dict[str, Any] | `{}` | Kind-specific settings (``nats``: ``max_payload_mb``; ``mooncake-master``: ``store_config`` for vLLM and TRT-LLM, ``device_names_by_gpu`` for vLLM). Unknown keys are rejected by the kind. |
 | `metrics` | list[[ServiceMetricsConfig](#servicemetricsconfig)] | `[]` | Prometheus endpoints this service serves: one mapping or a list of ``{port, path, nodes, name}`` (``path`` defaults to ``/metrics``, ``nodes`` to ``all``). Tachometer scrapes each on every node the service runs on, or on its first node with ``nodes: first``, as endpoint ``<name>_<node>`` where ``name`` defaults to the service name. The exporter kinds declare theirs; write it for a generic service that publishes metrics, or on a ``ray`` service whose head serves a trainer's collector and router. |
 
 ### PostEvalConfig

@@ -2538,6 +2538,11 @@
           "description": "`true` for the default ZMQ publisher, or a mapping with `publisher` / `topic`.",
           "default": null
         },
+        "mooncake_store_config": {
+          "type": "object",
+          "description": "TRT-LLM only: Mooncake client config keys for this role's workers (`role`, `global_segment_size`, ...), layered over the `mooncake-master` service's `options.store_config`. Pool-wide keys stay there.",
+          "default": {}
+        },
         "sidecar": {
           "anyOf": [
             {
@@ -2924,7 +2929,7 @@
         },
         "options": {
           "type": "object",
-          "description": "Kind-specific settings (``nats``: ``max_payload_mb``; ``mooncake-master``: ``store_config`` and ``device_names_by_gpu`` for vLLM). Unknown keys are rejected by the kind.",
+          "description": "Kind-specific settings (``nats``: ``max_payload_mb``; ``mooncake-master``: ``store_config`` for vLLM and TRT-LLM, ``device_names_by_gpu`` for vLLM). Unknown keys are rejected by the kind.",
           "default": {}
         },
         "metrics": {

@@ -95,7 +95,7 @@ services:
 | `type` | `generic` | Selects a [service type](#service-types) that supplies defaults, and for the typed kinds the command. |
 | `enabled` | `true` | `false` drops the service. Declaring an implied name with `enabled: false` switches it off. |
 | `external` | none | `etcd`, `nats`, `mooncake-master` only: an address of an already-running instance. Nothing launches; the address is injected where the job's own would have been. |
-| `options` | `{}` | Kind-specific knobs; unknown keys are rejected. `nats`: `max_payload_mb`. `dcgm-exporter`, `node-exporter`: `port`, `collect_interval_ms`. `mooncake-master`: `store_config` (vLLM). |
+| `options` | `{}` | Kind-specific knobs; unknown keys are rejected. `nats`: `max_payload_mb`. `dcgm-exporter`, `node-exporter`: `port`, `collect_interval_ms`. `mooncake-master`: `store_config` (vLLM, TRT-LLM), `device_names_by_gpu` (vLLM). |
 | `command` | type default | Argv passed directly to the process. `generic` has no default, so it is required there. |
 | `args` | `[]` | Appended to `command`. Handy with typed services that supply the command. |
 | `container` | type fallback, then job container | Aliases resolve through `srtslurm.yaml` like every other container key. |
@@ -299,7 +299,7 @@ environment its process needs; the launch path is shared by every kind. Register
 | `generic` | none (required) | `after_frontend` | `false` | Launches exactly what you wrote. |
 | `etcd` | `/configs/etcd` from the job container, advertising the node's IP | `infra` | `true` | Implied by the Dynamo frontend. Placement `head`, `infra`, or `dedicated`; supports `external`. Fresh data dir on node-local `/tmp` each job. |
 | `nats` | `/configs/nats-server -js` from the job container | `infra` | `true` | Implied by the Dynamo frontend. `options.max_payload_mb` writes a server config. Same placements as etcd; supports `external`. |
-| `mooncake-master` | `mooncake_master` with the RPC, HTTP metadata, and metrics ports srtctl owns | `before_workers` | `true` | Declared by name; see [Mooncake KV Store](mooncake-kv-store.md). `args` are appended; `options.store_config` is the vLLM connector JSON. Container falls back to the job container. Supports `dedicated` and `external`. |
+| `mooncake-master` | `mooncake_master` with the RPC, HTTP metadata, and metrics ports srtctl owns | `before_workers` | `true` | Declared by name; see [Mooncake KV Store](mooncake-kv-store.md). `args` are appended; `options.store_config` is the vLLM or TRT-LLM connector JSON (a TRT-LLM role layers its own keys in `roles.<role>.mooncake_store_config`). Container falls back to the job container. Supports `dedicated` and `external`. |
 | `dcgm-exporter` | `dcgm-exporter --collect-interval=<ms> --address :9401` in `nvcr.io/nvidia/k8s/dcgm-exporter` | `after_frontend` | `false` | Implied on worker nodes while tachometer runs. Shell-less (distroless image). `options`: `port`, `collect_interval_ms`. |
 | `node-exporter` | `/bin/node_exporter` with the cpu, infiniband, and meminfo collectors on 9101 in `quay.io/prometheus/node-exporter` | `after_frontend` | `false` | Implied on worker nodes while tachometer runs. Shell-less. `options`: `port`. |
 | `process-exporter` | `configs/process-exporter -config.path <log_dir>/process-exporter.yml -web.listen-address=:9256 -threads=true ...` on the bare node | `after_frontend` | `false` | Implied on every allocated node (`placement.node: all`) while tachometer runs. Host-native from the static binary `make setup` installs; skipped with a warning when it is missing. A declared `container` switches to the image's `/bin/process-exporter` with the group file under `/logs`. `options`: `port`, `binary`. |

@@ -8,7 +8,7 @@ Small starting points, one per frontend and topology. The matrix examples serve
 | --- | --- | --- | --- |
 | SGLang | `sglang/dynamo-agg.yaml`, `sglang/dynamo-disagg.yaml` | `sglang/sglang-router-agg.yaml`, `sglang/sglang-router-disagg.yaml`, `sglang/smg-disagg.yaml` (SMG) | `sglang/sglang-direct-agg.yaml` |
 | vLLM | `vllm/dynamo-agg.yaml`, `vllm/dynamo-disagg.yaml` | `vllm/vllm-router-agg.yaml`, `vllm/vllm-router-disagg.yaml`, `vllm/vllm-router-moriio-disagg.yaml` (ROCm, MoRI-IO discovery), `vllm/smg-agg.yaml`, `vllm/smg-dep16.yaml`, `vllm/smg-disagg-grpc.yaml` (SMG; P/D over gRPC with NIXL), `vllm/llm-d-agg.yaml`, `vllm/llm-d-disagg.yaml` (llm-d EPP behind Envoy; P/D through the llm-d sidecar), `vllm/llm-d-dp-ranks.yaml` (llm-d routing to every external-LB DP rank by its KV-cache events) | `vllm/vllm-direct-agg.yaml` |
-| TRT-LLM | `trtllm/dynamo-agg.yaml`, `trtllm/dynamo-disagg.yaml`, `trtllm/dynamo-disagg-multinode.yaml` (TP16 decode across two 8-GPU nodes: one srun, one host per rank under `--distribution=arbitrary`) | `trtllm/trtllm-serve-disagg.yaml`, `trtllm/smg-agg.yaml` (SMG) | `trtllm/trtllm-serve-agg.yaml` |
+| TRT-LLM | `trtllm/dynamo-agg.yaml`, `trtllm/dynamo-disagg.yaml`, `trtllm/dynamo-disagg-multinode.yaml` (TP16 decode across two 8-GPU nodes: one srun, one host per rank under `--distribution=arbitrary`), `trtllm/dynamo-disagg-mooncake.yaml` (Mooncake Store pool) | `trtllm/trtllm-serve-disagg.yaml`, `trtllm/trtllm-serve-disagg-mooncake.yaml` (Mooncake Store pool), `trtllm/smg-agg.yaml` (SMG) | `trtllm/trtllm-serve-agg.yaml` |
 | TokenSpeed | `tokenspeed/dynamo-agg.yaml`, `tokenspeed/dynamo-disagg.yaml` | `tokenspeed/smg-agg.yaml`, `tokenspeed/smg-disagg.yaml` (SMG, gRPC engines) | |
 | Mocker | `mocker/dynamo-agg.yaml` | | |
 

@@ -0,0 +1,97 @@
+# TRT-LLM disaggregated prefill/decode with a shared Mooncake Store pool, behind
+# the Dynamo frontend.
+#
+# Same pool as trtllm-serve-disagg-mooncake.yaml: the `mooncake-master` service runs
+# the upstream `mooncake_master` on the infra node, and srtctl gives each role's
+# workers its Mooncake client config through MOONCAKE_CONFIG_PATH. `dynamo.trtllm`
+# provisions no pool itself, so that variable is how its ranks find the master; no
+# wrapper around the worker is needed. Needs a TRT-LLM build with the refined
+# Mooncake connector API (NVIDIA/TensorRT-LLM#19706). See docs/mooncake-kv-store.md.
+#
+# Aliases (`qwen3-0.6b`, `trtllm`) resolve through srtslurm.yaml; see examples/README.md.
+
+schema: 2
+name: "qwen3-0.6b-trtllm-dynamo-disagg-mooncake"
+
+slurm:
+  time_limit: "00:30:00"
+
+model:
+  path: "qwen3-0.6b"
+  container: "trtllm"
+  precision: "bf16"
+
+resources:
+  gpu_type: "h100"
+  gpus_per_node: 8
+dynamo:
+  install: false
+
+frontend:
+  type: dynamo
+  enable_multiple_frontends: false
+
+engine:
+  type: trtllm
+  served_model_name: "Qwen/Qwen3-0.6B"
+roles:
+  prefill:
+    nodes: 1
+    workers: 1
+    gpus: 1
+    env:
+      PYTHONUNBUFFERED: "1"
+      TLLM_LOG_LEVEL: "INFO"
+    args:
+      backend: "pytorch"
+      tensor_parallel_size: 1
+      max_batch_size: 8
+      max_num_tokens: 8192
+      max_seq_len: 4096
+      trust_remote_code: true
+      disable_overlap_scheduler: true
+      kv_cache_config:
+        free_gpu_memory_fraction: 0.5
+        use_kv_cache_manager_v2: true
+      cache_transceiver_config:        # KV cache manager v2 needs the Python NIXL transceiver
+        backend: "NIXL"
+        transceiver_runtime: "PYTHON"
+      kv_connector_config:
+        connector: mooncake-store
+  decode:
+    nodes: colocate
+    workers: 1
+    gpus: 1
+    env:
+      PYTHONUNBUFFERED: "1"
+      TLLM_LOG_LEVEL: "INFO"
+    args:
+      backend: "pytorch"
+      tensor_parallel_size: 1
+      max_batch_size: 64
+      max_num_tokens: 4096
+      max_seq_len: 4096
+      trust_remote_code: true
+      kv_cache_config:
+        free_gpu_memory_fraction: 0.5
+        use_kv_cache_manager_v2: true
+      cache_transceiver_config:        # KV cache manager v2 needs the Python NIXL transceiver
+        backend: "NIXL"
+        transceiver_runtime: "PYTHON"
+      kv_connector_config:
+        connector: mooncake-store
+
+services:
+  - name: mooncake-master
+    type: mooncake-master
+    options:
+      store_config:                  # model_key defaults to the served model name
+        protocol: rdma
+        global_segment_size: 8GiB
+        stage_through_host: true
+
+benchmark:
+  type: "sa-bench"
+  isl: 128
+  osl: 128
+  concurrencies: "4x8"
@@ -0,0 +1,100 @@
+# TRT-LLM disaggregated prefill/decode with a shared Mooncake Store pool, fronted
+# by `trtllm-serve disaggregated`.
+#
+# The `mooncake-master` service runs the upstream `mooncake_master` on the infra
+# node, the same master SGLang and vLLM recipes use. Both roles select TRT-LLM's
+# `mooncake-store` KV connector with no `mooncake_store` block: srtctl renders each
+# role's Mooncake client config into /logs/mooncake_store_config_<role>.json and
+# sets MOONCAKE_CONFIG_PATH on that role's workers, which TRT-LLM reads instead.
+# The context server reads and writes the pool (role: both); the generation server
+# only lends its segment (role: capacity), and `roles.decode.mooncake_store_config`
+# gives it a larger one. model_key defaults to the served model name. No
+# `trtllm-serve mooncake_master` or donor process runs. Needs a TRT-LLM build with the refined Mooncake connector
+# API (NVIDIA/TensorRT-LLM#19706). See docs/mooncake-kv-store.md.
+#
+# Aliases (`qwen3-0.6b`, `trtllm`) resolve through srtslurm.yaml; see examples/README.md.
+
+schema: 2
+name: "qwen3-0.6b-trtllm-serve-disagg-mooncake"
+
+slurm:
+  time_limit: "00:30:00"
+
+model:
+  path: "qwen3-0.6b"
+  container: "trtllm"
+  precision: "bf16"
+
+resources:
+  gpu_type: "h100"
+  gpus_per_node: 8
+frontend:
+  type: trtllm_serve
+  enable_multiple_frontends: false
+
+engine:
+  type: trtllm
+  served_model_name: "Qwen/Qwen3-0.6B"
+roles:
+  prefill:
+    nodes: 1
+    workers: 1
+    gpus: 1
+    env:
+      PYTHONUNBUFFERED: "1"
+      TLLM_LOG_LEVEL: "INFO"
+    args:
+      backend: "pytorch"
+      tensor_parallel_size: 1
+      max_batch_size: 8
+      max_num_tokens: 8192
+      max_seq_len: 4096
+      trust_remote_code: true
+      disable_overlap_scheduler: true
+      kv_cache_config:
+        free_gpu_memory_fraction: 0.5
+        use_kv_cache_manager_v2: true
+      cache_transceiver_config:        # KV cache manager v2 needs the Python NIXL transceiver
+        backend: "NIXL"
+        transceiver_runtime: "PYTHON"
+      kv_connector_config:
+        connector: mooncake-store
+  decode:
+    nodes: colocate
+    workers: 1
+    gpus: 1
+    env:
+      PYTHONUNBUFFERED: "1"
+      TLLM_LOG_LEVEL: "INFO"
+    args:
+      backend: "pytorch"
+      tensor_parallel_size: 1
+      max_batch_size: 64
+      max_num_tokens: 4096
+      max_seq_len: 4096
+      trust_remote_code: true
+      kv_cache_config:
+        free_gpu_memory_fraction: 0.5
+        use_kv_cache_manager_v2: true
+      cache_transceiver_config:        # KV cache manager v2 needs the Python NIXL transceiver
+        backend: "NIXL"
+        transceiver_runtime: "PYTHON"
+      kv_connector_config:
+        connector: mooncake-store
+    mooncake_store_config:           # this role's client keys, over the master's store_config
+      global_segment_size: 16GiB     # decode ranks lend more host memory
+
+services:
+  - name: mooncake-master
+    type: mooncake-master
+    options:
+      store_config:                  # TRT-LLM Mooncake client config keys, shared by every role
+        protocol: rdma
+        global_segment_size: 8GiB    # per rank; binary units only ("GB" is refused)
+        stage_through_host: true     # no GPUDirect RDMA needed for the KV pools
+
+benchmark:
+  type: "sa-bench"
+  isl: 128
+  osl: 128
+  concurrencies: "4x8"
@@ -40,6 +40,7 @@ Rules for `src/srtctl/backends/`. Every consumer asks a backend through `Backend
 - The master is the `mooncake-master` service (`services/`); `services/normalize.py` maps a declared entry onto the internal `backend.mooncake_kv_store` field before schema load, and `engine.mooncake_kv_store` sets that field directly.
 - srtslurm stamps `MOONCAKE_MASTER`, `MOONCAKE_TE_META_DATA_SERVER`, and `MOONCAKE_LOCAL_HOSTNAME` on every worker; `MOONCAKE_LOCAL_HOSTNAME` is the worker's own IP on `runtime.network_interface`. A value in a role's `env` pins the NIC; `MOONCAKE_MASTER` is never set by hand.
 - vLLM reads its store config from JSON: `store_config` is rendered into the file `MOONCAKE_CONFIG_PATH` names.
+- TRT-LLM's `mooncake-store` connector reads JSON too, one file per role: `TRTLLMBackend.build_mooncake_store_config(mode, ...)` renders `store_config` plus `roles.<role>.mooncake_store_config` (read through `role_mooncake_store_config`) over a per-role `role` default and a `model_key` of the served model name, `Backend.mooncake_store_configs` lists the files the orchestrator writes, and `get_mooncake_worker_env(..., mode)` sets that role's `MOONCAKE_CONFIG_PATH` on the srun task so every MPI rank inherits it. Roles select the connector with `kv_connector_config: {connector: mooncake-store}` and no `mooncake_store` block; `trtllm-serve mooncake_master` and donors are not used.
 - SGLang disaggregated recipes must set `disaggregation-transfer-backend: mooncake` in the prefill and decode `args`; the validator rejects a master without it, because workers would silently fall back to the default transport.
 - Consumers read `backend.mooncake_kv_store` / `backend.get_mooncake_worker_env(...)`; a backend without Mooncake returns `None` / `{}`.
 

@@ -15,7 +15,7 @@
 from .sglang import MooncakeKVStoreConfig, SGLangBackend
 from .tilert import TileRTBackend
 from .tokenspeed import TokenSpeedBackend
-from .trtllm import TRTLLMBackend
+from .trtllm import TRTLLMBackend, TRTLLMMooncakeKVStoreConfig
 from .vllm import VLLMBackend, VLLMFailoverConfig, VLLMMooncakeKVStoreConfig
 
 # Union type for all backend configs
@@ -39,6 +39,7 @@
     "SrunConfig",
     # TRTLLM
     "TRTLLMBackend",
+    "TRTLLMMooncakeKVStoreConfig",
     # TileRT
     "TileRTBackend",
     # TokenSpeed