feat: support the LMCache server on ATOM and SGLang

docs/config-reference.md

@@ -56,6 +56,28 @@ srt-slurm owns the model path, HTTP port, tensor parallel size, and KV-transfer
 contract, so recipes cannot override those arguments. See the complete
 [ATOM/AToMesh recipe](../examples/atom/atomesh-disagg.yaml).
 
+To run another KV connector next to the Mooncake one (for example ATOM's in-process
+LMCache CPU offload on prefill), list it under `extra-kv-connectors` in that role's
+args. srtctl keeps generating the Mooncake entry, with its handshake port, and wraps
+both in ATOM's `multi` connector. An aggregate role with one extra connector runs it
+on its own. An `lmcache_mp` connector (ATOM's client for the `lmcache-server`
+service) dials the server on its own node, port 8750, unless its
+`kv_connector_extra_config` sets `lmcache.mp.port` or `lmcache.mp.server_urls`.
+
+```yaml
+roles:
+  prefill:
+    env:
+      PYTHONHASHSEED: "0"      # LMCache prefix hashes must agree across offload workers
+    args:
+      extra-kv-connectors:
+        - kv_connector: lmcache_offload
+          kv_role: offload
+          lmcache.local_cpu: true
+          lmcache.max_local_cpu_size: 180   # GB per worker
+          lmcache.chunk_size: 256
+```
+
 ```yaml
 schema: 2                      # Required: recipe layout version
 name: "my-benchmark"           # Required: job name

docs/schema-reference.md

@@ -623,7 +623,7 @@ vLLM protocol - implements BackendProtocol.
 |---|---|---|---|
 | `type` | one of `'vllm'` | `'vllm'` |  |
 | `set_visible_devices` | bool | `False` | Use an environment mask instead of the engine's --device-ids option. |
-| `connector` | str \| None | `'nixl'` | Default KV connector: "nixl", "lmcache", "kvbm", "moriio", or a raw JSON string for --kv-transfer-config. Can be overridden per role by setting "connector" in roles.<role>.args; connector_for_mode resolves it. "moriio" (ROCm MoRI-IO) registers workers with the vLLM Router and needs frontend.type: vllm-router. dynamo 1.0.0+: translated to --kv-transfer-config (--connector was removed). |
+| `connector` | str \| None | `'nixl'` | Default KV connector: "nixl", "lmcache", "lmcache-mp", "kvbm", "moriio", or a raw JSON string for --kv-transfer-config. Can be overridden per role by setting "connector" in roles.<role>.args; connector_for_mode resolves it. "moriio" (ROCm MoRI-IO) registers workers with the vLLM Router and needs frontend.type: vllm-router. dynamo 1.0.0+: translated to --kv-transfer-config (--connector was removed). |
 | `failover` | [VLLMFailoverConfig](#vllmfailoverconfig) \| None | `None` | Shadow engine recovery: when set, every worker runs shadow_engines standby engines on its GPUs next to an implied `gms` service that owns the weights. Dynamo frontend only. |
 | `allow_prefill_decode_colocation` | bool | `False` | Allow prefill and decode workers to share one node when the combined GPU request fits within gpus_per_node. Defaults off to preserve existing P/D node separation. |
 | `allow_prefill_decode_colocation_across_nodes` | bool | `False` | Extend P/D colocation to multi-node topologies. When enabled together with allow_prefill_decode_colocation, workers are packed contiguously across the minimum number of nodes instead of reserving separate P/D node pools. Defaults off to preserve the original one-node-only policy. |

docs/services.md

@@ -48,7 +48,7 @@ directory. `examples/features/services.yaml` is a runnable version of this.
 ```yaml
 services:
   - name: my-sidecar             # required, unique across the list
-    type: generic                # generic (default) | etcd | nats | mooncake-master | dcgm-exporter | node-exporter | mooncake-store
+    type: generic                # generic (default) | etcd | nats | mooncake-master | dcgm-exporter | node-exporter | mooncake-store | lmcache-server
     enabled: true                # false drops the service (the way to switch an implied one off)
     external: null               # typed kinds only: use an instance that already runs at this address
     command:                     # argv, not shell-interpreted; required for generic
@@ -104,10 +104,10 @@ services:
 | `placement.pool` | none | Run on the nodes another service owns, one instance per node of that pool. Replaces `node`. See [pools.md](pools.md). |
 | `placement.per` | `node` | `worker`: one instance per engine worker on each placed node instead of one per node, attached to that worker (its `CUDA_VISIBLE_DEVICES`, the `{worker_*}` placeholders). See [Placement](#placement). |
 | `nodes` | none | Whole nodes this service owns: its pool, added to the allocation after the engine roles' nodes, in declaration order. Any number of services may own nodes, next to engine roles or without them. An owner is placed on its own pool (`placement.node: workers`). Not supported with `resources.het_jobs`. See [pools.md](pools.md). |
-| `start` | type default | `etcd`, `nats`: `infra`. `mooncake-master`, `mooncake-store`: `before_workers`. `generic` and the exporters: `after_frontend`. |
+| `start` | type default | `etcd`, `nats`: `infra`. `mooncake-master`, `mooncake-store`, `lmcache-server`: `before_workers`. `generic` and the exporters: `after_frontend`. |
 | `readiness` | type default | One probe per node: `port` / `tcp`, `http`, or `log`, plus `timeout_seconds` and `interval_seconds`. The typed kinds gate on their well-known ports when no probe is written. See [Start Order and Readiness](#start-order-and-readiness). Timing out terminates what this stage started and fails the job. |
 | `inherit_discovery_env` | `true` | Inject the same `ETCD_ENDPOINTS` / `NATS_SERVER` the Dynamo frontend gets. |
-| `critical` | type default | `generic`: `false`. `mooncake-store`: `true`. |
+| `critical` | type default | `generic`: `false`. `mooncake-store`, `lmcache-server`: `true`. |
 | `terminal` | `false` | This service is the job's run: the job ends when every instance of every terminal service has exited, with the worst exit code as the job's. The recipe has no benchmark step (`benchmark.type` stays `manual`); combining the two is refused. See [pools.md](pools.md#ending-the-job-with-a-pool). |
 | `metrics` | kind default | Prometheus endpoints the service serves: one `{port, path, nodes, name}` or a list of them (`path` defaults to `/metrics`, `nodes` to `all`, `name` to the service name and is required when there are several). Tachometer scrapes each on every node the service runs on, pools included, as endpoint `<name>_<node>`, and the rows carry `service=<service>`; `nodes: first` scrapes only the service's first node (a cluster head that serves a collector or a router). The exporter kinds declare theirs; write it for anything else that publishes metrics. |
 | `preamble` | none | Shell run after the environment is exported and before `command`. |
@@ -302,6 +302,7 @@ environment its process needs; the launch path is shared by every kind. Register
 | `node-exporter` | `/bin/node_exporter` with the cpu, infiniband, and meminfo collectors on 9101 in `quay.io/prometheus/node-exporter` | `after_frontend` | `false` | Implied on worker nodes while tachometer runs. Shell-less. `options`: `port`. |
 | `process-exporter` | `configs/process-exporter -config.path <log_dir>/process-exporter.yml -web.listen-address=:9256 -threads=true ...` on the bare node | `after_frontend` | `false` | Implied on every allocated node (`placement.node: all`) while tachometer runs. Host-native from the static binary `make setup` installs; skipped with a warning when it is missing. A declared `container` switches to the image's `/bin/process-exporter` with the group file under `/logs`. `options`: `port`, `binary`. |
 | `mooncake-store` | `python -m mooncake.mooncake_store_service` | `before_workers` | `true` | Requires a `mooncake-master` entry. Container falls back to the master's. Injects the master's address. |
+| `lmcache-server` | `lmcache server` bound to the RPC and HTTP ports srtctl owns (8750, 8751) | `before_workers` | `true` | One per worker node (`placement.node: workers`); vLLM ranks reach it over localhost with `connector: lmcache-mp`; SGLang workers started with `enable-lmcache` get `LMCACHE_MP_HOST`/`LMCACHE_MP_PORT` pointing at it unless the recipe sets `lmcache-config-file` or `LMCACHE_MP_HOST`. `args` are appended (`--l1-size-gb`, `--chunk-size`, `--max-workers`, ...). Ready when `GET /healthcheck` answers; LMCache must be installed in the job container. |
 | `ray` | `ray start --head ...` on the first instance, `ray start --address=<head>:6379 ...` on the rest, both `--block` | `before_workers` | `true` | One Ray cluster across the service's nodes; see [Ray cluster](#ray-cluster). Placement `workers` (default), `head`, or `all`. `options`: `port` (GCS, 6379), `dashboard_port` (8265), `num_gpus` (the node's count). `args` are appended to every `ray start`. |
 | `gms` | one `python3 -m gpu_memory_service --device k` per GPU of the worker, supervised by bash, in the job container | `before_workers` | `true` | Implied by `engine.failover`; see [Shadow Engine Recovery](shadow-engine-recovery.md). `placement.per: worker` (required): one instance per vLLM worker in that worker's device view, sockets and lock file under `<shared_dir>/srtctl-<job>/<role>_<index>`. Ready when its log says `GMS ready:`; `readiness.timeout_seconds` (default 120) also bounds the servers' startup. |
 

examples/README.md

@@ -31,6 +31,9 @@ Every example is written in the 2.0 layout: `engine:` names the engine (a string
 | `features/mlperf-client.yaml` | `benchmark.type: custom` driving the MLPerf inference-endpoint client in its own image; placeholder paths, a reference rather than a runnable example |
 | `features/infra-services.yaml` | etcd and NATS as declared services on a dedicated node with a NATS payload limit; the implied exporters overridden or switched off |
 | `features/dynamo-source.yaml` | `dynamo.source:` building Dynamo from a git tag (or a PR head via `--set dynamo.source.rev=refs/pull/<n>/head`), pinned to a commit at submit |
+| `features/lmcache-server.yaml` | `services[].type: lmcache-server` plus `connector: lmcache-mp`: an LMCache DRAM tier, one server per worker node started before the workers and gated on its healthcheck; LMCache flags go under `args`. Needs a vLLM image that ships LMCache (the `vllm-lmcache` alias). See [../docs/services.md](../docs/services.md#service-types) |
+| `features/lmcache-server-disagg.yaml` | The same tier on the prefill side only: the server placed on prefill nodes and a hand-written `MultiConnector` (NIXL plus LMCache MP) in the prefill args, which srtctl passes through untouched |
+| `features/lmcache-server-sglang.yaml` | The same server next to aggregated SGLang workers: SGLang's `enable-lmcache` flag, with srtctl pointing each worker at the server on its node (`LMCACHE_MP_HOST`/`LMCACHE_MP_PORT`). Needs an SGLang image that ships LMCache (the `sglang-lmcache` alias) |
 | `features/vllm-failover.yaml` | `engine.failover:` shadow engine recovery: a GPU Memory Service sidecar and a parked standby engine per vLLM worker, relaunched in place after a crash. Needs a container that ships `gpu_memory_service` (the `dynamo-vllm` alias, an `nvcr.io/nvidia/ai-dynamo/vllm-runtime` image). See [../docs/shadow-engine-recovery.md](../docs/shadow-engine-recovery.md) |
 
 ## Cluster aliases
@@ -46,6 +49,8 @@ containers:
   vllm: /path/to/vllm.sqsh                  # vLLM image with the vllm-router executable
   trtllm: /path/to/tensorrtllm-runtime.sqsh # Dynamo TRT-LLM runtime image (ships ai-dynamo and trtllm-serve)
   dynamo-vllm: /path/to/vllm-runtime.sqsh   # Dynamo vLLM runtime image (ships ai-dynamo and gpu_memory_service), for features/vllm-failover.yaml
+  vllm-lmcache: /path/to/vllm-lmcache.sqsh  # vLLM image with LMCache installed, for features/lmcache-server.yaml and -disagg.yaml
+  sglang-lmcache: /path/to/sglang-lmcache.sqsh  # SGLang image with LMCache installed, for features/lmcache-server-sglang.yaml
 ```
 
 `resources.gpu_type` and `gpus_per_node` are set to `h100` and `8`; change them to match the partition you submit to.

examples/features/lmcache-server-disagg.yaml

@@ -0,0 +1,88 @@
+# An LMCache DRAM tier on the prefill side of a disaggregated vLLM job.
+#
+# Prefill keeps NIXL for the P/D transfer and adds LMCache for prefix reuse, so its
+# kv-transfer-config is a hand-written MultiConnector (srtctl passes raw JSON through
+# untouched; the `lmcache-mp` shorthand covers the single-connector case). Decode is
+# NIXL only. The server is placed on prefill nodes only (`placement.node: prefill`).
+#
+# Aliases (`qwen3-0.6b`, `vllm-lmcache`) resolve through srtslurm.yaml; see
+# examples/README.md and examples/features/lmcache-server.yaml.
+
+schema: 2
+name: "qwen3-0.6b-vllm-lmcache-disagg"
+
+slurm:
+  time_limit: "00:30:00"
+
+model:
+  path: "qwen3-0.6b"
+  container: "vllm-lmcache"
+  precision: "bf16"
+
+resources:
+  gpu_type: "h100"
+  gpus_per_node: 8
+dynamo:
+  source:
+    pypi: "1.4.2"
+
+environment:
+  PYTHONHASHSEED: "0"
+
+frontend:
+  type: dynamo
+  enable_multiple_frontends: false
+  args:
+    router-mode: "kv"
+
+engine:
+  type: vllm
+  connector: nixl
+roles:
+  prefill:
+    nodes: 1
+    workers: 1
+    gpus: 1
+    env:
+      DYN_HEALTH_CHECK_ENABLED: "false"
+      PYTHONUNBUFFERED: "1"
+    args:
+      # NIXL to decode plus LMCache MP on this node; a raw string here replaces the connector preset.
+      kv-transfer-config: '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both"},{"kv_connector":"LMCacheMPConnector","kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://localhost","lmcache.mp.port":8750}}]}}'
+      served-model-name: "Qwen/Qwen3-0.6B"
+      tensor-parallel-size: 1
+      gpu-memory-utilization: 0.5
+      max-model-len: 4096
+  decode:
+    nodes: colocate
+    workers: 1
+    gpus: 1
+    env:
+      DYN_HEALTH_CHECK_ENABLED: "false"
+      PYTHONUNBUFFERED: "1"
+    args:
+      served-model-name: "Qwen/Qwen3-0.6B"
+      tensor-parallel-size: 1
+      gpu-memory-utilization: 0.5
+      max-model-len: 4096
+
+services:
+  - name: lmcache
+    type: lmcache-server
+    placement:
+      node: prefill          # decode nodes get no server; with `colocate` that is still this one node
+    args:
+      - --l1-size-gb
+      - "16"
+      - --chunk-size
+      - "256"
+      - --max-workers
+      - "1"
+    preamble: |
+      ulimit -l unlimited
+
+benchmark:
+  type: "sa-bench"
+  isl: 128
+  osl: 128
+  concurrencies: "4x8"

examples/features/lmcache-server-sglang.yaml

@@ -0,0 +1,59 @@
+# An LMCache DRAM tier next to aggregated SGLang workers.
+#
+# `type: lmcache-server` runs the LMCache multiprocess server on every worker node
+# before the workers start (8750 RPC, 8751 HTTP). SGLang's own `enable-lmcache` flag
+# switches the worker to LMCacheUnifiedRadixCache; srtctl sets LMCACHE_MP_HOST and
+# LMCACHE_MP_PORT so it dials the server on its own node. A recipe that passes
+# `lmcache-config-file` or sets LMCACHE_MP_HOST in the role env keeps its own address.
+#
+# The SGLang image must ship LMCache, so this uses a `sglang-lmcache` alias. Aliases
+# (`qwen3-0.6b`, `sglang-lmcache`) resolve through srtslurm.yaml; see examples/README.md.
+
+schema: 2
+name: "qwen3-0.6b-sglang-lmcache-agg"
+
+slurm:
+  time_limit: "00:30:00"
+
+model:
+  path: "qwen3-0.6b"
+  container: "sglang-lmcache"
+  precision: "bf16"
+
+resources:
+  gpu_type: "h100"
+  gpus_per_node: 8
+
+frontend:
+  type: sglang
+  enable_multiple_frontends: false
+
+engine: sglang
+roles:
+  agg:
+    nodes: 1
+    workers: 1
+    gpus: 1
+    args:
+      enable-lmcache: true
+      mem-fraction-static: 0.5
+      context-length: 4096
+
+services:
+  - name: lmcache
+    type: lmcache-server
+    args:
+      - --l1-size-gb
+      - "16"
+      - --chunk-size
+      - "256"
+      - --eviction-policy
+      - LRU
+    preamble: |
+      ulimit -l unlimited
+
+benchmark:
+  type: "sa-bench"
+  isl: 128
+  osl: 128
+  concurrencies: "4x8"

examples/features/lmcache-server.yaml

@@ -0,0 +1,78 @@
+# An LMCache DRAM tier next to aggregated vLLM workers.
+#
+# `type: lmcache-server` runs the LMCache multiprocess server on every worker node
+# before the workers start, on the ports srtctl owns (8750 RPC, 8751 HTTP), and holds
+# the job until `GET /healthcheck` answers. `connector: lmcache-mp` points each vLLM
+# worker's LMCacheMPConnector at the server on its own node. Everything after the
+# ports is LMCache's own CLI: put `lmcache server --help` flags under `args`.
+#
+# The vLLM image must ship LMCache (the connector is imported inside the worker), so
+# this uses a `vllm-lmcache` alias. Aliases (`qwen3-0.6b`, `vllm-lmcache`) resolve
+# through srtslurm.yaml; see examples/README.md.
+
+schema: 2
+name: "qwen3-0.6b-vllm-lmcache-agg"
+
+slurm:
+  time_limit: "00:30:00"
+
+model:
+  path: "qwen3-0.6b"
+  container: "vllm-lmcache"
+  precision: "bf16"
+
+resources:
+  gpu_type: "h100"
+  gpus_per_node: 8
+dynamo:
+  source:
+    pypi: "1.4.2"
+
+environment:
+  PYTHONHASHSEED: "0"        # chunk hashes must agree across the server and every rank
+
+frontend:
+  type: dynamo
+  enable_multiple_frontends: false
+  args:
+    router-mode: "kv"
+
+engine:
+  type: vllm
+  connector: lmcache-mp      # every role; or per role under roles.<role>.args.connector
+roles:
+  agg:
+    nodes: 1
+    workers: 2
+    gpus: 1
+    env:
+      DYN_HEALTH_CHECK_ENABLED: "false"
+      PYTHONUNBUFFERED: "1"
+    args:
+      served-model-name: "Qwen/Qwen3-0.6B"
+      tensor-parallel-size: 1
+      gpu-memory-utilization: 0.5
+      max-model-len: 4096
+
+services:
+  - name: lmcache
+    type: lmcache-server
+    # placement.node: workers (default) — one instance per worker node.
+    # start: before_workers, critical: true (defaults): a dead server fails the run.
+    args:                    # appended to `lmcache server --host ... --port 8750 --http-host ... --http-port 8751`
+      - --l1-size-gb
+      - "16"
+      - --chunk-size
+      - "256"
+      - --max-workers
+      - "2"
+      - --eviction-policy
+      - LRU
+    preamble: |
+      ulimit -l unlimited
+
+benchmark:
+  type: "sa-bench"
+  isl: 128
+  osl: 128
+  concurrencies: "4x8"