feat: support the LMCache server on ATOM and SGLang

[cquil11]

Adds the out-of-process LMCache server from NVIDIA/srt-slurm#507 and extends LMCache support to SGLang and ATOM.

• services[].type: lmcache-server runs lmcache server on every worker node by default, starts before workers, and waits for GET /healthcheck. The service is critical: a failed server fails the job. srtctl sets RPC port 8750 and HTTP port 8751; recipe args pass through to LMCache for settings such as cache size and chunk size.
• vLLM: connector: lmcache-mp uses LMCacheMPConnector at tcp://localhost:8750, so each rank connects to the server on its own node. KVConnector gains optional static extra_config for these defaults. A direct vllm serve aggregate worker (frontend.type: vllm) now runs the connector its role names under roles.agg.args.connector; it still skips the engine-wide default, which stays a prefill/decode setting. The disaggregated example combines LMCache MP with NIXL through MultiConnector.
• SGLang: roles with enable-lmcache: true get LMCACHE_MP_HOST and LMCACHE_MP_PORT set to the local server. Recipes can supply their own connection settings through lmcache-config-file or LMCACHE_MP_HOST in the role environment.
• ATOM: extra-kv-connectors supports in-process CPU offload (lmcache_offload) and the LMCache server client (lmcache_mp) on any role. Prefill/decode roles combine extras with the generated Mooncake connector using ATOM's multi connector, preserving the Mooncake handshake port. Aggregate roles use a single connector directly or wrap several in multi. lmcache_mp defaults to port 8750 and preserves recipe overrides.

For vLLM:

engine:
  type: vllm
  connector: lmcache-mp
services:
  - name: lmcache
    type: lmcache-server
    args: [--l1-size-gb, "180", --chunk-size, "256", --max-workers, "4"]

For ATOM CPU offload alongside Mooncake:

roles:
  prefill:
    env:
      PYTHONHASHSEED: "0"
    args:
      extra-kv-connectors:
        - kv_connector: lmcache_offload
          kv_role: offload

Includes aggregated and disaggregated vLLM examples, an aggregated SGLang example, and ATOM configuration docs. LMCache must be installed in the job container, either in the image or through the recipe's setup_script.

Validation

Tested on hardware through InferenceX, applied as a patch:

Path	Hardware	Run	Result
ATOM extra-kv-connectors (Mooncake + lmcache_offload)	MI355X 1P1D, conc 256 (InferenceX#3543)	36461236354	Passed
lmcache-server + vLLM LMCacheMPConnector	MI300X TP8, conc 16 (InferenceX#3545)	36460859852	Passed
lmcache-server + SGLang enable-lmcache	B200 TP8, conc 76 (InferenceX#3554)	36490342974	Passed

Not yet tested on hardware: ATOM lmcache_mp.

---

Note

Medium Risk
Changes KV connector generation and job startup order for inference workers; misconfiguration could break P/D or prefix-cache behavior, but defaults are additive and covered by snapshot tests.

Overview
Adds services[].type: lmcache-server, which starts the out-of-process LMCache MP server on each worker node (RPC 8750, HTTP 8751) before workers, treats it as critical, and gates on GET /healthcheck.

vLLM gains connector: lmcache-mp (LMCacheMPConnector → node-local server). Direct aggregate vllm serve workers now honor a per-role connector (e.g. lmcache-mp) while still skipping the engine-wide P/D default. SGLang workers with enable-lmcache get LMCACHE_MP_HOST/LMCACHE_MP_PORT unless the recipe sets lmcache-config-file or LMCACHE_MP_HOST. ATOM supports extra-kv-connectors in role args: P/D roles merge extras with the generated Mooncake connector via ATOM’s multi wrapper; aggregate roles use a single extra or multi when several are listed; lmcache_mp defaults to port 8750.

Docs, schema updates, three feature examples (vLLM agg/disagg, SGLang agg), and launch snapshots/tests cover the new paths.

^{Reviewed by Cursor Bugbot for commit 0f595d2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cquil11]

now agg configs can also go this path

[cquil11]

Closing because the LMCache server and its ATOM/SGLang integration were merged upstream in NVIDIA#507 and NVIDIA#528.