feat(trtllm): run TRT-LLM's mooncake-store connector on the native mooncake-master service

[nv-yna]

Problem

SGLang and vLLM recipes share a Mooncake Store pool by declaring a mooncake-master service. TRT-LLM recipes cannot: on main, a TRT-LLM recipe with that service fails to load with engine.mooncake_kv_store: Unknown field. The only TRT-LLM route today is TRT-LLM's own provisioning (trtllm-serve mooncake_master and a mooncake_store block in the engine config), which dynamo.trtllm does not run at all.

TRT-LLM's mooncake-store connector reads its Mooncake client config from the JSON file named by MOONCAKE_CONFIG_PATH before it looks at its own mooncake_store block (config.py), and trtllm-serve skips its own provisioning when that variable is set (master.py). That is the same mechanism srtctl already uses for vLLM, so TRT-LLM can join the same master.

What a TRT-LLM recipe looks like with this PR

From examples/trtllm/trtllm-serve-disagg-mooncake.yaml (Mooncake-relevant keys only):

engine:
  type: trtllm
  served_model_name: "Qwen/Qwen3-0.6B"     # also the pool's model_key (default)
roles:
  prefill:
    args:                                  # TRT-LLM's own config: switch the connector on
      kv_cache_config:
        use_kv_cache_manager_v2: true
      cache_transceiver_config:
        backend: "NIXL"
        transceiver_runtime: "PYTHON"
      kv_connector_config:
        connector: mooncake-store          # no mooncake_store block
  decode:
    args:                                  # the same three keys
      kv_cache_config:
        use_kv_cache_manager_v2: true
      cache_transceiver_config:
        backend: "NIXL"
        transceiver_runtime: "PYTHON"
      kv_connector_config:
        connector: mooncake-store
    mooncake_store_config:                 # new: this role's client keys
      global_segment_size: 16GiB           # decode ranks lend more host memory
services:
  - name: mooncake-master                  # the same service SGLang and vLLM use
    type: mooncake-master
    options:
      store_config:                        # client keys shared by every role
        protocol: rdma
        global_segment_size: 8GiB          # per rank; byte counts or binary units
        stage_through_host: true

Before workers start, srtctl writes one client config per role that selects the connector (output of the mock orchestrator for this example):

/logs/mooncake_store_config_prefill.json       /logs/mooncake_store_config_decode.json
{ "role": "both",                              { "role": "capacity",
  "model_key": "Qwen/Qwen3-0.6B",                "model_key": "Qwen/Qwen3-0.6B",
  "protocol": "rdma",                            "protocol": "rdma",
  "global_segment_size": "8GiB",                 "global_segment_size": "16GiB",
  "stage_through_host": true,                    "stage_through_host": true,
  "master_server_address": "<infra>:8700" }      "master_server_address": "<infra>:8700" }

Each file layers, later wins: the role default (both for prefill and agg, capacity for decode) and model_key (the served model name), the service's options.store_config, then roles.<role>.mooncake_store_config; master_server_address is always the master srtctl launched. The master is the usual mooncake_master --port=8700 --enable_http_metadata_server=true ... on the infra node, gated on ports 8700, 8701, and 8702. Each worker's srun task gets its role's file, so every rank trtllm-llmapi-launch starts inherits it:

export MOONCAKE_CONFIG_PATH=/logs/mooncake_store_config_prefill.json && ...
exec trtllm-llmapi-launch trtllm-serve <model> --host 0.0.0.0 --port 6100 --tensor_parallel_size 1 \
  --config /logs/trtllm_config_prefill.yaml --served_model_name Qwen/Qwen3-0.6B

With frontend.type: dynamo the same recipe launches trtllm-llmapi-launch python3 -m dynamo.trtllm ... --extra-engine-args /logs/trtllm_config_prefill.yaml with the same file and variable (examples/trtllm/dynamo-disagg-mooncake.yaml); no wrapper is needed because dynamo.trtllm provisions nothing and the connector reads the variable.

How the structure lines up with SGLang and vLLM

Piece	SGLang	vLLM	TRT-LLM (this PR)
Pool server	mooncake-master service	same	same service, same command
Client keys shared by every role	MOONCAKE_* in each role's env	options.store_config	options.store_config (same field as vLLM)
Client keys for one role	roles.<role>.env	not supported (one file for every role)	roles.<role>.mooncake_store_config
How workers receive them	environment variables	MOONCAKE_CONFIG_PATH → /logs/mooncake_store_config.json	MOONCAKE_CONFIG_PATH → /logs/mooncake_store_config_<role>.json
Connector switch, in the role's own engine args	disaggregation-transfer-backend: mooncake and the cache-linker flags	kv-transfer-config with MooncakeStoreConnector	kv_connector_config: {connector: mooncake-store}
Mooncake library knobs (MC_*)	roles.<role>.env	roles.<role>.env	roles.<role>.env
Decode nodes' memory	mooncake-store services (by default SGLang decode workers run no Mooncake client)	each rank lends global_segment_size in-process	each rank lends in-process; decode defaults to role capacity (lends memory, moves no KV)
Set by srtctl, never by the recipe	master address	master address	master address, plus the role and model_key defaults

Per-role settings follow the rest of the schema: they live under roles.<role>, layered over a shared default by one accessor (role_mooncake_store_config, like role_kv_events). Keys every server of the pool must agree on (model_key, namespace, metadata_server, protocol, master_server_address) are refused there and stay in options.store_config.

Changes

• TRTLLMBackend.mooncake_kv_store (TRTLLMMooncakeKVStoreConfig), filled from a declared mooncake-master service.
• RoleConfig.mooncake_store_config (TRT-LLM only), read through role_mooncake_store_config; refused on SGLang and vLLM roles with a pointer to their existing knobs.
• Backend.mooncake_store_configs(infra_node_ip, served_model_name) lists the files the orchestrator writes; get_mooncake_worker_env takes the worker mode so TRT-LLM can point each role at its own file.
• model_key defaults to the served model name. TRT-LLM deliberately has no default (config.py) so two checkpoints sharing a pool cannot read each other's KV; this pool belongs to one job, and an external master is refused for TRT-LLM.
• Load-time checks for what TRT-LLM would otherwise reject in every rank after the model loads: a mooncake_store block or its block-only keys (segment_size, pool, run_dir, master_timeout) next to the master, an empty model_key, an unknown role, sizes that are not byte counts or binary units (TRT-LLM reads GB as 1000³ and refuses it), non-positive transfer_batch_size or local_buffer_size, and stage_through_host given as a string. Without a master service, a connector role must find a client config itself (MOONCAKE_CONFIG_PATH in its env, or a mooncake_store block with its own run_dir when trtllm-serve runs the engine).
• Two examples (trtllm-serve, Dynamo), docs (docs/mooncake-kv-store.md, docs/services.md), regenerated schema docs, tests, and launch snapshots.

Validation

• Tests: 4,485 passed, 2 skipped; ruff and the schema-docs check pass.
• ty check: CI's typecheck fails with the same 11 invalid-attribute-override diagnostics main currently fails with under ty 0.0.86 (frozen dataclass fields overriding the RoleSettings / Backend attributes), plus one of the same kind for the new RoleSettings.mooncake_store_config. With ty 0.0.85 it passes.
• make snapshots: snapshots for the two new examples are included; the per-role field and model_key default change no srun launch of any example.
• On GPUs, both frontends (trtllm-serve and Dynamo), Qwen3-0.6B with one prefill and one decode worker on a GB300 node, TRT-LLM 1.3.0rc23.post1 built from feat/m3_with_msa at 1ab6bbc990 (the connector API from [None][feat] BREAKING: Refine Mooncake connector API TensorRT-LLM#19706) and Mooncake 0.3.13: the native master took 1,375 pages (5.05 GB across the prefill segment and the decode capacity segment); after the prefill GPU cache evicted them, repeated requests read 1,374 pages (5.04 GB) back from the pool with identical greedy outputs. The prefill client config that run wrote is identical to what this branch renders.

Limitations

• roles.<role>.mooncake_store_config is read by TRT-LLM only in this PR. vLLM still writes one file for every role, and SGLang takes Mooncake settings as environment variables.
• srtctl still exports MOONCAKE_MASTER, MOONCAKE_TE_META_DATA_SERVER, and MOONCAKE_LOCAL_HOSTNAME to TRT-LLM workers; TRT-LLM reads only the file, so they are inert there (documented).
• Engine-side limits are TRT-LLM's and differ by build, so srtctl does not repeat them. TRT-LLM main ([TRTLLM-12891][feat] Integrate Mooncake connector into KVCMv2 TensorRT-LLM#19171) keeps the same file contract; GPU validation used the feat/m3_with_msa build.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.47090% with 1 line in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@009a4fa). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/srtctl/backends/trtllm.py	98.96%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #595   +/-   ##
=======================================
  Coverage        ?   85.34%           
=======================================
  Files           ?      161           
  Lines           ?    23363           
  Branches        ?        0           
=======================================
  Hits            ?    19939           
  Misses          ?     3424           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.