feat(frontends): add the llm-d router

[cquil11]

• Add frontend.type: llm-d: Envoy and the Endpoint Picker (EPP) for aggregated and prefill/decode vLLM jobs.
• Generate worker discovery and router configs after workers pass health checks; gate readiness on Envoy and EPP.
• Run a pd-sidecar per decode worker, with allocated proxy ports and NIXL KV transfer.
• Add frontend.epp_config, docs/llm-d.md, and aggregated/disaggregated/DP-rank examples.
• Route to vLLM DP ranks: data-parallel-external-lb: true launches every rank as its own vllm serve --data-parallel-rank on its own port, one EPP endpoint (and decode sidecar) per rank, on one or several nodes.
• Route by KV-cache events: roles.<role>.kv_events connects every worker's (or rank's) vLLM publisher to the EPP's precise-prefix-cache-producer on one socket srtctl owns, with topics naming each endpoint, and points the token-producer at a worker's render endpoint.
• Without frontend.epp_config, run the scorers of llm-d's no-Kubernetes guide (prefill and decode profiles for P/D); the EPP's own default never applied, since srtctl's file always carries discovery.
• Map a decode MultiConnector (NIXL next to OffloadingConnector CPU offloading) to the nixlv2 sidecar protocol.
• Scope: vLLM only, one router replica, static discovery (a failed worker or sidecar ends the job). KV events use the EPP's global socket: v0.10 file discovery has no per-endpoint rankIndex, so per-pod subscription and vLLM's multi-port DP supervisor are not used.
• Reuse the SMG frontend's worker readiness gate. Follow llm-d-router v0.10.0's file-discovery API.

flowchart TD
    Client --> Envoy
    Envoy <-->|Endpoint selection<br/>over gRPC| EPP
    Envoy -->|Aggregated request| Agg["vLLM aggregate worker"]
    Envoy -->|Disaggregated request| Sidecar["Decode P/D sidecar"]
    Sidecar -->|Run prefill| Prefill["vLLM prefill worker"]
    Sidecar -->|Decode with<br/>transfer parameters| Decode["vLLM decode worker"]
    Prefill -.->|KV cache via NIXL| Decode

Loading

Validation

AgentX through InferenceX #3741, with image quay.io/rh-ee-imarkov/llm-d-nokube-vllm:dspark-0814-nightly (EPP and sidecar v0.10.0). MiniMax-M3 NVFP4 at concurrency 8 unless noted. The first four rows ran cd129c2e; the rest ran 0c209710, the same commits before the rebase onto upstream main.

Scenario	Hardware	Run
DeepSeek-V4-Pro 1P1D, DEP8 prefill / DEP8 decode, NIXL, c64	GB200	37396765122
DeepSeek-V4-Pro aggregated, DEP8, c16	GB200	37396830944
MiniMax-M3 1P1D, TP4 / TP4, NIXL, c8	GB300	37396770618
MiniMax-M3 aggregated, 2x TP4, c8	GB300	37396773335
Precise prefix, aggregated 2x TP4	GB300	37553005166
Precise prefix, 1P1D TP4 / TP4	GB300	37553008202
DP ranks, one node, DP2 x TP2, precise prefix	GB300	37553011355
DP ranks, two nodes, DP4 x TP2, precise prefix	GB300	37553014497
DP ranks, 1P1D DP2 x TP2 each, precise prefix	GB300	37553017562
Precise prefix over CPU offload, 1P1D TP4, NIXL + OffloadingConnector in a MultiConnector	GB300	37553020113
No epp_config (guide scorers), aggregated 2x TP4	GB300	37553023865
Baseline rerun, aggregated 2x TP4	GB300	37553026821

[cquil11]

this isn't necessarily relevant to this pr (ie llm-d enablement)
this was redundant code that I am removing as part of this pr

[cquil11]

it is redundant because it can be handled by the existing _get_model_parallel_size method

[devin-ai-integration]

Review summary for #588 (llm-d router)

The llm-d frontend itself looks solid. It rides existing machinery (StaticRouterFrontend, an implied per-worker llm-d-sidecar service, PROXY_PORTS from the allocator), validates config at load, cites llm-d-router upstream at a pinned commit, and ships docs, three examples, and snapshots. make check passes on 01947da: 4402 passed, 9 skipped. The only snapshot changes are the three new vllm__llm-d-* files; no existing recipe's launch changes.

Blocking

1. Unrelated fork commits. The branch is based on the fork's main, so the diff against upstream also includes the fork's .github/CODEOWNERS change and the launch-plan recorder feature (core/launch_plan.py, output.record_launch_plan, ClusterConfig.record_launch_plan). Neither is in the description. Please drop them or split them out (inline comments).
2. Design Rule in the recorder, if it stays: do_sweep.run calls get_srtslurm_setting, which reloads load_cluster_config() from a stage (inline).
3. Upstream vLLM behavior isn't cited. The 5557 - r publisher port and the connect-vs-bind behavior depend on vLLM's ZmqEventPublisher and on which DP index the scheduler passes (inline).
4. Version mismatch. The code and docs/llm-d.md pin llm-d-router v0.11.0, but the validation table says every run used an EPP/sidecar v0.10.0 image. Please state which version the integration supports and what the example image ships, or re-validate on v0.11.0. epp_config_document notes that v0.10 and v0.11 accept the same discovery spelling; the other cited v0.11 behavior (the llm_d_epp_ready_endpoints gauge, the sidecar flags, render tokenization) should be checked against v0.10 as well.

Nits (inline): the carve-out in the port-arithmetic design-rule test, the private _is_dp_mode call, getattr(process, "dp_rank", None), and the repeated _validate_epp_args call.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 97.92746% with 8 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@9d65c0a). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/srtctl/frontends/llm_d.py	97.71%	5 Missing ⚠️
src/srtctl/backends/vllm.py	98.07%	1 Missing ⚠️
src/srtctl/frontends/base.py	80.00%	1 Missing ⚠️
src/srtctl/services/llm_d_sidecar.py	97.91%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #588   +/-   ##
=======================================
  Coverage        ?   84.99%           
=======================================
  Files           ?      161           
  Lines           ?    23106           
  Branches        ?        0           
=======================================
  Hits            ?    19639           
  Misses          ?     3467           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.