feat(frontends): add the Shepherd Model Gateway (SMG) router by cquil11 · Pull Request #38 · SemiAnalysisAI/srt-slurm

cquil11 · 2026-10-01T21:56:40Z

Add frontend.type: smg for Shepherd Model Gateway, reusing StaticRouterFrontend for launch, worker registration, and readiness.
Support aggregated SGLang, vLLM, and TRT-LLM workers. P/D supports SGLang with bootstrap rendezvous and vLLM over gRPC with NIXL. TRT-LLM P/D is unsupported.
Probe HTTP workers on /health and gRPC workers over TCP before starting SMG. Check registered worker counts through /workers.
Manage SMG's separate Prometheus port and use each engine's native worker metrics endpoint for Tachometer and AIPerf.
Keep one API endpoint per multi-node vLLM DP deployment when the frontend uses direct workers without expanding node-local DP pools.
Run the recipe's setup_script in the router container. Add documentation and five examples using SMG v1.11.0, including a separate frontend.container_image option.

Validation

agentx-fast against srt-slurm dba469b6 through InferenceX #3665.

Scenario	Cluster	Run	Result
SGLang aggregated	b300-dsxe	36971299038	Passed
SGLang 1P1D, Mooncake	gb300-nv	36971304852	Passed
vLLM DEP16, 2 nodes	b200-nscale	36971310560	Passed
vLLM 1P1D, gRPC + NIXL	gb300-nv	36971316742	Passed
TRT-LLM aggregated, 2 nodes	b200-nscale	36971322666	Passed

Note

Medium Risk
New frontend type touches launch topology, health gates, and metrics scraping across engines; misconfiguration is mostly caught at recipe load, but P/D and gRPC paths add operational complexity.

Overview
Adds frontend.type: smg so jobs can run the engine-agnostic Shepherd Model Gateway in front of static worker URLs (aggregated SGLang/vLLM/TRT-LLM, SGLang HTTP P/D with bootstrap ports, vLLM gRPC+NIXL P/D). Recipe validation rejects unsupported P/D mixes (e.g. TRT-LLM P/D, HTTP vLLM P/D).

Orchestration and readiness: SMGFrontend extends the static-router path—smg launch with managed --prometheus-port, optional frontend.container_image and setup script in the router container, HTTP /health or TCP probes for gRPC workers before start, and readiness via GET /workers counts.

Metrics and topology: Frontends gain worker_metrics_path(backend, mode) so Tachometer/AIPerf scrape each engine’s route (/metrics vs TRT-LLM /prometheus/metrics; none for gRPC). Backends expose prometheus_metrics_path and is_grpc_mode; static routers advertise grpc:// when needed. vLLM drops bootstrap port allocation where KV uses NIXL, and multi-node DP behind non–DP-expanding frontends stays one API URL on the leader.

Docs (docs/smg.md), schema updates, five examples, launch snapshots, and tests cover the new frontend.

^{Reviewed by Cursor Bugbot for commit ad0eef7. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

cursor

Stale Bugbot comment from a previous run.

Register frontend.type: smg, a StaticRouterFrontend that launches `smg launch` with the allocated worker URLs: --worker-urls for aggregated workers of any backend, --pd-disaggregation with --prefill URL BOOTSTRAP_PORT and --decode URL for prefill/decode. Readiness counts SMG's GET /workers registry, workers are probed before SMG starts, and the Prometheus listener is bound on SMG_METRICS_PORT and scraped by tachometer. Add docs/smg.md, a vLLM aggregated and an SGLang P/D example with their launch snapshots, and tests. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

Share vLLM Router's setup-script preamble as setup_script_preamble in static_router.py and use it for SMG too, so a recipe can pip install smg into the model image instead of pulling a separate router image. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

A direct vLLM worker behind a router that does not expand node-local DP pools (smg) is one vllm serve that balances its DP ranks itself: the leader node serves the API and the other nodes run headless ranks. The backend gave every node of such a DP endpoint an HTTP port, so SMG was handed the headless nodes' URLs too. Keep the standard topology (an HTTP port on the leader only) for every direct worker launch except routers that expand node-local DP, as frontend.type: vllm already did. Add examples/vllm/smg-dep16.yaml (two-node DP16 with EP behind SMG) and document why SMG's --dp-aware rank pinning is not used for vLLM HTTP workers. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

Promote is_grpc_mode(mode) to BackendProtocol. SGLang keeps grpc-mode, vLLM answers roles.<role>.args.grpc (vllm serve --grpc, served by smg-grpc-servicer), and the HTTP-only engines answer False. StaticRouterFrontend.worker_scheme now picks grpc:// from that one hook for every static router, so sglang-router's override goes away and smg advertises gRPC workers the same way. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

…tarts The pre-start probe only polled HTTP /health, so gRPC workers were handed to the router before they were up. Wait for a gRPC worker's port to accept connections instead (vLLM's gRPC server binds it after the engine starts), within the same health-check budget. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

The mock orchestrator patches every readiness wait; the gRPC worker port probe added to StaticRouterFrontend.start_frontends was not, so a dry run of a gRPC recipe waited out the health-check budget. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

SMG learns a vLLM worker's KV connector only over gRPC (the servicer reports kv_transfer_config in GetServerInfo), so vLLM P/D behind SMG uses grpc: true workers with engine.connector: nixl; SMG tags the prefill leg for a remote decode and relays the NIXL handoff to decode. Add examples/vllm/smg-disagg-grpc.yaml, its snapshot, and docs. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

trtllm-serve serves Prometheus text at /prometheus/metrics; its /metrics is JSON iteration stats. A static router in front of trtllm-serve workers advertised /metrics for them, so the tachometer targets and the AIPerf metrics URLs read the wrong route. Each backend now names its native server's Prometheus path (native_metrics_path), and frontends answer the worker path through worker_metrics_path(backend): Dynamo keeps /metrics, frontends whose workers are the engine's own server use the backend's path. metrics_path stays the router's own route. return_perf_metrics, which mounts the trtllm-serve route, now defaults on behind every direct-worker frontend, not only trtllm_serve. Adds examples/trtllm/smg-agg.yaml and its launch snapshot. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 5f5b055. Configure here.}

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

cquil11 · 2026-10-05T18:35:06Z

    bootstrap_port: int | None = None


+def setup_script_preamble(config: Any) -> str | None:


this alr existed for vllm frontend as well so move as common function to be used by vllm router and smg

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

…path Signed-off-by: Cam Quilici <cjquilici@gmail.com>

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

vLLM hands the KV cache over its NIXL side channel, so a static router such as SMG was told about a bootstrap port nothing listened on. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

A gRPC-mode worker (vllm serve --grpc) starts no HTTP listener, so worker_metrics_path is now per mode and None for gRPC roles; Tachometer and the AIPerf server-metrics URLs skip them. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

Like the other direct-worker frontends, check the recipe layout at load: a P/D topology must use one engine in both roles, may not be TRT-LLM, and with vLLM must serve gRPC in both roles, since over HTTP the decode worker recomputes the prompt with no KV handoff. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

github-actions · 2026-10-06T22:03:42Z

src/srtctl diff: +241 -80

cquil11 · 2026-10-07T22:18:59Z

Merged upstream in NVIDIA#569.

cquil11 requested a review from ishandhanani as a code owner October 1, 2026 21:56

github-advanced-security Bot found potential problems Oct 1, 2026

View reviewed changes

Comment thread src/srtctl/dsight/assets/explorer.js Fixed

cquil11 changed the base branch from main to chore/sync-upstream-20261001 October 1, 2026 22:27

cquil11 removed the request for review from ishandhanani October 1, 2026 22:28

cquil11 changed the base branch from chore/sync-upstream-20261001 to main October 2, 2026 01:31

This was referenced Oct 2, 2026

[DNM] test SMG with SGLang (aggregated and P/D) through srt-slurm SemiAnalysisAI/InferenceX#3655

Closed

feat(vllm): run vLLM behind SMG, with DP and P/D over gRPC #41

Merged

feat(smg): route TRT-LLM workers through SMG #42

Merged

cursor Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread src/srtctl/frontends/static_router.py Resolved

cursor Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread src/srtctl/frontends/static_router.py Resolved

cursor Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread tests/snapshots/launch/vllm__smg-disagg-grpc.txt Resolved

This was referenced Oct 2, 2026

[DNM] Test SMG with SGLang, vLLM, and TRT-LLM through srt-slurm SemiAnalysisAI/InferenceX#3665

Draft

feat(tokenspeed): serve TokenSpeed behind SMG #43

Closed

cquil11 added 9 commits October 5, 2026 09:55

docs(smg): note that vLLM gRPC workers serve no Prometheus metrics

090eb5b

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

cquil11 force-pushed the feat/smg-frontend branch from dba469b to 5f5b055 Compare October 5, 2026 14:55

cursor Bot reviewed Oct 5, 2026

View reviewed changes

Comment thread src/srtctl/frontends/static_router.py Resolved

cquil11 added 2 commits October 5, 2026 10:03

docs(smg): link the examples by URL for the strict docs build

2573698

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

refactor(smg): inherit required_backend from Frontend

9f11ad1

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

cquil11 commented Oct 5, 2026

View reviewed changes

cquil11 added 2 commits October 5, 2026 13:40

refactor(telemetry): pass no worker metrics path for a services-only job

7435667

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

refactor(backends): rename native_metrics_path to prometheus_metrics_…

b7d87f5

…path Signed-off-by: Cam Quilici <cjquilici@gmail.com>

cquil11 mentioned this pull request Oct 5, 2026

feat(tokenspeed): add a TokenSpeed backend for Dynamo and SMG #37

Closed

cquil11 added 3 commits October 5, 2026 14:45

fix(smg): stop waiting for gRPC workers once a critical process exits

7892df2

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

fix(vllm): allocate no prefill bootstrap port, which vLLM never binds

987c3c5

vLLM hands the KV cache over its NIXL side channel, so a static router such as SMG was told about a bootstrap port nothing listened on. Signed-off-by: Cam Quilici <cjquilici@gmail.com>

cquil11 mentioned this pull request Oct 5, 2026

feat(frontends): add the llm-d router #45

Open

cquil11 closed this Oct 7, 2026

Navigation Menu

Repository navigation

feat(frontends): add the Shepherd Model Gateway (SMG) router - #38

feat(frontends): add the Shepherd Model Gateway (SMG) router#38
cquil11 wants to merge 17 commits into
mainfrom
feat/smg-frontend

cquil11 commented Oct 1, 2026 •

edited by cursor Bot

Loading

Uh oh!

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

cquil11 Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

cquil11 commented Oct 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		bootstrap_port: int \| None = None


		def setup_script_preamble(config: Any) -> str \| None:

Conversation

cquil11 commented Oct 1, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Validation

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cquil11 Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

cquil11 commented Oct 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cquil11 commented Oct 1, 2026 •

edited by cursor Bot

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading