Skip to content

feat(trtllm): add OpenEngine sidecar#11610

Draft
connorcarpenter15 wants to merge 10 commits into
ai-dynamo:mainfrom
connorcarpenter15:feat/trtllm-openengine-sidecar
Draft

feat(trtllm): add OpenEngine sidecar#11610
connorcarpenter15 wants to merge 10 commits into
ai-dynamo:mainfrom
connorcarpenter15:feat/trtllm-openengine-sidecar

Conversation

@connorcarpenter15

@connorcarpenter15 connorcarpenter15 commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

  • add the engine-neutral Rust dynamo-openengine-sidecar worker under lib/sidecar/openengine, configured by endpoint and OpenEngine capability discovery
  • map Dynamo preprocessed requests to OpenEngine aggregate and context-first prefill/decode streams, including sampling, stopping, logprobs, standard priority/DP-rank metadata, cache controls, ordered media, media options, and LoRA selection
  • forward cancellation, drain, health, load, and KV event sources through the existing backend-common worker framework
  • reuse backend-common's LoRA controller for adapter downloads, deterministic IDs, model cards, and control endpoints
  • add aggregate, 1P1D, multimodal, audio, and LoRA launch examples and DGD manifests
  • preserve media modality placeholders and adapter/media semantics through frontend preprocessing
  • merge current Dynamo main and adopt its consolidated lib/sidecar/ workspace layout

Why

This adds a process-isolated TRT-LLM deployment path without replacing Dynamo's existing in-process TRT-LLM backend. The sidecar remains engine-neutral and communicates with the GPU engine exclusively through OpenEngine gRPC.

Dependencies

Validation

  • cargo fmt --all -- --check
  • cargo metadata --locked --format-version 1
  • cargo clippy -p dynamo-openengine-sidecar --all-targets -- -D warnings
  • generated CODEOWNERS validation: 100% explicit coverage
  • focused fake OpenEngine sidecar suite on current Dynamo main: 29 passed
  • OpenEngine Buf, generated-code drift, descriptor, package, and Python/Rust fixture checks passed
  • TRT-LLM OpenEngine fake-servicer and protocol suite: 66 passed
  • baseline computelab frontend smoke matrix passed before the current OpenEngine split-service refresh:
    • Qwen3-0.6B aggregate text and LoRA lifecycle on 1 H100
    • Qwen3-0.6B context-first 1P1D text and LoRA on both roles on 2 NVLink H100s
    • Qwen3-VL-2B-Instruct image/video aggregate and 1P1D on 1 then 2 H100s
    • Phi-4-multimodal-instruct aggregate audio on 1 H100
  • the current OpenEngine refresh is CPU-verified; its GPU smoke rerun is pending

The baseline GPU smokes verified nonempty output, terminal usage, zero post-request load, and graceful shutdown; disaggregated cases also verified the actual prefill/decode handoff.

Compatibility

The existing in-process TRT-LLM backend remains available. OpenEngine deployments are opt-in through the new sidecar binary and launch examples.

@connorcarpenter15
feat(trtllm): add OpenEngine sidecar
b903166 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
fix(llm): preserve adapter and media semantics
a73cbc2 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
fix(openengine-sidecar): harden role and media handling
669a773 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
fix(llm): reject cross-message media placeholder collisions
ac8502c 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
connorcarpenter15 temporarily deployed to external_collaborator July 13, 2026 18:23 — with GitHub Actions Inactive
@connorcarpenter15
connorcarpenter15 temporarily deployed to external_collaborator July 13, 2026 18:23 — with GitHub Actions Inactive
@github-actions github-actions Bot added the feat label Jul 13, 2026
@github-actions

github-actions Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🌿 Fern Docs Preview: https://nvidia-preview-019f86fd-9e22-74cf-abdd-c2057b9bdd63.docs.buildwithfern.com/dynamo/dev

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation backend::trtllm Relates to the trtllm backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Jul 13, 2026
@datadog-official

datadog-official Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 3 Pipeline jobs failed

Docs link check | lychee   View in Datadog   GitHub Actions

PR | deploy-operator   View in Datadog   GitHub Actions

PR | deploy-status-check   View in Datadog   GitHub Actions

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 42.14% (-3.34%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 14fc0de | Docs | Give us feedback!

@connorcarpenter15
fix(openengine): update sidecar protocol client
24397a0 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
connorcarpenter15 temporarily deployed to external_collaborator July 21, 2026 22:47 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

/ok to test 24397a0

@connorcarpenter15
chore: merge current main
d9cd037 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>

# Conflicts:
#	Cargo.toml
#	lib/llm/Cargo.toml
@connorcarpenter15
connorcarpenter15 temporarily deployed to external_collaborator July 21, 2026 22:54 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

/ok to test d9cd037

@connorcarpenter15
fix(openengine): update deployment schema pins
244cc38 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
connorcarpenter15 temporarily deployed to external_collaborator July 21, 2026 22:56 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

/ok to test 244cc38

@connorcarpenter15
fix(openengine): pin CI-resolvable protocol dependency
2b0707c 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
connorcarpenter15 temporarily deployed to external_collaborator July 21, 2026 23:07 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

/ok to test 2b0707c

@connorcarpenter15
fix(build): synchronize KVBM bindings lockfile
fb18bc6 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
connorcarpenter15 temporarily deployed to external_collaborator July 21, 2026 23:18 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

/ok to test fb18bc6

@connorcarpenter15
fix(build): synchronize Python bindings lockfile
14fc0de 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
connorcarpenter15 temporarily deployed to external_collaborator July 21, 2026 23:20 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

/ok to test 14fc0de

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backend::trtllm Relates to the trtllm backend documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@connorcarpenter15 @dynamo-ops