[Protocol]: Track comprehensive cross-engine service coverage

[connorcarpenter15]

Existing proposals

• I searched the repository's open and closed issues for overlapping proposals.

Intended consumers

Both engine implementers and orchestration/framework authors.

OpenEngine is intended to provide a vendor-neutral service boundary across TensorRT-LLM, SGLang, and vLLM, with Dynamo as the primary distributed runtime while remaining usable by llm-d and SGLang Model Gateway deployments.

Problem and use case

The current service is a strong generation-oriented base, but it does not yet cover the full portable surface needed by the three major inference engines in distributed deployments.

For this roadmap, comprehensive means:

1. Every shared, externally observable engine capability has a typed protocol contract.
2. Optional capabilities are grouped into discoverable conformance profiles; an engine is not required to implement every profile.
3. Lifecycle, failure, cancellation, ownership, and cleanup semantics are explicit enough for cross-process and cross-framework interoperability.
4. Engine-specific behavior is isolated behind typed extension boundaries rather than unstructured option maps.
5. The repository contains cross-engine conformance tests and a maintained support matrix.

Child proposals

• [Protocol]: Add portable non-generative inference task RPCs #10 — Add portable non-generative inference task RPCs
• [Protocol]: Add tokenizer and KV cache-key interoperability APIs #11 — Add tokenizer and KV cache-key interoperability APIs
• [Protocol]: Add application generation sessions and optional streaming input #12 — Add application generation sessions and optional streaming input
• [Protocol]: Add a secured runtime administration service #13 — Add a secured runtime administration service
• [Protocol]: Generalize the LoRA lifecycle into portable adapters #14 — Generalize the LoRA lifecycle into portable adapters
• [Protocol]: Define conformance profiles and streaming health/load discovery #15 — Define conformance profiles and streaming health/load discovery
• [Protocol]: Add lifecycle-safe disaggregated KV session orchestration #16 — Add lifecycle-safe disaggregated KV session orchestration
• [Protocol]: Add an encoder role and EPD media service #17 — Add an encoder role and EPD media service

Engine and framework evidence

The child proposals are grounded in public interfaces and deployment patterns from:

• vLLM's engine/client APIs and distributed serving architecture
• SGLang's engine, scheduler, tokenizer, multimodal, and disaggregation interfaces
• TensorRT-LLM's executor/runtime and disaggregated serving APIs
• Dynamo's routing, discovery, KV transfer, and distributed serving components
• llm-d's inference scheduling and disaggregated-serving architecture
• SGLang Model Gateway / Shepherd gateway routing and lifecycle needs

Each child issue contains the relevant engine-specific requirements and proposed portable boundary.

Proposed contract shape

Keep the protocol modular rather than growing one monolithic service:

• Inference: generation plus portable embedding, classification, scoring, and reranking tasks
• Discovery: typed capabilities, conformance profiles, health, and load streams
• Lifecycle: application generation sessions and request/session ownership
• KV: lease-based disaggregated KV session orchestration
• Media: encoder-role and EPD media exchange
• Adapters: generic LoRA and prompt-adapter lifecycle
• Observability: request progress and standardized operational state
• Administration: authenticated, policy-controlled runtime operations

A minimal mandatory core should remain deployable by small engines. Optional services and RPC groups should be declared through machine-readable capability/profile negotiation.

Presence, defaults, validation, and compatibility

The child proposals must define their own field presence and validation rules. Across the roadmap:

• Absence of an optional profile means the capability is unsupported, not silently emulated.
• Capability support is distinct from instantaneous health, readiness, and load.
• Defaults must be deterministic and documented.
• Unknown enum values and fields must be handled compatibly.
• Opaque payloads are limited to explicitly versioned extension points.
• Any cache, tokenizer, tensor, adapter, or session identity crossing a process boundary must carry enough version/fingerprint information to reject incompatible reuse.

Lifecycle and failure semantics

Every stateful or streaming proposal must define:

• acceptance and ordering points
• idempotency and retry behavior
• cancellation, deadline, and disconnect behavior
• ownership, leases, renewal, and terminal states
• partial failure and rank/worker failure behavior
• deterministic cleanup and resource reclamation
• reconnectability for long-running operations where appropriate

This tracker is complete only when the child contracts are merged, generated bindings and documentation are updated, and the conformance suite demonstrates portable behavior across TensorRT-LLM, SGLang, and vLLM adapters.

Discovery and capabilities

Typed capability/profile discovery is a prerequisite for safely adding optional services. Clients must be able to determine support without probing RPC failures or relying on engine names and versions.

Alternatives considered

• Keep the narrow generation-only v1 surface: simpler, but leaves frameworks dependent on engine-specific side channels.
• Put every method on one service: easy to discover manually, but weakens modularity, security boundaries, and optional implementation.
• Mirror the union of engine APIs: maximizes exposure but bakes unstable implementation details into the portable contract.
• Use generic JSON/options maps: expedient but prevents validation, compatibility analysis, and reliable code generation.

Out of scope

• OpenAI HTTP API pass-through
• framework-specific chat templates, reasoning parsers, tool calling, or MCP behavior
• Kubernetes worker registration and control-plane APIs
• routing, retry, autoscaling, or placement policy
• Prometheus exposition formats
• engine-internal scheduler, kernel, or speculative-decoding implementation details

Checklist

• The proposal is intended to be portable across engines and frameworks.
• Documentation and conformance-test requirements are identified.
• All child proposals are resolved or explicitly deferred with rationale.
• A cross-engine/profile support matrix is published.
• Conformance tests cover the mandatory core and each accepted optional profile.