[Protocol]: Define conformance profiles and streaming health/load discovery

[connorcarpenter15]

Existing proposals

• I searched the existing issues and did not find this proposal.

Intended consumers

Both direct engine clients and distributed frameworks.

Problem and use case

The single OpenEngine service mixes mandatory generation, distributed KV/events, lifecycle, LoRA, and observability. Current discovery reports schema revisions and selected booleans, but clients cannot discover which optional RPC groups, task families, session modes, admin operations, or feature combinations are implemented except by probing and handling UNIMPLEMENTED.

A comprehensive protocol needs a small mandatory core, versioned optional conformance profiles, typed capabilities, standard gRPC health interoperability, and a streaming load signal for scheduler fleets.

Evidence from engine protocols

• vLLM EngineClient exposes supported tasks and many conditionally enabled operations.
• SGLang, vLLM, and TensorRT-LLM conditionally enable tasks, adapters, disaggregation, multimodal, resource, and management features.
• gRPC health checking defines standard unary Check and streaming Watch.
• Current GetLoad is polling-only and does not scale as a fleet-wide live signal.

Proposed contract

service OpenEngineDiscovery {
  rpc GetEngineInfo(GetEngineInfoRequest) returns (EngineInfo);
  rpc GetModelInfo(GetModelInfoRequest) returns (ModelInfo);
  rpc GetCapabilities(GetCapabilitiesRequest) returns (Capabilities);
}

service OpenEngineObservability {
  rpc GetLoad(GetLoadRequest) returns (LoadInfo);
  rpc WatchLoad(WatchLoadRequest) returns (stream WatchLoadResponse);
  rpc SubscribeRuntimeEvents(SubscribeRuntimeEventsRequest)
      returns (stream SubscribeRuntimeEventsResponse);
}

message ServiceProfile {
  string name = 1;
  uint32 major = 2;
  uint32 minor = 3;
}

message Capabilities {
  repeated ServiceProfile profiles = 1;
  GenerationCapabilities generation = 10;
  ObservabilityCapabilities observability = 11;
  KvCapabilities kv = 12;
  AdapterCapabilities adapters = 13;
  AdminCapabilities admin = 14;
}

Define a mandatory core generation/discovery/cancellation/standard-health profile and atomic optional profiles for scheduling/load, KV handoff/events, tasks, encoder/EPD, sessions, adapters, runtime events, and admin. Each profile lists required RPCs/messages and typed limits. Sibling services provide policy and security boundaries. Require grpc.health.v1.Health/Check and /Watch per advertised service while retaining richer OpenEngine diagnostics.

Presence, defaults, and validation

• Core profile is always present.
• Profile names are canonical/non-empty; major versions are positive; duplicates are invalid.
• Omitted optional profile means unsupported, not temporarily unavailable.
• Capability submessages are valid only for advertised profiles.
• Watch intervals use presence-aware defaults; zero behavior is explicit.
• The server reports effective clamped intervals.
• Load instance identity matches discovery, timestamps are monotonic per stream, and sequence numbers increase.

Request and response lifecycle

Capabilities are stable for an engine configuration epoch; legal-request changes increment a capability revision and emit a change event or require reconnect.

WatchLoad begins with effective stream settings, emits an initial full snapshot by default, then updates/heartbeats. The server may coalesce under backpressure but delivered snapshots remain complete and gaps are visible. Cancellation only unsubscribes. GetLoad remains the one-shot fallback.

Standard health is per service: discovery/admin may remain serving while inference is paused, draining, or not serving.

Discovery and capability impact

Every protocol addition declares whether it extends core or defines an optional profile, its version, model/role scope, typed limits, prerequisites, and incompatible combinations. Static support, current availability, and current load remain separate concepts: capabilities, health, and load respectively.

Alternatives considered

• Probe every RPC: expensive, racy, and ambiguous.
• Flat feature booleans do not version behavior or express combinations/limits.
• Require every RPC: forces fake unsupported implementations.
• Reflection reports method existence, not configured/model support.
• Prometheus is not a request-path scheduler contract and health is categorical rather than load.

Out of scope

Prometheus replacement, router/autoscaler policy, transport keepalive, mandatory reflection, or encoding volatile health/load as static capability flags.

Proposal checklist

• The behavior is portable and does not expose one engine's internal scheduler model as the common API.
• I have identified the corresponding documentation changes.
• Publish mandatory-core and optional-profile conformance/version rules.
• Add discovery/profile messages and split or alias sibling services.
• Require/test standard gRPC Health Check/Watch.
• Add WatchLoad lifecycle, backpressure, heartbeat, and fallback tests.
• Require every capability to declare profile, scope, defaults, limits, and incompatible combinations.