[Protocol]: Add a secured runtime administration service

[connorcarpenter15]

Existing proposals

• I searched the existing issues and did not find this proposal.

Intended consumers

Both direct engine clients and distributed frameworks.

Problem and use case

OpenEngine colocates public inference with lifecycle mutations but has no portable way to flush caches, pause/resume admission, release/restore GPU resources, update weights, collect profiles, or reconfigure serving roles/topology. Distributed frameworks and operators must retain engine-specific management clients.

These operations are privileged and often long-running. Define a separately bindable and separately authorized administration service rather than exposing destructive controls with public inference.

Evidence from engine protocols

• vLLM EngineClient exposes pause/resume, prefix/encoder/MM cache reset, sleep/wake, profiling, weight update, and elastic-EP scaling.
• SGLang's RL/runtime API exposes memory release/resume and disk, tensor, and distributed weight updates.
• TensorRT-LLM's LLM API exposes runtime resource and model controls.
• llm-d calls for management APIs and dynamic P/D reassignment.

Proposed contract

service OpenEngineAdmin {
  rpc FlushCaches(FlushCachesRequest) returns (stream AdminOperationEvent);
  rpc Pause(PauseRequest) returns (stream AdminOperationEvent);
  rpc Resume(ResumeRequest) returns (stream AdminOperationEvent);
  rpc ReleaseResources(ReleaseResourcesRequest)
      returns (stream AdminOperationEvent);
  rpc RestoreResources(RestoreResourcesRequest)
      returns (stream AdminOperationEvent);
  rpc UpdateWeights(UpdateWeightsRequest)
      returns (stream AdminOperationEvent);
  rpc StartProfile(StartProfileRequest) returns (StartProfileResponse);
  rpc StopProfile(StopProfileRequest) returns (StopProfileResponse);
  rpc ReconfigureServing(ReconfigureServingRequest)
      returns (stream AdminOperationEvent);
  rpc GetOperation(GetAdminOperationRequest) returns (AdminOperation);
  rpc CancelOperation(CancelAdminOperationRequest)
      returns (CancelAdminOperationResponse);
}

Use typed, capability-gated cache/resource/pause/source/reconfiguration enums rather than arbitrary method names or maps. The service must be deployable on a different listener and protected independently of inference.

Presence, defaults, and validation

• UNSPECIFIED and empty selectors are invalid; dangerous all-resource behavior is never the proto3 default.
• Mutations carry a non-empty idempotency key. Reuse with the same payload returns the operation; conflicting reuse returns ALREADY_EXISTS.
• Optional expected revision provides compare-and-swap semantics; mismatch returns ABORTED.
• Pause mode is required. Restore/resume use epochs so stale requests fail.
• Models, roles, resources, sources, profiler kinds, and reconfiguration fields are validated against capabilities before acceptance.

Request and response lifecycle

Pre-acceptance failures use gRPC status. Once accepted, the first event contains a stable operation ID and sequence; exactly one terminal complete/failed/cancelled event follows. Stream cancellation detaches observation and does not ambiguously roll back a mutation. GetOperation permits reconciliation and explicit CancelOperation is best-effort and capability-gated.

Distributed mutations complete only when all participating ranks commit. Partial failure identifies ranks and fails closed. Weight update is atomic or rolls back. Pause distinguishes drain, abort, and keep-resumable-work behavior. Profiling is idempotent and returns an artifact reference.

Discovery and capability impact

Advertise an optional admin service profile with supported operations, cache/resource kinds, pause modes, weight sources, profiler kinds, reconfigurable fields, cancellation, and limits. Core discovery reveals profile presence; sensitive details may require authorization. Standard gRPC health is published separately for admin and inference.

Alternatives considered

• Extend Drain: cannot express resume, frozen queues, caches/resources, weights, profiling, or reconfiguration.
• Put methods on OpenEngine: prevents a clean inference/admin authorization boundary.
• Pass arbitrary method names or JSON: reproduces unsafe engine-private APIs.
• Put credentials in protobuf: authentication belongs in transport metadata/interceptors.

Out of scope

Scheduler internals, autoscaling policy, a universal checkpoint/tensor format, arbitrary engine method execution, or standardizing credentials in protobuf.

Proposal checklist

• The behavior is portable and does not expose one engine's internal scheduler model as the common API.
• I have identified the corresponding documentation changes.
• Add admin messages/service and capability profile.
• Document authorization, idempotency, rank atomicity, and terminal states.
• Add conformance tests for retry, disconnect, partial-rank failure, stale epochs/revisions, and unsupported operations.
• Document which existing Abort, Drain, and adapter mutations are privileged and their migration path.