[Protocol]: Add application generation sessions and optional streaming input

[connorcarpenter15]

Existing proposals

• I searched the existing issues and did not find this proposal.

Intended consumers

Both direct engine clients and distributed frameworks.

Problem and use case

KvSessionRef is a P/D transfer handle, not an application-level generation session. OpenEngine cannot represent multi-turn state retained across requests or explicitly open, inspect, renew, and close it. It also cannot stream input because Generate requires one complete unary request before output begins.

These features share ownership, lease, affinity, and cleanup concerns but must remain distinct from P/D handoff.

Evidence from engine protocols

• SGLang exposes open/close session operations and continuation parameters in its engine API.
• vLLM bidirectional NIXL transfer uses stateful multi-turn KV information.
• Realtime text/audio/video input requires client-to-server chunks and cannot be expressed by unary-request/server-streaming Generate.

Proposed contract

service OpenEngineSessions {
  rpc OpenGenerationSession(OpenGenerationSessionRequest)
      returns (GenerationSession);
  rpc GetGenerationSession(GetGenerationSessionRequest)
      returns (GenerationSession);
  rpc RenewGenerationSession(RenewGenerationSessionRequest)
      returns (GenerationSession);
  rpc CloseGenerationSession(CloseGenerationSessionRequest)
      returns (CloseGenerationSessionResponse);
}

service OpenEngineStreaming {
  rpc GenerateStream(stream GenerateStreamRequest)
      returns (stream GenerateResponse);
}

message GenerationSessionRef {
  string session_id = 1;
  string owner_instance_id = 2;
}

Keep application sessions distinct from KvSessionRef. Define model binding, ownership, lease, expiry, and idempotent close; allow unary Generate to continue a session; begin with portable append-only continuation and capability-gate branching/replacement. An optional bidirectional RPC uses one start frame, ordered input chunks, explicit end-of-input, and one logical generation request per stream.

Presence, defaults, and validation

• Caller-supplied IDs are explicit, non-empty, and unique; server-minted IDs remain supported.
• Timeout absence requests the engine default; zero has one unambiguous meaning.
• Model/adapter compatibility is fixed or validated on every continuation.
• Owner mismatch is never silently accepted by another engine instance.
• Stream sequence numbers define duplicate and gap handling.
• Start is first and unique; input after end-of-input is invalid.
• Unsupported mutation or early-input execution returns a capability error.

Request and response lifecycle

Opening reserves logical state—and possibly KV capacity—until close, expiry, eviction, or engine failure. The response reports actual lease and state. Close is idempotent; expired and unknown sessions are distinguishable.

For GenerateStream, the client sends start, ordered chunks, then end-of-input. The server may emit output early only when advertised. Exactly one terminal generation outcome or terminal gRPC error ends the request. Cancellation, deadline, disconnect, and Abort(request_id) stop active work without implicitly closing a reusable session unless requested.

Discovery and capability impact

Advertise application-session support; append/branch/replace/drop-output modes; lease limits; migration or owner-affinity requirements; streaming input by modality; early-inference support; and chunk/session limits. Unary Generate remains usable without either optional profile.

Alternatives considered

• Reuse KvSessionRef: owners, lifetimes, and failure semantics differ.
• Resend full prompts with prefix caching: useful fallback but no retained-state guarantee.
• Add only session_id: lacks allocation, expiry, cleanup, routing, and discovery.
• Make existing Generate bidirectional: breaks the simple core; a sibling optional RPC preserves it.

Out of scope

Gateway chat history, OpenAI Responses state, tool/MCP sessions, initial cross-engine opaque-state migration, or multiplexing several requests over one stream.

Proposal checklist

• The behavior is portable and does not expose one engine's internal scheduler model as the common API.
• I have identified the corresponding documentation changes.
• Define session identity, ownership, lease, expiry, renewal, and idempotent close.
• Add session continuation to unary generation without reusing KvSessionRef.
• Define optional bidirectional framing, ordering, cancellation, and terminal behavior.
• Add independent session and streaming-input capabilities.
• Add conformance cases for expiry, owner mismatch, duplicates, cancellation, and cleanup.
• Update docs/api.md and generated package documentation.