Skip to content

refactor(api): separate deployment and routing metadata#22

Merged
connorcarpenter15 merged 2 commits into
mainfrom
refactor/deployment-routing-metadata
Jul 15, 2026
Merged

refactor(api): separate deployment and routing metadata#22
connorcarpenter15 merged 2 commits into
mainfrom
refactor/deployment-routing-metadata

Conversation

@connorcarpenter15

@connorcarpenter15 connorcarpenter15 commented Jul 15, 2026

Copy link
Copy Markdown
Member

Summary

  • move deployed KV and scheduler capacity out of ModelInfo into ServerInfo.capacity
  • add DeploymentCapacity.max_loras
  • retain only effective per-model context and output limits in ModelInfo
  • define canonical gRPC metadata keys for routing, admission priority, and W3C trace propagation
  • remove arbitrary request metadata maps and the duplicate request-level data-parallel rank
  • use KvSessionRef.dp_rank as the authoritative KV-affinity value
  • keep concise field-local documentation as inline comments

Why

Model identity and deployment capacity have different lifecycles. KV layout, allocatable blocks, scheduler ceilings, and resident LoRA limits describe a running server configuration, while context/output limits and inference capabilities describe a selected model.

Routing, admission, and tracing are properties of the RPC call rather than model input. Keeping them in gRPC metadata lets frameworks route or reject a call without decoding and rewriting its protobuf payload, while preserving KvSessionRef.dp_rank for actual KV-session affinity.

Metadata contract

  • openengine-routing-key: opaque application routing key
  • openengine-target-dp-rank: base-10 uint32 routing target
  • openengine-priority: base-10 int32 admission priority
  • traceparent and tracestate: W3C Trace Context

The openengine- namespace is reserved. Malformed or repeated OpenEngine keys return INVALID_ARGUMENT. If a visible target-rank key disagrees with KvSessionRef.dp_rank, the KV-session affinity is authoritative and the request is rejected.

Compatibility

This is an intentional pre-adoption API refinement:

  • ModelInfo fields 6–9 move to ServerInfo.capacity
  • GenerateRequest.priority and GenerateRequest.metadata move to gRPC metadata
  • task priority and arbitrary metadata move out of TaskRequestContext
  • KvOptions.data_parallel_rank is removed in favor of transport routing metadata and KvSessionRef.dp_rank affinity
  • removed field names and numbers remain available for reuse during this pre-adoption phase

Validation

  • Buf lint
  • direct protoc descriptor compilation
  • Markdown lint
  • documentation link checks
  • DCO sign-off

Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15 connorcarpenter15 marked this pull request as ready for review July 15, 2026 17:44
@connorcarpenter15 connorcarpenter15 merged commit a2ad663 into main Jul 15, 2026
5 checks passed
@connorcarpenter15 connorcarpenter15 deleted the refactor/deployment-routing-metadata branch July 15, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant