refactor(api): separate deployment and routing metadata#22
Merged
Conversation
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ModelInfointoServerInfo.capacityDeploymentCapacity.max_lorasModelInfoKvSessionRef.dp_rankas the authoritative KV-affinity valueWhy
Model identity and deployment capacity have different lifecycles. KV layout, allocatable blocks, scheduler ceilings, and resident LoRA limits describe a running server configuration, while context/output limits and inference capabilities describe a selected model.
Routing, admission, and tracing are properties of the RPC call rather than model input. Keeping them in gRPC metadata lets frameworks route or reject a call without decoding and rewriting its protobuf payload, while preserving
KvSessionRef.dp_rankfor actual KV-session affinity.Metadata contract
openengine-routing-key: opaque application routing keyopenengine-target-dp-rank: base-10uint32routing targetopenengine-priority: base-10int32admission prioritytraceparentandtracestate: W3C Trace ContextThe
openengine-namespace is reserved. Malformed or repeated OpenEngine keys returnINVALID_ARGUMENT. If a visible target-rank key disagrees withKvSessionRef.dp_rank, the KV-session affinity is authoritative and the request is rejected.Compatibility
This is an intentional pre-adoption API refinement:
ModelInfofields 6–9 move toServerInfo.capacityGenerateRequest.priorityandGenerateRequest.metadatamove to gRPC metadataTaskRequestContextKvOptions.data_parallel_rankis removed in favor of transport routing metadata andKvSessionRef.dp_rankaffinityValidation
protocdescriptor compilation