[Protocol]: Add tokenizer and KV cache-key interoperability APIs

[connorcarpenter15]

Existing proposals

• I searched the existing issues and did not find this proposal.

Intended consumers

Both direct engine clients and distributed frameworks.

Problem and use case

Distributed schedulers need to tokenize before selecting a worker for admission, token accounting, P/D routing, and precise prefix-cache-aware routing.

OpenEngine currently exposes only tokenizer modes, KV block size, and engine-produced KV events. It does not provide an authoritative tokenizer identity/revision, tokenizer artifacts, portable tokenize/detokenize operations, a complete cache-key specification, or an authoritative key-computation fallback. A router can therefore tokenize or hash differently while appearing operational, causing missed reuse or incorrect locality scores.

Evidence from engine protocols

• SGLang's native gRPC service exposes typed Tokenize and Detokenize.
• SMG's engine worker services expose GetTokenizer; its common protocol streams tokenizer bytes with a fingerprint.
• llm-d precise prefix-cache-aware routing requires router tokenizer and block-size agreement.
• vLLM cache keys include chained token blocks and request-dependent context such as adapters, multimodal identifiers, salts, and prompt embeddings; see kv_cache_utils.py.

Proposed contract

service OpenEngineTokenizer {
  rpc GetTokenizerInfo(GetTokenizerInfoRequest) returns (TokenizerInfo);
  rpc GetTokenizerArtifacts(GetTokenizerArtifactsRequest)
      returns (stream TokenizerArtifactChunk);
  rpc Tokenize(TokenizeRequest) returns (TokenizeResponse);
  rpc Detokenize(DetokenizeRequest) returns (DetokenizeResponse);
  rpc GetCacheKeySpec(GetCacheKeySpecRequest) returns (CacheKeySpec);
  rpc ComputeCacheKeys(ComputeCacheKeysRequest)
      returns (ComputeCacheKeysResponse);
}

Provide a compatibility fingerprint, format and artifact metadata, special-token IDs, chunked checksummed artifacts, authoritative tokenization, a versioned cache-key profile, deterministic test vectors, and engine-side key computation when a client cannot reproduce the algorithm. Key context covers model/tokenizer revision, adapter set, salt, multimodal or prompt-embedding identity, and KV group/spec metadata.

Presence, defaults, and validation

• Empty text and empty token sequences have explicit valid semantics.
• Special-token, truncation, and normalization options use presence-aware fields.
• Artifact chunks carry ordered offsets, total size, and digest; paths and traversal are invalid.
• Client-computable profiles are valid only when all required fields and fingerprints are present.
• Unknown algorithms are never guessed; clients compute remotely or treat precise matching as unsupported.
• Partial blocks, empty prompts, salts, adapters, multimodal inputs, and prompt embeddings have defined key behavior.

Request and response lifecycle

Tokenize, Detokenize, and key computation are deterministic unary RPCs for a given fingerprint and options. Artifact transfer is server-streaming and is installed only after size/digest verification; cancellation discards incomplete artifacts. Any behavior change produces a new fingerprint.

Discovery and capability impact

ModelInfo advertises tokenizer fingerprint, supported operations, artifact format/availability, cache-key profile/version, whether keys are client-computable, and whether authoritative computation is available. KV event sources and hashes identify the compatible cache-key profile.

Alternatives considered

• Independent model downloads cannot prove parity with engine behavior.
• Remote tokenization alone adds a routing-path dependency and does not solve hashing.
• Artifacts alone do not cover proprietary tokenizers or hash semantics.
• ScorePrefixCache centralizes a latency-sensitive routing operation and does not enable external indexing.

Out of scope

Chat-template rendering, reasoning/tool parsing, credentialed downloads, cache injection, or forcing one hash algorithm on every engine.

Proposal checklist

• The behavior is portable and does not expose one engine's internal scheduler model as the common API.
• I have identified the corresponding documentation changes.
• Define tokenizer identity, artifact, tokenize, and detokenize messages.
• Define versioned cache-key profiles and authoritative computation fallback.
• Cover adapters, salts, multimodal inputs, prompt embeddings, and KV groups.
• Add cross-language deterministic test vectors.
• Document artifact integrity, revision changes, and event compatibility.
• Update docs/api.md and generated package documentation.