Bug: Token rate limiter uses inaccurate heuristic tokenizer for input token accounting

[nXtCyberNet]

/kind enhancement

What happened

The token rate limiter currently uses:

tokenizer: tokenizer.NewSimpleEstimateTokenizer()

which estimates tokens using a /4 character heuristic instead of model-accurate tokenization.

This causes inaccurate input token rate limiting:

• code-heavy prompts are significantly over-counted
• some model families may be under-counted
• users can hit limits earlier or later than expected

The issue only affects input token accounting. Output token accounting is already accurate because it uses actual completion_tokens returned by the model response.

A TikToken-based tokenizer already exists in the codebase, but using it directly as default also introduces problems:

• TikToken is GPT/OpenAI specific
• token estimation becomes inaccurate for non-GPT models (Claude, LLaMA, Mistral, etc.)
• remote tokenizer calls can increase latency under concurrency

Related discussion:
[#1100 comment](#1100)

That discussion also highlights another issue:

• tokenizer requests currently randomly select vLLM pods
• tokenization latency becomes dependent on inference pod load
• under high concurrency this can create unpredictable admission latency

So the current local estimator solves latency and model-compatibility concerns, but its estimation error can still become large enough to affect practical quota enforcement.

---

[nXtCyberNet]

what you guys this whats the best way to solve this too as this inaccurate token count will cause the change in billing amount significantly and also for a paas the complience will also became a problem. this same problem was with the kvcacheaware routing plugin too. ?

[shivansh-gohem]

Thanks for bringing this up! I've been looking into the token rate limiting codebase and the issues raised in #1100. It's clear that the current len(prompt) / 4 heuristic is insufficient for billing/compliance, and routing tokenization to random inference pods adds unpredictable latency.

To solve this while balancing accuracy and latency, I see three potential architectural approaches we could take:

1. Native Go Local Tokenization: We integrate a Go library (like github.com/sugarme/tokenizer or github.com/daulet/tokenizers) directly into kthena-router. Operators would mount the respective tokenizer.json files. This eliminates network latency entirely and is highly accurate, but adds overhead for managing tokenizer configs in the router environment.
2. Dedicated Tokenizer Microservice: We decouple this from vLLM and deploy a lightweight, highly-concurrent tokenizer service (e.g., Rust + HF tokenizers) that the router calls. This offloads tokenization to a dedicated scalable component, ensuring low latency, but adds a new microservice to the deployment stack.
3. CRD-Driven Adaptive Tokenization: We expand ModelRoute or ModelServer CRDs to let operators explicitly define the tokenizer strategy per model (e.g., TikToken for GPT models, remote tokenizer endpoint, or specific heuristic ratios).

What are your thoughts on these options? Which direction do you feel aligns best with the project's goals for latency versus architectural complexity?

[hzxuzhonghu]

which estimates tokens using a /4 character heuristic instead of model-accurate tokenization.

The original intention is not to add too much latency, we all know the accurate tokenizer is costy of cpu. Prompt length/4 is a approximate.

[hzxuzhonghu]

I am not an expert on tokenizer, wanna to ask if we introduce a separate service serving tokenizing, what will the service need to download for different models. like deepseek, qwen, or other opensource models.

And also if we want to support proxying requests to thirdparty maas, how could we do that?

[shivansh-gohem]

@hzxuzhonghu That's a very valid concern! Regarding latency and cpu cost, using a heuristic definitely helps for latency, but the issue is when we strictly need accurate tracking for billing.

To answer your questions:

1. What will the service need to download for different models?
   The tokenizer service would only need the tokenizer configuration files (e.g., tokenizer.json, tokenizer_config.json, and occasionally vocab.json/merges.txt). It does not need to download the heavy model weights (safetensors or bin files). These tokenizer files are very lightweight (usually under a few megabytes). We could allow operators to mount them via a ConfigMap or PVC, or download them automatically at startup, just like vLLM does, so they can even run in air-gapped environments without any issue.

2. How to support proxying requests to third-party MaaS (like OpenAI, Anthropic, Gemini)?
   For third-party MaaS, we usually don't have access to the exact proprietary tokenizer (though we do have tiktoken for OpenAI). We can handle this through a fallback mechanism:

   • Allow the operator to configure a "compatible" local tokenizer in the CRD (e.g., use cl100k_base for OpenAI models, which runs very fast locally).
   • If no specific tokenizer is configured for a MaaS model, we seamlessly fallback to the fast len(prompt) / 4 approximation that is currently in place.

This hybrid approach ensures that we keep the ultra-low latency heuristic as the default, while providing an opt-in path for operators who require precise token counting for billing and compliance.

What do you think of making the accurate tokenization an opt-in configuration in the CRD, and keeping the /4 heuristic as the default?

[nXtCyberNet]

@shivansh-gohem – Hi, just want to add a small correction: OpenAI and Anthropic already return token info through the API response, so we don't need local tokenization for those. same with openai api-compadible schemas
For the local tokenizer problem though—the main issue is we can't just download config files at startup based on model name. Even if we use huggingFace model SHA codes, that only works for HF models. Different registries use different identification schemes,
So each model uses a different tokenizer.json file, and we'd need a way to map each model (across all registries) to its correct tokenizer config. This means:

Model name uniqueness isn't standardized across registries
We'd need a model registry that tracks which tokenizer belongs to which model
Managing multiple model versions becomes an overhead

If we create a tokenizer service, we first need to solve model identity/versioning that works across registries—not just HuggingFace. Everything else cascades from that.
Everything's solvable, but it requires refactoring around model uniqueness first. Thanks!

[shivansh-gohem]

@nXtCyberNet Thanks for the excellent points!

Regarding the MaaS APIs: You are right that OpenAI and Anthropic return usage stats in their responses, which works perfectly for post-request quota deduction. However, for strict pre-request rate limiting (i.e., rejecting requests at the router before they are sent to the provider to prevent cost overruns or hitting upstream rate limits), we still need to estimate or count the input tokens locally before forwarding the request.

For the local tokenizer and model identity issue: I completely agree that building a centralized model registry mapping would be a massive headache due to non-standardized names across registries.

Instead of solving global model uniqueness, what if we push this responsibility to the cluster operator via our CRDs?

We could introduce a tokenizerRef field inside the ModelRoute (or ModelServing) CRD. This way, the operator explicitly declares the tokenizer source for the model they are deploying or proxying. For example:

apiVersion: networking.volcano.sh/v1alpha1
kind: ModelRoute
metadata:
  name: my-custom-model
spec:
  tokenizerRef:
    # Option A: Pull directly from HF Hub
    huggingface:
      repoId: "mistralai/Mistral-7B-Instruct-v0.2"
    # Option B: Use a local ConfigMap containing tokenizer.json
    configMap:
      name: custom-model-tokenizer-cm

By doing this:

1. We completely sidestep the need for a global model registry or standardizing model names.
2. The tokenizer service just blindly loads the tokenizer.json from the specified tokenizerRef.
3. If an operator omits tokenizerRef, we cleanly default to the len(prompt) / 4 heuristic.

This keeps the system decentralized and un-opinionated about model registries, while giving operators the exact control they need. How does this CRD-driven approach sound to you?

[hzxuzhonghu]

The tokenizer service would only need the tokenizer configuration files (e.g., tokenizer.json, tokenizer_config.json, and occasionally vocab.json/merges.txt). It does not need to download the heavy model weights (safetensors or bin files). These tokenizer files are very lightweight (usually under a few megabytes). We could allow operators to mount them via a ConfigMap or PVC, or download them automatically at startup, just like vLLM does, so they can even run in air-gapped environments without any issue.

Yeah, different models may use different tokenizer, so the files can vary, right?

[hzxuzhonghu]

What do you think of making the accurate tokenization an opt-in configuration in the CRD, and keeping the /4 heuristic as the default?

Can we know exactly which tokenizer.json, tokenizer_config.json, a model like qwen, deepseek use? If not, we should provide a field in the api to let the tokenizer service automatically download according to the api configuration

[nXtCyberNet]

Regarding the MaaS APIs: You are right that OpenAI and Anthropic return usage stats in their responses, which works perfectly for post-request quota deduction. However, for strict pre-request rate limiting (i.e., rejecting requests at the router before they are sent to the provider to prevent cost overruns or hitting upstream rate limits), we still need to estimate or count the input tokens locally before forwarding the request.

currently there is no maas support for now , its in the lfx term 2 as a project so arch can be changed for this as much as ik

and other thing looks cool ,

[shivansh-gohem]

@hzxuzhonghu Yes, exactly! Different models use different tokenizer configurations, so the files definitely vary. Because we can't always know exactly which tokenizer.json a model uses just from its name, adding a field to the API is the perfect solution. The tokenizerRef field I proposed above does exactly that—it explicitly tells the tokenizer service where to download the config from (e.g., via a HuggingFace repoId), so the service doesn't have to guess or maintain a complex mapping.

@nXtCyberNet Ah, that makes sense regarding MaaS support being scheduled for LFX Term 2. Good to know! Since that is the case, we can focus the architecture purely on the local/self-hosted model tokenization for now, using the tokenizerRef approach.

If we are all aligned on the tokenizerRef CRD design and keeping the /4 heuristic as the default, I'd be happy to start looking into drafting a more formal proposal or beginning the implementation!

[nXtCyberNet]

What do you think of making the accurate tokenization an opt-in configuration in the CRD, and keeping the /4 heuristic as the default?

Can we know exactly which tokenizer.json, tokenizer_config.json, a model like qwen, deepseek use? If not, we should provide a field in the api to let the tokenizer service automatically download according to the api configuration

yup its easy each model comes with there tockenizer file and the registories also have these files with other model weight files ,
for example -
https://huggingface.co/Qwen/Qwen3.5-397B-A17B/tree/main
the files required for tokenization is

• tokenizer_config.json
• tokenizer.json

Some models also include:

• vocab.json + merges.txt — used by older GPT-2 style BPE tokenizers (fallback if tokenizer.json is absent)
• special_tokens_map.json — maps special token names to IDs (often embedded in tokenizer_config.json already)

[nXtCyberNet]

@hzxuzhonghu @shivansh-gohem , i have a better solution and design for the tokenizer then the current design given by
@shivansh-gohem , see i think we can just add a small identifier then crd change

i was thinking to use a side car approach in which we can add a side car contain with the router pod that manages this issue , for the tokenizer we are gonna use - https://github.com/daulet/tokenizers , its a go binding for the hugging face tokenizer , also for the concern about the model registory issue it was already solved by the lora adaptor that uses

modelURI:
  -  pvc://<location>/llama-3-8b <- for local model
  -   hf://<org>/<repo>  <- for huggingface registoy 

so , its not an issue for the multi model registory

and also its gonna be configurable per workload based on latency vs accuracy needs — operators who need billing/compliance accuracy enable the sidecar, everyone else keeps the heuristic.

what you think about this , if you like my approach i can create the proposal for the same thanks!