feat(qwen35): add joint prefix cache

[Ke-Wng]

Description

This PR is a follow-up of #257 and #423. It adds opt-in prefix caching for Qwen3.5 on both single-GPU and tensor-parallel serving.

Qwen3.5 is a hybrid model, so full-attention KV alone is not a valid reusable prefix. A cache hit is reported only when full-attention KV and the complete recurrent/conv state are available at the same token boundary.

Prefix caching remains disabled by default and can be enabled with:

--qwen35-prefix-cache-mib <MiB>

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

What changed

• Unified Qwen3.5 serving and accuracy paths on KvCacheManager / RequestKv, with exact-boundary joint KV prefix lookup and attachment.
• Added a bounded GPU recurrent/conv snapshot cache with 256-token boundaries, content-hashed KV keys, pinning, LRU eviction, and cache statistics.
• Added suffix-only prefill after joint hits; non-aligned tails continue normally without creating snapshots.
• Added TP support with controller-owned logical KV state, rank-local KvBuffer / RecurrentStateStore, shared snapshot slot IDs, and capacity limited by the smallest rank.
• Added TokenEvent::Scheduled.cached_tokens reporting.

Core components

Component	Responsibility
Qwen35PrefixCache	Coordinates request KV, joint lookup, reservation, publication, restore validation, and metrics.
PrefixCacheState	Owns prefix entries, slot allocation, pinning, LRU, reservation, publication, and eviction.
PrefixEntry	Owns one recurrent slot and the KV lease for a reusable prefix.
PrefixGuard	Pins an entry during restore and releases the pin when dropped.
PrefixReservation	Represents a reserved slot before physical recurrent-state copy completes.
RequestKv	Owns one request's logical KV pages and transactional lifecycle.
RecurrentStateStore	Stores rank-local recurrent/conv tensors for cached snapshots.

Inference flow

1. Allocate — Check admission limits, create request-local RequestKv and recurrent state, then run begin_request().
2. Lookup and restore — Select the longest boundary containing both KV and a recurrent snapshot. The executor restores the selected recurrent_slot, then finish_restore() validates the positions and reports cached_tokens; otherwise the request starts cold.
3. Prefill — Call schedule_prefill() and prefill_view(), run the remaining prompt suffix, and call apply_prefill() only after a successful forward.
4. Publish snapshots — At each non-zero 256-token boundary, call reserve_prefix(). If a reservation is returned, save the recurrent state and call publish_prefix() after the copy succeeds. Abort the reservation on failure. Non-aligned tails apply without creating a snapshot.
5. Decode — For each output token, call schedule_decode(), run the model through decode_view(), sample, and call apply_decode().
6. Release — Release request-owned KV and recurrent state. Published PrefixEntry objects retain their KV leases and snapshots for later requests; non-published KV returns to the pool.

Tensor-parallel differences

1. Central ownership — The controller owns the only KvCacheManager, RequestKv map, PrefixCacheState, and page-id namespace.
2. Rank-local storage — Each rank owns its physical KvBuffer and RecurrentStateStore; logical capacity is limited to the smallest rank-local KV capacity.
3. Shared page tables — The controller schedules KV once and sends the same KvView page IDs to every rank.
4. All-rank restore — A hit is reported only after every rank restores the same recurrent slot and confirms the same token boundary.
5. All-rank publication — A prefix is published only after every rank saves into the same slot number and confirms the boundary.

Detailed prefix_cache.rs API sequence

1. begin_request() creates RequestKv, probes the longest joint boundary, and attaches the selected KV prefix.
2. The caller restores PrefixGuard::recurrent_slot() into request-local recurrent state on each rank.
3. finish_restore() validates KV and recurrent positions, records the hit, returns cached_tokens, and releases the restore pin.
4. schedule_prefill() / schedule_decode() reserve pages; prefill_view() / decode_view() expose the kernel page table.
5. apply_prefill() / apply_decode() apply a successful forward; revert_schedule() rolls back failed reservations.
6. reserve_prefix() returns an optional PrefixReservation; the caller saves the physical recurrent state and then calls publish_prefix() or abort_prefix().
7. release_request() releases request-owned KV and preserves cold behavior when caching is disabled.

Consistency guarantees

• A reusable boundary must contain both registered full-attention KV and a complete recurrent/conv snapshot for the same token lineage and token position.
• Snapshot entries are published only after every required physical copy succeeds.
• KV-only and snapshot-only candidates fall back to cold prefill and are never reported as hits.
• Forward failures revert scheduled KV pages; failed snapshot copies abort their reservations.
• Request completion, rejection, and disconnect paths release request-owned state.

Validation

• Core tests passed: openinfer-kv-cache (19), Qwen3.5 release library (77; 6 TP2 cases ignored in the normal run), and Qwen3.5 server config (9).
• Prefix-cache tests cover joint-boundary restore, extension, echo bypass, multi-token logprob parity, LRU/pressure behavior, and KV-only cold fallback. Cold requests report cached_tokens=0; warm requests restore the longest valid joint boundary.
• TP1 and TP2 scheduler, chunked-prefill, HF golden, prefix-restore, and HTTP serving tests passed. Prefix traces completed 64 zero-delta logprob comparisons on each TP size; TP2 HF mean/p99 deltas were 0.0251/0.0983 (short) and 0.0215/0.0684 (long).
• Release Clippy passed with -D warnings, and release openinfer and bench_serving builds passed with --features qwen35.

Performance

Measured on 2026-08-06, using local Qwen3.5-4B weights and idle RTX 4090s. TP1 used GPU 1 with CUDA Graphs enabled; TP2 used GPUs 1/2 with --cuda-graph=false. The prefix A/B matrix used a 1,024 MiB cache, 2 warmups, and 10 measured iterations.

For single-token generation, the table shows TTFT p50 in milliseconds (cache off -> cache on):

Prompt tokens	Cached tokens	TP1 TTFT p50	TP1 reduction	TP2 TTFT p50	TP2 reduction
320	256	29.22 -> 15.56	46.7%	46.18 -> 18.75	59.4%
576	512	46.85 -> 16.92	63.9%	72.97 -> 18.82	74.2%
1,088	1,024	88.75 -> 15.93	82.1%	126.14 -> 19.36	84.7%
2,112	2,048	161.00 -> 16.40	89.8%	230.07 -> 19.82	91.4%
4,160	4,096	308.64 -> 17.71	94.3%	439.37 -> 21.68	95.1%

For 128 generated tokens, cache reuse primarily reduces prefill/TTFT; decode remains the dominant part of E2E latency:

Prompt tokens	TP1 TTFT p50 off -> on	TP1 E2E p50 off -> on	TP2 TTFT p50 off -> on	TP2 E2E p50 off -> on
1,088	94.58 -> 24.13 ms	1,542.02 -> 1,452.78 ms	127.05 -> 20.38 ms	1,445.78 -> 1,324.97 ms
2,112	166.25 -> 24.23 ms	1,700.09 -> 1,535.84 ms	231.56 -> 21.89 ms	1,640.17 -> 1,421.95 ms
4,160	313.51 -> 24.30 ms	2,006.08 -> 1,696.04 ms	442.01 -> 24.88 ms	2,036.57 -> 1,606.08 ms

TP1 serving-load checks also completed:

• At a 2,112-token prompt and 128 output tokens, concurrency 1/4/8 produced TTFT p50 of 24.67/94.41/152.25 ms, request throughput of 83.11/75.87/69.04 tok/s, and a 2,048-token cache hit for every request.
• Under mixed load, baseline ITL p50/p99 was 12.03/12.25 ms; mixed-load ITL was 12.03/36.12 ms. After initial insertion, 4,096-token warm-prefix injections restored 3,840 tokens with 41.42–51.00 ms prefill and no warnings.
• TP1 HTTP warm TTFT p50 for prompt lengths 320/576/1088/2112/4160 improved from 31.84/48.71/93.34/146.05/282.10 ms to 18.51/21.42/21.34/29.16/25.51 ms.

Follow-ups

• May need Qwen35PrefixCache export more interfaces for DFlash feat(qwen35): add opt-in DFlash speculative decoding #626. Migrage prefix cache when DFlash is implemented.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 40d56ffdff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reclaim cached KV before admission stalls

When prefix caching is enabled and a near-capacity prompt publishes a snapshot, these guards can retain most of the KV pool after the request finishes. scheduler_loop checks available_blocks() before calling begin_request, so a repeated request that could reuse those blocks is deferred as though it needed a cold allocation; because the only eviction path runs from reserve_prefix after a request has already been admitted and prefilled, an idle engine can defer that request forever. Admission must account for reusable blocks or evict unpinned prefix entries under KV pressure.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve the requested benchmark seed

When a Qwen3.5 benchmark uses non-greedy sampling with --seed other than 42, this new call routes through launch_with_options_and_policy, whose EngineLoadOptions still hardcodes seed: 42; the previous path passed command_seed(&cli). Consequently the advertised seed no longer controls engine sampling for Qwen3.5, breaking reproducibility and making different seed runs use the same sampling stream. Pass the command seed through the launch options or retain the explicit EngineLoadOptions path.

Useful? React with 👍 / 👎.