Design: from-scratch iteration-level scheduler for concurrent staged serving

[i386]

Status: design proposal · Date: 2026-08-24 · Owner: TBD

Goal

A from-scratch, iteration-level scheduler that serves many concurrent sessions through a single shared KV context, giving mesh vLLM/SGLang-class concurrency behavior (continuous batching, chunked-prefill interleave, KV-pressure preemption, zero-copy prefix reuse) without rewriting the KV cache and without porting external scheduler code.

This is the "step 3" workstream from the concurrency analysis (Buzz #general, 2026-08-23/24). It is a clean-sheet serving loop by explicit decision — the existing pieces were built for per-request-behind-a-lock serving and carry those assumptions.

Non-goals (hard boundaries)

• Not a KV-cache rewrite. The paged substrate already exists upstream in our pinned llama.cpp (3af988fab) and Skippy already drives it. See "Substrate we build on" below. Re-deriving it is the trap Replace Skippy KV cache with a typed, block-paged cache system #1263 fell into.
• Not a port of vLLM/SGLang. We reimplement the policy from their public design and cite it; the code is ours (avoids Apache-2.0 notice drag and their Python/CUDA-runtime assumptions).
• Not the disk-KV tier. That is a separate workstream/PR.

Substrate we build on (verified from source at pin 3af988fab)

• Unified KV / paging is already reachable: lane_count > 1 enables llama.cpp unified-KV — "every lane shares one n_ctx cell pool" (crates/skippy-runtime/src/config.rs:14-17), wired through to RawRuntimeConfig.lane_count (config.rs:129).
• Non-contiguous allocation: find_slot(ubatch, cont=false), slot_info::is_contiguous() (llama-kv-cache.h:79, .cpp:842).
• Index-based K/V gather: get_k_idx/set_input_k_idxs (llama-kv-cache.h:209,248).
• Zero-copy prefix sharing with refcount: cells carry a std::bitset of owning sequences (llama-kv-cells.h:30,6); same-stream seq_cp shares cells with no data copy; seq_rm frees a cell only when seq[i].none() (llama-kv-cells.h:247-266). Exception: ISWA/SWA/hybrid families do not yet share the sliding-window companion cache (if (other) return;, upstream TODO [TAG_KV_CACHE_SHARE_CELLS], llama-kv-cache.h:320-321).

Dependencies (must land under the scheduler; not part of this issue's code)

1. Remove the prefill serialization. restore_or_record_kv takes .lock() (token_generation.rs:377) and holds it across every .prefill() (:409,419,436,444); it self-reports llama_stage.runtime_lock_acquires=1 (:350). One request's prefill serialises all others.
2. Raise built-in parallelism. BUILTIN_PARALLEL = 1 (resolver/types.rs:13) and run with lane_count = target concurrency so unified KV is active.

A pristine scheduler over a locked single-lane runtime is still single-lane. These are prerequisites, tracked separately.

Architecture

New component in its own crate crates/skippy-scheduler/ (decision 2026-08-24, James — deliberately not inside skippy-server, to keep the serving loop independently testable and free of server-crate assumptions), owning a single serving loop per stage runtime:

flowchart LR
    R[Incoming requests] --> WQ[WaitingQueue]
    WQ -->|admit while KV budget allows| RS[RunningSet]
    RS -->|compose one ubatch per step| SB["StepBatch<br/>decode tokens + prefill chunk"]
    SB --> DEC["runtime.decode<br/>one mixed ubatch"]
    DEC -->|sampled tokens| POST{Per-sequence result}
    POST -->|continue| RS
    POST -->|EOS / limit| DONE["seq_rm<br/>frees cells; shared prefix survives via refcount"]
    RS -->|KV pressure| PRE["Preempt newest /<br/>lowest priority"]
    PRE -->|recompute on resume| WQ

Loading

• Request: session id, token ids, exact-prefix identity (reuse Skippy's existing identity — do not reinvent), sampling params, assigned seq_id (lane).
• WaitingQueue: admitted in order subject to KV budget.
• RunningSet: sequences with live KV in the unified context.
• StepBatch: the single ubatch built each iteration.

The per-step loop (core)

flowchart TD
    S([Step start]) --> A["1. Admit: waiting to running while<br/>KV budget allows; assign seq_id"]
    A --> C["2. Compose one ubatch: decode for running +<br/>bounded prefill chunk for admitted, under token budget"]
    C --> E["3. Execute: single runtime.decode of the mixed ubatch"]
    E --> P[4. Sample per sequence]
    P --> D{EOS or limit?}
    D -->|yes| RM["seq_rm - free cells"]
    D -->|no| K{KV pressure?}
    RM --> K
    K -->|yes| PRE["5. Preempt newest / lowest priority<br/>recompute on resume"]
    K -->|no| N([Next step])
    PRE --> N
    N --> S

Loading

Each iteration:

1. Admit: while KV budget (free cells / n_seq_max) allows, move requests waiting→running and assign a seq_id. Reuse the KV-token-budget accounting that already exists in frontend/admission.rs as reference logic.
2. Compose one ubatch: decode tokens for every running sequence + a bounded prefill chunk for newly admitted sequences, under a fixed per-step token budget (chunked prefill). Harvest logic from frontend/prefill.rs::PrefillChunkSchedule and frontend/decode_batcher.rs, rewritten for multi-request.
3. Execute: one runtime.decode(ubatch) for the whole step (llama.cpp decodes the mixed ubatch across sequences).
4. Post: sample per sequence; on EOS/limit, seq_rm the sequence (frees its cells; shared prefix cells survive via refcount).
5. Preempt under KV pressure: evict the newest/lowest-priority running sequence (recompute-on-resume first; swap-to-disk deferred to the disk tier).

Policies (each pluggable, simple default)

• Admission: KV-budget gated, FCFS default.
• Ordering/fairness: FCFS default; priority hook.
• Prefix sharing: on admission, look up exact-prefix identity; if a live sequence holds the prefix, seq_cp to share cells zero-copy (dense families). SWA companion falls back to recompute until TAG_KV_CACHE_SHARE_CELLS.
• Preemption: recompute-newest default.

Telemetry (day one)

Per-step: running count, waiting count, admitted, preempted, tokens (prefill vs decode), KV cells used/free, prefix-share hits/misses. Replace the single runtime_lock_acquires=1 marker with real concurrency metrics.

Test plan

• Unit: admission under budget, batch composition (prefill/decode mix within token budget), preemption selection, prefix-share hit path.
• Concurrency (the gate): N concurrent sessions; measure throughput/latency vs. the single-lane baseline; assert no request's prefill blocks others.
• Correctness: per-family exact-output invariants unchanged — same identity, same bytes, no regression across supported families (this is the line we do not cross for a throughput win).
• Loom/stress on the serving loop's admission/preemption transitions.

Per-family tests + splits (decision 2026-08-24, James)

We carry correctness tests and pinned split topologies for each of the 5 KV-class representatives. Current state (verified in-tree):

• crates/skippy-correctness/tests/parity_models.rs already declares all 5 picks — llama, gemma2, deepseek2, falcon_h1, qwen3next — all status: certified in docs/skippy/llama-parity-candidates.json. Each gets three tests: manifest_row_is_complete, activation_handoff_matches_full_model (splits the model into stages and asserts multi-stage handoff == full model), and cache_state_restore_matches_recompute (asserts cache restore == recompute — this already validates the recompute-preemption decision macos menu app #1 per family).

What the concurrency work must add on top:

1. Concurrent-parity per family. The parity harness above is single-sequence. The new scheduler needs the same exact-output parity to hold with N sessions in flight (mixed prefill/decode, seq_cp prefix-share). Add one concurrent-parity case per family = the correctness pre-gate.
2. Pinned splits per family. All 5 currently fall back to the harness default 1/3–2/3 layer boundary (splits/split_layer are empty in the manifest). Carry a pinned representative multi-stage split per family so the per-stage admission + min-across-stages ceiling path is exercised deterministically. qwen3next and falcon_h1 are recurrent: all — their splits must respect recurrent-state stage boundaries (KV is a recurrent state, not a plain cell pool), so pin those explicitly rather than trusting the default boundary.
3. Reuse the existing recompute-restore test as the per-family evidence for decision macos menu app #1 (recompute-newest preemption).

Cutover gate & benchmark methodology

The new engine replaces the old completely and only when it beats the old engine on concurrency and performance across every major KV family class, with zero correctness regression. Old engine = current single-lane locked path; new engine = this scheduler over kv_unified.

What to measure

Primary cutover metric: goodput @ SLO — requests/sec that meet a fixed latency target (TTFT + TPOT below thresholds). It's the honest single number because raw throughput can be inflated by tanking latency.

Supporting metrics:

• Throughput — output tokens/sec and requests/sec at fixed concurrency N.
• TTFT (time to first token) p50/p99 — prefill responsiveness under queueing.
• TPOT / inter-token latency p50/p99 — decode smoothness under batching.
• End-to-end request latency p50/p99.
• Concurrency scaling curve — sweep N ∈ {1,2,4,8,16,32,64}; plot throughput and p99 latency vs N. Old engine flatlines early (single-lane); new engine should scale. The whole curve is the evidence, not one point.
• KV-caching win (the "better performance" claim): prefix-cache hit rate and the TTFT/throughput it buys on shared-prefix workloads (zero-copy seq_cp), plus KV bytes per session (the paging/fragmentation win).

Workloads (benchmarks are meaningless without these)

• Length mixes: short-in/short-out (chat), long-in/short-out (RAG/summarize), short-in/long-out (generation), long-in/long-out.
• Prefix shape: shared-prefix (system prompt / few-shot / multi-turn) vs. unique-prefix — isolates the prefix-cache win.
• Arrival: open-loop Poisson at target QPS (industry standard for serving) and closed-loop fixed-N (clean scaling curve). Run both.

Family matrix — one representative per KV structural class

Not all 95 families — the KV structure is what varies for this work. All picks are already in the bench roster (evals/skippy-cache-production-bench.py) with GGUFs downloaded:

KV class	Model (roster id)	Why this rep
Dense full-attention (GQA)	Llama-3.2-1B (llama)	bread-and-butter; full zero-copy seq_cp sharing
MLA (latent KV)	DeepSeek-Coder-V2-Lite (deepseek2)	real MLA, small Q4; deepseek3 is layer-package-only/huge — stretch only
ISWA / sliding-window	Gemma-2-2B (gemma2)	canonical alternating SWA/full — hits the TAG_KV_CACHE_SHARE_CELLS sharing boundary
Recurrent / SSM-hybrid	Falcon-H1-1.5B (falcon_h1)	Mamba/attention hybrid — exercises llama-memory-recurrent + memory-hybrid
Linear-attention hybrid	Qwen3-Next (qwen3next) or MiniMax M2.7 (minimax_m27)	gated-DeltaNet / lightning attention — linear-attn state, distinct paging/sharing behavior

5-model matrix = the scientific minimum, one per KV structural class. GLM-4, OLMo, Gemma-3/4 are skins on one of these classes — add as confirmation runs, not gate rows.

MoE is orthogonal to KV structure — it changes FFN/expert compute, not KV cache layout — so it is not a separate benchmark axis. Coverage comes free via reps that happen to be MoE (DeepSeek, MiniMax); no dedicated MoE row.

Gate (all must hold, per family class)

1. Concurrency: new ≥ old on goodput/throughput at target N, and the new scaling curve dominates old everywhere — including no regression at N=1 (scheduler overhead negligible for single-user).
2. Performance / KV: shared-prefix TTFT/throughput improvement demonstrated; KV bytes/session no worse than old.
3. Correctness (pre-gate, non-negotiable): exact-output parity per family — same identity, same bytes. A faster wrong engine never cuts over; correctness is checked before speed.

Proposed default margins (yours to set): ≥2× throughput at N=32 with equal-or-better p99, and no N=1 regression. Cutover is per-family-class capable — a class that passes can cut over even if another is still being tuned, gated behind a flag.

Harness (decision: in-house is the gate)

Gate harness = extended skippy bench. evals/skippy-cache-production-bench.py today is single-request (--runtime-lane-count 1 --llama-parallel 1). We extend it with a concurrency sweep: fire N parallel OpenAI requests, record per-request timings, derive TTFT / TPOT / throughput / goodput, and sweep N ∈ {1,2,4,8,16,32,64}. It stays in-tree, family-aware, reproducible, no external dependency to vet, and it already owns the correctness + server-side KV-footprint side (via the skippy-cache-family-bench skill and Skippy's own telemetry — KV bytes/session, prefix-share hit/miss counters).

Optional external cross-check = eugr/llama-benchy. OpenAI-compatible, so it can drop a real vLLM / SGLang / llama-server reference number next to ours for context. Not the gate; not on the critical path. If used, smoke-test it against a Skippy endpoint first to confirm it emits per-request timings usable for p99/goodput.

Two methodology items we own regardless of tool: goodput @ SLO (derived from per-request latencies) and open-loop Poisson arrival (the concurrency sweep is closed-loop fixed-N, which covers the scaling curve; add an open-loop driver later only if arrival realism is needed).

Lane sizing & admission (decision 2026-08-24)

Lane count is derived from hardware, not configured — and it is not a static number. The coordinator planner already fits topologies resource-aware (skippy-coordinator/src/topology.rs): a nested search over context length (largest→smallest), node count (fewest→most), then lanes (most→fewest), keeping the subset whose VRAM fits weights + KV + 15% compute reserve. The winner's parallel_lanes → plan.slots → RuntimeConfig.lane_count (runtime/local.rs:834), and lane_count > 1 flips unified KV on.

Key physics: in unified KV the cells are one shared n_ctx allocation indexed by sequence id — the fit calc deliberately does not multiply KV by lanes (topology.rs:525-542; _parallel_lanes is unused, with a comment stating exactly this). So a lane costs almost no memory; it is permission to have another in-flight sequence over the same pool. The real limiter is cells per active session = n_ctx / concurrently-active sessions. Lanes and per-session context length trade against each other inside one fixed pool. Today's planner obscures this by hardcoding MAX_AUTO_PARALLEL_LANES = 4 and sizing context independently.

New design — two levels, not one number:

1. Derived ceiling (n_seq_max), set at load. Size the shared pool from VRAM (planner already computes this), then ceiling = pool_cells / floor_ctx_per_session. Because lanes are near-free, set it generously. Remove the hardcoded MAX_AUTO_PARALLEL_LANES = 4 cap; derive it.
2. Dynamic KV-gated admission at runtime. Admit a session only if the shared pool has cells for its actual need now. This is what delivers "good out of the box with as many lanes as possible": a high ceiling plus admission that packs the real pool with whatever mix of long/short prompts arrives — no static knob can get that right.

Multi-stage mesh: each stage has its own KV context sized by its node's VRAM, so the pipeline's effective ceiling is min across stages (a weaker node bounds concurrency). Per-stage scheduler (decision #4) fits cleanly: each stage admits against its own pool, and the min-stage becomes the natural backpressure point.

Open questions — resolved (2026-08-24, James)

1. Preemption → recompute for the first cut; swap arrives with the disk tier.
2. SWA/hybrid prefix sharing → carry a small patch, with an in-patch comment stating it tracks upstream TAG_KV_CACHE_SHARE_CELLS so the upstream dependency is explicit and easy to drop when it merges.
3. Per-step token budget → no static default. The resource is a shared pool, not partitioned; the step token budget is a global draw on it (default n_batch = 1024 in unified mode) and concurrency is admission against the same pool. Resolved by the lane sizing & admission design above — one pool, one budget, two views.
4. Scheduler → per-stage. Each stage runs its own loop over its own kv_unified context; a cross-stage coordinator is deferred until benchmarks show pipeline bubbles dominate.

Directive: no backwards compatibility (2026-08-24, James)

Cut the old code out; simplify aggressively; leave no back-compat shims. Concretely for this work: unified KV is always on (delete the lane_count == 1 single-lane branch and the dual default_n_batch_for_lane_count path, config.rs:206); lane_count stops being a user/config knob and becomes a derived ceiling; MAX_AUTO_PARALLEL_LANES = 4 is removed. Simpler and more capable.

Implementation tasks

• Prerequisite: remove the prefill-serialization lock (token_generation.rs:377 held across .prefill()); raise BUILTIN_PARALLEL / run with lane_count > 1.
• Gate harness: extend evals/skippy-cache-production-bench.py with a concurrency sweep (N parallel OpenAI requests, per-request timings → TTFT/TPOT/throughput/goodput, sweep N ∈ {1..64}), old-vs-new engine comparison. In-tree, family-aware. (approved 2026-08-24)
• Baseline: run the concurrency sweep on the current engine across the 5-model matrix to capture the old-engine curve before any scheduler work.
• Lane sizing: replace hardcoded MAX_AUTO_PARALLEL_LANES = 4 with a pool-derived ceiling (n_seq_max = pool_cells / floor_ctx_per_session); multi-stage ceiling = min across stages. (design 2026-08-24)
• Simplification: delete the single-lane branch and dual default_n_batch_for_lane_count path (config.rs:206); unified KV always on; lane_count becomes a derived ceiling, not a config knob. (no-back-compat directive 2026-08-24)
• Scheduler: new crate crates/skippy-scheduler/ (not inside skippy-server) — per-step loop (admit → compose ubatch → decode → sample → seq_rm/preempt), driving kv_unified. (own-crate decision 2026-08-24)
• Per-family concurrent-parity: add one N-session exact-output parity case per pick (llama, gemma2, deepseek2, falcon_h1, qwen3next) alongside the existing single-sequence parity tests. (2026-08-24)
• Pinned splits: populate splits/split_layer for the 5 picks in docs/skippy/llama-parity-candidates.json (respect recurrent-state boundaries for qwen3next/falcon_h1) so multi-stage tests use deterministic boundaries, not the 1/3–2/3 default. (2026-08-24)
• Prefix sharing: wire seq_cp zero-copy reuse on Skippy exact-prefix identity; recompute fallback for SWA companion pending TAG_KV_CACHE_SHARE_CELLS.
• Telemetry: per-step concurrency + KV metrics; replace runtime_lock_acquires=1 marker.
• Cutover: per-family-class flag flip once the gate passes for that class.

Provenance

Design references: vLLM (continuous batching, PagedAttention, admission/preemption) and SGLang (RadixAttention prefix sharing, chunked prefill). Both Apache-2.0; we reimplement concepts, cite them, and copy no code.

---

Supersedes the "scheduler" portion of the reshaped #1263 discussion; the KV-substrate and disk-tier workstreams are tracked separately.

[michaelneale]

Review findings — MeshBuilder + Galadriel (2026-08-24)

Two independent reviews. Verified against origin/main 8b18122 and the llama.cpp deps tree (76314ca9 / 1b009977c). Galadriel independently re-checked findings 1–3 at source and confirmed each.

Verdict from both: the direction is right. Continuous batching, KV-gated admission, recompute preemption, per-stage loop, own crate, in-tree bench, correctness-as-pre-gate — no objection to any of it. The issues below are corrections to specific claims, plus two missing workstreams.

1. BLOCKING CORRECTION — "lanes are near-free" is dense-only; false for 2 of the 5 gate families

The Lane sizing & admission section's load-bearing claim holds for attention KV but not for recurrent/hybrid memory. Recurrent state is per-sequence and scales linearly with n_seq_max, per layer:

• llama-model.cpp:2158 passes std::max(1u, cparams.n_seq_max) as mem_size
• llama-memory-recurrent.cpp:33 size = mem_size
• :100-104 each layer allocates n_rows = mem_size * (1 + n_rs_seq) rows of n_embd_r / n_embd_s
• hybrids pass n_seq_max into both halves (llama-memory-hybrid.cpp:43,60; llama-model.cpp:2215,2234)

So falcon_h1 and qwen3next allocate ceiling-proportional state at load. A "set it generously" derived ceiling OOMs them or silently shrinks context. The candidate_bytes_per_layer comment at topology.rs:525-542 ("do not multiply by lanes") is correct for attention and wrong for these.

Required change: the derived ceiling must be family-aware — shared-pool division for attention KV, n_seq_max × per-seq recurrent bytes charged against VRAM for recurrent/hybrid. As written, the sizing rule and the family matrix contradict each other.

2. Hard ceiling of 256, and the prefix cache is a co-tenant

LLAMA_MAX_SEQ = 256 (llama-cparams.h:8), and under unified KV llama.cpp uses exactly that as the sequence bound (llama-context.cpp:1428,1688). The formula needs:

ceiling = min(pool_cells / floor_ctx_per_session, 256 − reserved)

reserved is real: the resident prefix cache pins prefixes on their own seq ids in the same context, reserved_seq_count = lane_count * 2 (skippy-cache/src/config.rs:30), and by default it may consume half the cell pool (derive_max_resident_tokens). Admission that counts only the lanes will reproduce the exact failure that cap was added to fix: HTTP 502 decode: failed to find a memory slot (family_policy.rs:299-330).

Latent bug to fix under this work (one-liner): ResidentPrefixCache::next_sequence_id bails at seq_id >= 1024 (skippy-cache/src/resident/prefix.rs:323) while llama rejects >= LLAMA_MAX_SEQ (llama-kv-cells.h:345, llama-memory-recurrent.cpp:516). Unreachable today (family policy caps max_entries at 16), reachable as soon as lanes/entries are raised.

3. Admission must be per-memory-component, not one pool (Galadriel)

3 of the 5 gate families have two memory components with different accounting: gemma2 (ISWA: full-attn pool + SWA window pool), falcon_h1 and qwen3next (attention pool + per-seq recurrent slots). Admission must be "admit iff every component has room", and the multi-stage ceiling is a min across stages × components, not stages alone. Counting only the shared attention pool means hybrids fail allocation while admission reports headroom.

4. MISSING WORKSTREAM — the core loop step is not expressible in today's ABI

"Compose one ubatch: decode tokens + a bounded prefill chunk, single runtime.decode" cannot be built over the current surface. skippy_decode_batch_sampled is strictly one token per session: batch.n_tokens = request_count, batch.pos[i] = session->n_past, n_seq_id[i] = 1 (patch 0008). The frame variant is the same shape. Prefill is per-session (skippy_prefill_chunk).

Mixed prefill/decode co-batching therefore needs a new ABI entry point and a patch-queue change, not just a Rust crate. Two constraints that follow:

• The plain batch path refuses non-first slices ("batched token decode requires the first runtime slice or a full model"), so downstream stages must go via skippy_decode_step_frame_batch_sampled…
• …which additionally requires uniform GLM-DSA sideband shape across the batch — a real restriction on which requests may share a step.

5. Prefix sharing: extend the resident cache, do not add a parallel seq_cp path

There is no seq_cp in the public ABI. llama_memory_seq_cp appears only inside patch 0006, driving skippy_session_save_prefix / skippy_session_restore_prefix — i.e. the resident prefix cache is already the zero-copy prefix-sharing mechanism, on reserved seq ids. Building a second cell-level sharing path gives two prefix systems competing for one cell pool.

Galadriel's addition: prefix sharing on recurrent families is neither zero-copy nor positional. Recurrent state is reusable only at an exact whole-state snapshot boundary — you cannot share a prefix of a Mamba/DeltaNet state at an arbitrary token position the way you share attention cells. The existing resident cache already handles this (patch 0020 preserves recurrent-state metadata on restore). Consequence for the benchmark: for recurrent families a "prefix-share hit" is a state restore, so the prefix-cache-win row should expect a much smaller or zero win — a flat gate margin would fail those rows for physics reasons, not defects.

6. The simplification cut can itself cause the N=1 regression the gate forbids

Deleting the dual default_n_batch_for_lane_count makes unified the only path, and unified's default is 1024 vs single-lane 2048 (skippy-runtime/src/config.rs:10,18) — a 2× smaller prefill batch for the single-user case. Revisit the constant in the same change, or expect gate row 1 to fail for a reason unrelated to the scheduler.

Smaller

• "Remove the prefill serialization" understates the change. RuntimeState is Arc<Mutex<_>> across ~20 sites in skippy-server and llama_context is not thread-safe — the lock cannot simply be removed; ownership has to move. The scheduler becomes sole owner and every caller submits over a channel. Stated as "remove the lock", someone will try to make the mutex finer-grained instead.
• Add an overload gate row: past the ceiling the system must queue/backpressure, never return llama_decode failed. That is the historically observed break and it is invisible to a goodput-only gate.
• Baseline task should record peak VRAM per N alongside latency — otherwise finding 1 surfaces as a mystery OOM mid-sweep.

Suggested task additions

• Family-aware ceiling derivation (attention: pool division; recurrent/hybrid: n_seq_max × per-seq bytes charged to VRAM) — supersedes the single formula in Lane sizing & admission.
• Clamp ceiling to LLAMA_MAX_SEQ (256) − reserved_seq_count; account for the resident prefix cache's half-pool budget in admission.
• Fix resident/prefix.rs:323 seq-id bound 1024 → LLAMA_MAX_SEQ.
• Per-memory-component admission + min across stages and components.
• Patch-queue workstream: ABI entry point for a mixed prefill/decode ubatch (multi-token-per-sequence batch with per-token positions); document the non-first-slice and GLM-DSA-sideband constraints.
• Prefix sharing = extend save_prefix/restore_prefix, not a new seq_cp path; recurrent families restore whole-state only.
• Revisit SKIPPY_UNIFIED_KV_DEFAULT_N_BATCH when the single-lane branch is deleted (N=1 no-regression gate).
• Reframe the lock prerequisite as ownership transfer (scheduler-owned runtime + submission channel).
• Gate row: overload backpressure. Baseline: peak VRAM per N.