feat(skippy): execute mixed prefill and decode iterations safely

[i386]

Summary

• reserve live decode rows first, then spend remaining iteration capacity on bounded prefill/recompute work
• return sampled tokens with explicit request indexes so sparse sampled rows remain correct when unsampled prefill rows are interleaved
• extend the native iteration ABI to version 41 and carry the indexed result through FFI, runtime, server, and scheduler completion
• add scheduler simulation/lab coverage plus dense CI and opt-in P0/P1 family parity coverage against serial execution

Correctness and safety

The native batch now exposes request-major activation frames separately from compact sampled outputs. Every sampled token carries its originating request index. Rust validates sample counts, bounds, and duplicate indexes before scheduler state is mutated. Missing or duplicate predictions fail only the affected sampled sequence.

The mixed planner preserves the latency invariant: every live decode row is scheduled before prompt work, subject to the global token budget. Remaining tokens are used for prefill/recompute while respecting the configured chunk and prefill-sequence limits.

flowchart LR
    D[Live decode rows] --> R[Reserve latency-sensitive rows first]
    P[Prefill and recompute rows] --> F[Fill remaining token budget]
    R --> B[One mixed native iteration]
    F --> B
    B --> A[Request-major activation frames]
    B --> S[Compact sampled outputs]
    S --> M[Explicit request index plus token]
    M --> C[Complete scheduler work by index]

Loading

Scheduler lab

The deterministic staggered workload compares bounded phase-homogeneous scheduling with mixed execution:

Metric	Bounded	Mixed	Change
Makespan	19.656 ms	19.256 ms	-2.0%
Throughput	254.38 req/s	259.66 req/s	+2.1%
p95 TTFT	7.264 ms	7.084 ms	-2.5%
p95 max ITL	1.796 ms	1.726 ms	-3.9%
Mean batch size	1.28	1.44	+12.5%
Occupancy	2.85%	3.21%	+12.6%

Mixed execution produced eight mixed iterations on this workload; the simulation contract locks improved occupancy, inter-token latency, and makespan.

Studio54 exact-head A/B

Two order-reversed passes per arm used fixed output lengths on the Qwen3.5 0.8B Metal runtime. The live result validates the mechanism but does not support a blanket latency-win claim:

Trace	Throughput	Makespan	p50 TTFT	p95 TTFT	p50 stream gap	p95 stream gap	Occupancy
Representative	+1.1%	-1.1%	+23.7%	+12.5%	-36.0%	+14.1%	+19.2%
Mechanism stress	-3.0%	+3.1%	+1.2%	+5.2%	-25.2%	+39.1%	+38.0%

The candidate reduced native scheduler iterations by 16.1% and 27.5%, respectively, and produced 21/66 mixed iterations per pass versus zero on the base. Longer mixed steps worsened TTFT and tail streaming gaps, however, and outweighed iteration savings under stress. Duration-aware admission and a latency budget for joined prefill work remain necessary before claiming a broad performance improvement.

xychart-beta
    title "Studio54 mechanism change (base = 100)"
    x-axis [Rep-iterations, Rep-occupancy, Stress-iterations, Stress-occupancy]
    y-axis "Percent of base" 0 --> 145
    bar [100, 100, 100, 100]
    bar [83.9, 119.2, 72.5, 138.0]

Loading

Versioned method, per-pass numbers, and the normalized latency/occupancy charts are in docs/skippy/MIXED_PREFILL_DECODE_STUDIO54.md. The compact raw metrics are in docs/skippy/mixed-prefill-decode-studio54-summary.json.

Validation

• cargo fmt --all
• strict Clippy across skippy-ffi, skippy-runtime, skippy-scheduler, skippy-server, and skippy-correctness with all targets and dynamic runtime/lab features
• full scheduler unit, simulation, and lab runs
• full correctness contract: 31 binary tests and 81 manifest/non-native parity tests passed; 240 artifact-backed family tests were enumerated as ignored
• full server dynamic suite: 518 passed; the remaining nine require a loaded native dynamic library
• patch queue materialized and applied cleanly; scripts.tests.test_prepare_llama passed
• model-gated runtime test compares one mixed decode/long-prefill/short-prefill iteration with serial execution and verifies sparse sample mapping plus native session positions; CI runs it against the pinned Qwen correctness model

The local macOS static native build could not be used as evidence because the host Apple C++ toolchain cannot resolve standard <mutex>/<array> headers and lacks OpenMP. Linux native CI is the authoritative static/native gate.

Stack

Stacked on #1453 (scam/scheduler-adaptive-prefill). Review this PR as the final mixed-execution layer after that base.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 59d7b235-37c0-46e3-a95a-50d7f2038d4d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[i386]

Independent review + benchmark (Jian Yang)

Verdict: approve once the branch head includes the sparse-logits fix (15005c80) — that commit fixes a real bug present in the pinned review head.

Verified independently (source, at pinned head 0b7265e5 + main c383b411)

• Production overrides max_consecutive_prefill_iterations: 1 — Jared's usize::MAX starvation claim was stale for prod; your correction was right.
• Mutex serializes multi-lane batch ABI calls, not individual requests.
• plan_mixed_iteration reserves decode rows first, fills remaining budget with prefill chunks, sample_last only on final rows. Prod wiring sets mixed_prefill_decode: true (not dead config).
• Duration guard: decode-only EWMA calibration per row count; mixed step must fit clamp(0.2×decode_us, 8ms, 20ms) extra or one-way fallback. Reasonable.
• Your "serving-equivalent binary" claim checks out: 0b7265e5 vs my benched 6ad68fae differ only in evals/skippy-adaptive-prefill-ab.py.

Independent benchmark (M1 Ultra, mirrors your protocol; llama.cpp v0.3.0 c1d0e7a00 fresh build)

Median aggregate tg tok/s, no-cache, pp=512, 3 runs:

tg	c	main c383b41	this stack	llama v0.3.0
64	1	97.8	97.8	110.6
64	4	119.0	124.7	410.8
256	4	148.7	155.6	340.9

Both skippy arms are flat across concurrency; llama scales ~3.7×. Main vs this PR: within ±5% everywhere — no regression, no headline win, matching your own A/B and the PR's narrow claim.

The decisive read — answered with telemetry

stage.scheduler_iteration decode-token histogram over 4,138 iterations per arm: rows=1 in 33%, rows=2 in 32%, rows=4 in 30% of iterations. Coalescing works. Decode-only step time medians: 1 row = 9.7 ms, 2 = 14.8 ms, 4 = 24.6 ms — native step cost scales ~linearly with rows on Metal. The competitive gap is native batch-GEMM efficiency, not scheduler admission. Mixed iterations engaged in 1/4,136 iterations of my simultaneous-arrival sweep (engagement is workload-shape dependent — worth stating in the PR description).

Findings

• F1 (blocker on pinned head, fixed by 15005c80): ABI passed sparse sampled_output_index to skippy_sample_token_ith, but llama_get_logits_ith wants the original batch row index — wrong sampled token whenever a sample_last row isn't the batch tail. Merge only with this included; the 51b2b588 lane bump (3→6) that lets the parity test exercise it should land too.
• F2: mixed_prefill_viable one-way latch never re-probes after fallback. Safe but sticky; note it.
• F3: missing-sample semantics diverge — frontend unwrap_or(-1) → Finished, engine complete_iteration → Failed. Align.
• F4: your own representative trace shows p50 TTFT +23.7%. Short agent-turn cells (the Carrack 8-token TTFT-dominated cell that started this thread) won't benefit; the guard bounds per-step extra but every decode token rides prefill work.
• F5 (pre-existing, main): ignore_eos → HTTP 400 "parsed but not yet implemented" (request.rs UNSUPPORTED_FIELDS). Needs a follow-up issue.

Detailed write-up with artifacts: RESEARCH/PR1456_MIXED_SCHEDULER_REVIEW_2026_08_27.md in the agent workspace; raw JSON in .scratch/scheduler-review/results/.

[i386]

Superseded by #1478, which contains this branch's full content (verified by file-tree comparison: zero content diff in skippy-scheduler + prefill.rs, every touched file also appears in 1478's diff) plus the signal-hotpath stack on top. Closing to avoid duplicate review/merge of a stale head.