Measured the cross-request prefill cost on a multi-stage local pipeline — numbers + LRUPromptCache orchestration

[thekiraproject]

We run a local compound pipeline — one 9B serving all roles (classify →
reason-with-tools → examine → synthesize), 8-bit, on an M4 Pro 24GB via mlx-lm.
Before optimizing anything, we instrumented where model time actually goes, and
found that 44–47% of it was prefill — re-encoding byte-identical stage
prefixes (system prompts, tool schemas, few-shots) on every call of every query.
We'd been blind to it since migrating off llama.cpp's automatic cross-request
prefix reuse; in-process mlx-lm didn't carry that behavior for us, and nobody
reopened the "we get this for free" assumption.

Per-query measurement (pre-cache):

Tier	Calls	Prompt tokens	Prefill	Decode	Prefill share
standard	4	8,880	20.0s	22.4s	44%
agentic	4	5,749	13.1s	14.4s	47%
research	12	21,989	49.7s	231.5s	17%

The fix used mlx-lm's own machinery: wired LRUPromptCache in, pinned each
stage's static prefix once per process lifetime, and chained within-query so
agent-loop turn N+1 extends turn N's cache. The biggest single lever was a
static-first prompt partition — a cache only reuses an unbroken prefix, so
one dynamic block early in the prompt ends the reusable region. Reordering
(system ∥ schemas ∥ few-shots first, dynamic content last) took our worst stage
from 26% → 96% cacheable.

End-to-end after activation: standard −34%, agentic −45%, research −24%.

Two things that might be useful upstream:

1. Surfacing per-call prefill stats by default made the whole problem
   visible — we were flying blind because our client reported prompt-eval as
   zero. Anything that nudges these into view would save others the blind months
   we had.
2. Parity-gating matters once an app restructures prompts to widen the
   cacheable prefix. "Exact reuse is exact, so output can't change" stops being
   a correctness argument the moment the prompt itself is reordered — the model
   sees different bytes. We applied the partition unconditionally in both configs
   and gated activation on a byte-equivalence output-parity battery before
   turning it on.

Full write-up with the activation discipline and a defensive disclosure of the
persistence design:
https://github.com/thekiraproject/kira-project-site/blob/main/posts/2026-07-06-prefill-tax.md

Happy to share measurement methodology if it's useful — curious whether others
running multi-stage setups on mlx-lm see similar prefill shares.