feat(pflash): prefill compress up to 128k -> 2-12× prefill (content-dependent), decode at parity

[dusterbloom]

TL;DR

Adaptive composition stack: pflash compression + dflash spec-decode + ee7 drafter + anchor-transitive cascade + gated rare-token bridge + adaptive anchor_radius (64K NIAH cliff fix) + per-request fa_window_override + C2 mechanism gate + type-gate regime router (default-OFF).

Prefill speedup is content-dependent:

• ~12× prefill-TPS on sparse/NIAH cells (Round5),
• ~2.2× wall on dense agentic 64–128K (full-sweep 2026-05-29),
• 3.00× prefill-TPS / 2.91× wall on real dense >32K agentic turns with the router ON (verified on this branch's tip, 2026-05-31 — see §Verified A/B).

Decode at parity (composition); +35% with the router on (shorter compressed KV).

Adaptive levers

1. Adaptive anchor_radius (4a838dd): <32K=2 / 32–64K=4 / ≥64K=8 based on n_chunks. Eliminates the 64K NIAH cliff. Override via PFLASH_COMPRESS_ANCHOR_RADIUS.

2. Per-request fa_window_override (2726608, Design 1): when pflash compresses, http_server sets fa_window_override = effective_prompt + 256. Never caps visibility.

3. C2 mechanism gate (2726608 + refactor 548be0a): if eff_fa_window ≤ 2 × cfg_.fa_window (= 4096 default), spec-decode runs. Else AR fallback. Extracted to server/src/qwen35/c2_gate.h as a pure tested predicate + 5 unit tests. Zero behavior change.

4. Effective-size admission gate + keep-ratio guard (8c1f37d): the context-window admission check gates on the post-compression effective size, not the raw prompt — a prompt larger than max_ctx that compresses to fit is admitted (previously a hard 400) without forcing an oversized KV reservation. Default --prefill-keep-ratio 0.05→0.10 (anchors yield ~40% effective; keep=0.25 over-forces the cascade to ~100% = near-no-op). Verified at >128K: raw 166,911 → eff 71,110 (42.6%), admitted at max_ctx=139264.

5. Type-gate regime router (fbc2d41 / b31544f / 8fc961b, default-OFF behind PFLASH_ROUTER_ENABLE): correct-by-construction admission-time gate. server/src/common/regime_router.h is pure stdlib (96 LOC, 59 standalone -Werror tests). Logic:

   • agentic request (structured tools array / tool_use / tool_calls present) → keep floor 0.25, transitive cascade OFF. The cascade keys on rare-token co-occurrence, and the 151K drafter vocab makes ~every token "rare" → it over-forces dense agentic content to 70–100% keep (a near-no-op). Disabling it + flooring at 0.25 is what delivers the win below.
   • retrieval request → full keep, cascade ON (cascade is correct for QA/NIAH recall).
   • empty/degenerate guard: a compressed agentic turn that returns empty/degenerate skips the keep-bandit and one-shot recovers to full keep next turn (PFLASH_GUARD_MIN_TOKENS, default 8).
   • floor-clamp: the online keep-bandit (PR feat(pflash): adaptive keep_ratio bandit MVP #264) is clamped up to the 0.25 agentic floor so it can't undercut the router.

Verified A/B (router ON vs OFF) — 2026-05-31

Binary built from this branch tip (8fc961b), verified to contain both the admission-gate (dflash::common::check_admission symbol) and the router. 4 cold-prefill turns on real organic agentic sessions (claude-code transcripts, user turns at 34–37K est tokens, native tool blocks + tools array → agentic path fires). Fresh server per arm (true cold prefill). Run dir: bench/2026-05-31_definitive_ab/run_20260531_111334/.

Metric	ROUTER OFF	ROUTER ON	Speedup
keep %	72.3% (44.7–100)	24.9% (25)	—
prefill_TPS	1248	3747	3.00×
wall_s	64.9	22.3	2.91×
decode_TPS	20.6	27.8	+35%
tool_parse	4/4 OK	4/4 OK	held

All 4 ON turns routed type=agentic, keep=0.25, cascade=off. 0 floor-clamps, 0 guard fires. The admission-gate does not suppress the cascade over-forcing (OFF still 72% avg), so the router win is real on the shipped base. N=4 cold turns; numbers match the earlier-base measurement (2.85×/2.89×) within noise.

Reproducibility

./server/build/dflash_server /path/to/Qwen3.6-27B-Q4_K_M.gguf \
  --draft /path/to/dflash-draft-3.6-q4_k_m.gguf \
  --prefill-drafter /path/to/Qwen3-0.6B-Q8_0.gguf \
  --prefill-compression always --prefill-keep-ratio 0.10 \
  --prefill-threshold 32000 --max-ctx 139264 \
  --chat-template-file qwen3-coder-chat-template.jinja

Env: PFLASH_COMPRESS_ANCHOR_TRANSITIVE=1 PFLASH_DRAFTER_EARLY_EXIT_N=7 PFLASH_DRAFTER_SCORE_LAYERS=7 DFLASH27B_KV_K=tq3_0 DFLASH27B_KV_V=tq3_0. Router: add PFLASH_ROUTER_ENABLE=1.

Bench dashboard: bench/2026-05-28_adaptive_stack/Round5_4arm_matrix_174641/DASHBOARD.md. Public evidence: https://github.com/dusterbloom/pflash-evidence.

Notes

• The "+47% decode TPS" figure is C_pflash_only vs A_baseline, not D_composition vs A_baseline. Pflash compression delivers it. Composition decode TPS is at parity-or-slightly-below pflash-only because the C2 gate (correctly) disables spec-decode on compressed KV.
• The router (lever 5) is default-OFF; merging it changes no default behavior. It adds opt-in code + 401 LOC of tests.

[cubic-dev-ai]

12 issues found across 37 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[dusterbloom]

Addressed all cubic + howard0su review feedback. PR body updated with correct Q4_K_M target (was Q3_K_S — factual error) and per-context drafter speedup table (1K-128K + per-client agentic). Howard0su's two fixes applied (gguf_get_tensor_type, passbv docstring). All P1+P2 cubic items fixed across 6 commits.

[davide221]

why layer 7, and not some other layer? Env naming should be PFlash not DFlash to make it more clear that is the prefill drafter

[dusterbloom]

great question @davide221 I started removing from layer 14 then it worked. Then I push and it workd on layer 7. I stopped but actually am benching right now to see if it works on layer 5 or even 3. Let's see

As for the dFlash is a typo, agree to rename all for making sure it is clear this is about prefill!

[cubic-dev-ai]

1 issue found across 15 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 66 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

6 issues found

_{You’re at about 90% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="dflash/bench/run_ee_n_sweep_multihop.py">

<violation number="1" location="dflash/bench/run_ee_n_sweep_multihop.py:30">
P2: Context sweep parameter `ctx` is passed to `start_server` but ignored; `--max-ctx` is hardcoded to 139264, making all benchmark runs use the same allocation regardless of labeled context size</violation>
</file>

<file name="dflash/src/qwen3/anchor_scan.cpp">

<violation number="1" location="dflash/src/qwen3/anchor_scan.cpp:105">
P2: `prev_forced` is refreshed at the top of every transitive iteration, making `max_iters > 1` ineffective and preventing multi-iteration cascade expansion. For `it > 0`, `forced[c] && !prev_forced[c]` is always false because `prev_forced = forced` was just executed, so the rare-token worklist is empty and the loop breaks at `forced == prev_forced` before any iteration-1+ work can happen. The intended behavior requires carrying the pre-bottom-scan snapshot from the previous iteration so that newly-forced chunks from `scan_and_force` can be processed in the next rare-token pass.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Context sweep parameter ctx is passed to start_server but ignored; --max-ctx is hardcoded to 139264, making all benchmark runs use the same allocation regardless of labeled context size

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At dflash/bench/run_ee_n_sweep_multihop.py, line 30:

<comment>Context sweep parameter `ctx` is passed to `start_server` but ignored; `--max-ctx` is hardcoded to 139264, making all benchmark runs use the same allocation regardless of labeled context size</comment>

<file context>
@@ -0,0 +1,276 @@
+CONDITION_SPECS = {"ee7": (7, 7)}  # (EARLY_EXIT_N, SCORE_LAYERS)
+
+
+def start_server(condition, ctx, log_path, compression_mode="always", keep_ratio=0.05):
+    # server accepts off|auto|always; allow "none" as alias for "off"
+    srv_compression = "off" if compression_mode == "none" else compression_mode
</file context>

[cubic-dev-ai]

P2: prev_forced is refreshed at the top of every transitive iteration, making max_iters > 1 ineffective and preventing multi-iteration cascade expansion. For it > 0, forced[c] && !prev_forced[c] is always false because prev_forced = forced was just executed, so the rare-token worklist is empty and the loop breaks at forced == prev_forced before any iteration-1+ work can happen. The intended behavior requires carrying the pre-bottom-scan snapshot from the previous iteration so that newly-forced chunks from scan_and_force can be processed in the next rare-token pass.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At dflash/src/qwen3/anchor_scan.cpp, line 105:

<comment>`prev_forced` is refreshed at the top of every transitive iteration, making `max_iters > 1` ineffective and preventing multi-iteration cascade expansion. For `it > 0`, `forced[c] && !prev_forced[c]` is always false because `prev_forced = forced` was just executed, so the rare-token worklist is empty and the loop breaks at `forced == prev_forced` before any iteration-1+ work can happen. The intended behavior requires carrying the pre-bottom-scan snapshot from the previous iteration so that newly-forced chunks from `scan_and_force` can be processed in the next rare-token pass.</comment>

<file context>
@@ -0,0 +1,169 @@
+    // Cascade loop: expand pool with newly-forced tokens and re-scan.
+    std::vector<uint8_t> prev_forced;
+    for (int it = 0; it < max_iters; ++it) {
+        prev_forced = forced;
+
+        // Rare-token single-match: worklist-driven so cascades within a pass are
</file context>

[dusterbloom]

@cubic-dev-ai re: prev_forced is refreshed at the top of every transitive iteration:

Looked at this carefully — the implementation handles the case you're describing, just differently than the diff suggests. Two pieces:

1. anchor_scan.cpp:115-120 seeds the worklist from forced directly on it==0, not from forced && !prev_forced. So pass-1's anchor results enter the rare-token worklist regardless of the snapshot.

2. anchor_scan.cpp:133 makes the worklist self-cascading within a single iteration. When a rare-token hop forces a new chunk, that chunk is push_back'd into the same worklist and processed by the same for-loop (line 121 indexes a growing vector by wi). The header comment on lines 107-108 says it: "worklist-driven so cascades within a pass are caught (e.g. hop3 forces hop2 which forces hop1 in one outer iteration)."

So the rare-token closure transitively exhausts in a single outer iteration by design. The purpose of it > 0 is the 4-gram re-scan at lines 148-159, which is gated by a fresh prev_forced reassignment on line 158 right before scan_and_force runs — so iteration N+1's rare-token phase does see new chunks from the 4-gram re-scan via the it==0 all-forced seeding mechanism applied at the start.

T6 in test_anchor_transitive.cpp:298-344 is the executable proof: it builds a 20-link rare-token chain with max_iters=25 and max_forced_count=5, and asserts the cap engages. If the worklist were empty after it==0 as your comment describes, the chain couldn't propagate past chunk 0 and the cap would never fire. T6 passes — the in-iteration worklist cascades chunks 0→1→...→N until the cap stops it.

Not a bug — resolving as by design. Will add a clarifying inline comment if the area is touched again.

[cubic-dev-ai]

@cubic-dev-ai re: prev_forced is refreshed at the top of every transitive iteration:

Looked at this carefully — the implementation handles the case you're describing, just differently than the diff suggests. Two pieces:

1. anchor_scan.cpp:115-120 seeds the worklist from forced directly on it==0, not from forced && !prev_forced. So pass-1's anchor results enter the rare-token worklist regardless of the snapshot.
   ...

@dusterbloom You're right — this is by design, not a bug. T6 is the definitive proof.

The two mechanisms you describe cover the concern fully:

1. it==0 seeding (lines 115-120): seeding from all forced chunks on the first iteration ensures pass-1's 4-gram anchor results enter the rare-token worklist immediately, regardless of the prev_forced snapshot. The snapshot only gates the 4-gram re-scan in subsequent iterations, not the rare-token cascade.

2. Self-cascading worklist (line 133 worklist.push_back inside the loop): the rare-token phase exhausts its own transitive closure within a single outer iteration because newly-forced chunks get appended to the same worklist and processed by the same wi loop. This is the same pattern as BFS with a FIFO queue — no need for multiple outer iterations to propagate across hops.

The it > 0 iterations are there for the 4-gram re-scan (lines 148-159), which does need prev_forced to detect delta between the expanded pool from the rare-token cascade and the previous round's forced set. That checkpoint on line 158 (prev_forced = forced) is correct — it happens after the rare-token phase completes for that iteration, so the next iteration's 4-gram scan sees the full set.

If T6 passes with a 20-link chain and a cap of 5, the implementation is sound. No bug here.

_{Tip: get faster answers by chatting with cubic’s review copilot in the review UI}

[cubic-dev-ai]

You're iterating quickly on this pull request. To help protect your rate limits, cubic has paused automatic reviews on new pushes for now—when you're ready for another review, comment @cubic-dev-ai review.