Add --moe-mode sweep-prefill: streamed expert supply for CPU mul_mat_id

[Zimo41650079726sub]

Motivation

When a MoE model is much larger than RAM and served from NVMe via mmap (CPU backend), prefill touches nearly every expert: on DeepSeek-V4-Flash 284B we measured 44–100% of the 256 experts per layer used within a single 512-token ubatch. The default demand-paging path reads them through page faults in routing order, which looks random to the kernel, so readahead never engages and prefill becomes fault-latency bound.

On systems with a GPU backend, expert upload overlaps with GPU compute. On CPU-only hosts (this work was done on Snapdragon X Elite under WSL2) compute and I/O share the same cores, so there is no natural overlap — the only leverage left is telling the kernel what to read next.

Mechanism

The CPU mul_mat_id kernel already iterates experts in ascending id order over contiguous mmap slices, so expert access during one node evaluation is a physical-order sweep. This PR exploits that:

• At load time, expert tensors verified to lie inside the file-backed mapping are marked via their (unused) leaf op_params — the compute graph is unchanged.
• During eval, thread 0 of the kernel issues madvise(WILLNEED) for the next 2 used expert slices ahead of the loop, and a lagged madvise(DONTNEED) 2 slices behind, keeping the page-cache footprint at a few slices.
• Decode ubatches below --moe-sweep-min-tokens (default 32) take the regular path, so decode behavior is untouched.
• GGML_SWEEP_STATS=1 logs per-node sweep statistics.

The math is untouched; only page-cache population timing changes.

Correctness

• DeepSeek-V4-Flash 284B IQ2_XXS: greedy output byte-identical to stock golden run; KL divergence vs stock logits max 0.000044, top-1 agreement 100%.
• Qwen3.5-35B-A3B IQ2_XXS (this repo's qwen35moe arch): greedy output byte-identical to stock, in both unconstrained and 8 GB-cgroup conditions (sweep engagement confirmed via GGML_SWEEP_STATS; loader registered 120 expert tensors).

Results

Snapdragon X Elite (12-core Oryon, WSL2 Ubuntu, NVMe ~7 GB/s seq), cold page cache:

Model	Condition	stock prefill	sweep prefill	speedup
DS4-Flash 284B IQ2_XXS (77.9 GB)	54 GB RAM, cold	4.82 tok/s	6.54 tok/s	1.36x
DS4-Flash 284B IQ2_XXS	8 GB cgroup	3.61 tok/s	5.20 tok/s	1.44x
Qwen3.5-35B-A3B IQ2_XXS (9.9 GB)	8 GB cgroup, cold, 5063-tok prompt, ABBA n=2	29.1 tok/s (29.75 / 28.51)	39.1 tok/s (37.72 / 40.39)	1.34x

When the model fits in RAM (warm cache, no cgroup) sweep-prefill is slightly slower (52.8 vs 56.1 tok/s on the same Qwen3.5 setup) because DONTNEED evicts cache that would have been reused — this is why the mode is opt-in rather than default.

Scope and limitations

• CPU backend only (ggml-cpu); opt-in via --moe-mode sweep-prefill.
• Helps only when the model does not fit in RAM. If everything is cached, WILLNEED is a no-op and DONTNEED would evict useful cache — hence opt-in, not default.
• Prefill only; decode is unaffected by design.
• Arch-independent: the marker is applied by scanning expert tensors generically in the loader; no per-model code.

Notes from the road not taken

We tried the textbook interventions first; all were neutral or harmful in this regime: fadvise(DONTNEED) after reads destroyed the kernel's (correct) caching decisions, --no-mmap traded thrash for slower steady state, and O_DIRECT is incompatible with letting the page cache act as a second-level expert cache. The kernel LRU behaves well here — the single gap is that it cannot predict expert access order, which this PR fills with a two-slice-ahead hint.

🤖 Generated with Claude Code