Family-tier CI battery: parity oracle on llama bumps, hybrid boundary sweeps, tier-3 family removals

[i386]

Context

Split-model serving on mixed-hardware labs shows inconsistent, family- and split-boundary-dependent failures. Root-cause review of #1420 and #1429 shows the failures are bugs, not methodology, and that CI currently proves "the splits we tested still run" rather than "any split we'd ship is equivalent to not splitting at all."

Key findings motivating this issue:

• B1 (High): model_requires_recurrent_state (crates/skippy-server/src/kv_integration/config.rs:173-186) probes only the first layer file in [layer_start, layer_end). Interleaved hybrids (Kimi-Linear, Qwen3-Next) whose stage starts on an attention layer misresolve Auto → ResidentKv, so recurrent state never crosses the boundary. Whether it fails depends on where the planner cuts. Not fixed by feat(skippy): replace flat prefix caches with unified radix #1429; the unified radix tree makes payload selection the single point of failure.
• B2 (High): patch 0001 drops synchronize() before embd_seq.clear() in encode/decode and from ~llama_context() while async copies may be in flight → nondeterministic corruption scaling with model size.
• B4/B5 (High): stage_export_kv_page never exports k_idx indexer rows (GLM-DSA/MSA silently wrong; blocks context-shift) and requires uniform K/V row sizes across stage layers.
• feat(skippy): replace flat prefix caches with unified radix #1429 adds two HIGH bugs in the radix insert/replace path (concurrent duplicate resident record leaks a native seq; replace resets active_refs), plus a regression: serving-KV record ladder removed (partial compensation for KvRecurrent).

This issue covers the CI and family-matrix work to keep these bug classes from regressing (bug fixes themselves tracked separately).

Family tiers

Artifacts are the certification set CI actually runs — not the largest available model per family. File sizes are the GGUF download size per artifact, pulled from the artifact source (HF tree API); ~ marks an estimate for repos behind auth where the tree API is unavailable. Package-backed rows show the full layer-set size; CI materializes only the owned stage range.

Tier 1 — full battery on every llama bump

Parity oracle (full-model vs split, token-exact) + dtype lanes + boundary sweep for hybrids.

Family	Cert artifact	File size	Why Tier 1
Qwen3 dense	Qwen3-0.6B Q8_0	0.64 GB	Core prod family; state-size baseline
Qwen2	Qwen2.5-0.5B Q8_0	0.53 GB	Prod dense lineage
Llama	Llama-3.2-1B Q4_K_M	0.81 GB	Prod dense
GLM-4.7 Flash	GLM-4.7-Flash Q4_K_M	18.3 GB	Prod; MLA runtime path shared with DeepSeek2
GLM4 9B	glm-4-9b Q4_K_M	6.2 GB	Prod dense
DeepSeek2	Coder-V2-Lite Q4_K_M	10.4 GB	Prod MLA path
DeepSeek3	V3.2 UD-Q4_K_XL layer package	406.8 GB (layer set; per-stage materialization)	Prod MoE; package-backed lane only
Qwen3-MoE	Qwen3-MoE-4x0.6B Q4_K_M	0.96 GB	Prod MoE (also not in graph_max_nodes allowlist — B8)
Qwen3-VL-MoE	Qwen3-VL-30B-A3B MXFP4_MOE + mmproj F16	17.1 GB	Prod multimodal MoE
MiniMax M2.7	M2.7 UD-Q2_K_XL (3 shards)	75.3 GB	Prod MoE; minimax family carries MSA/indexer risk (B4)
Jamba	Jamba2-3B Q4_K_M	1.9 GB	Hybrid — boundary sweep
Kimi Linear	Kimi-Linear-48B-A3B IQ2_XXS	12.1 GB	Hybrid, 3:1 interleaving — boundary sweep (B1 canonical case)
Qwen3Next	Qwen3-Coder-Next IQ2_XS	22.2 GB	Hybrid (3 linear : 1 attention) — boundary sweep (B1)
Mamba	mamba-130m Q4_K_M	90 MB	Recurrent — cheap boundary sweep
Mamba2	mamba-2.8b Q4_K_M	1.6 GB	Recurrent — boundary sweep
RWKV6	finch-1.6B Q4_K	0.99 GB	Recurrent, v_first sideband crossing
RWKV7	rwkv7-191M Q4_K	264 MB	Recurrent, cheap
Falcon-H1	H1-1.5B Q4_K_M	0.94 GB	Hybrid — boundary sweep
Granite-Hybrid	Granite-4.0-H-350M MXFP4	335 MB	Hybrid — cheap boundary sweep
LFM2	lfm2-350m Q4_K_M	229 MB	Hybrid — cheap boundary sweep
Laguna S 2.1	pinned Q4_K_M layer package	per-stage materialization only	Hybrid, 131k-context contract

~21 rows. Final prod list should be confirmed against serving telemetry.

Tier 2 — compile + single fixed-split parity, llama bumps only

Catch gross graph/tensor-layout breakage cheaply.

Family	Cert artifact	File size	Note
Qwen2-VL / Qwen3-VL	2B Q4_K_M + mmproj	0.99 GB / 1.1 GB	Multimodal sideband paths
Gemma2 / Gemma3 / Gemma3n / Gemma4 A4B / Gemma4 E4B	Q4_K_M ×3, Q6_K, Q4_K_M	1.7 / 0.8 / 2.8 / 22.6 / 5.0 GB	Boundary-constrained (E4B) + AltUp sidebands (3n)
Phi2 / Phi3 / PhiMoE	Q4_K_M ×3	1.8 / 2.4 / 25.3 GB	Dense + small MoE
Granite (dense)	3.2-2B Q4_K_M	1.5 GB	Dense
Hunyuan-Dense / -MoE / -VL	Q4_K_M / IQ2_XXS / OCR Q8+mmproj	1.1 / 27.6 / 0.58 GB	Includes multimodal
Falcon (dense)	7B Q4_K_M	5.0 GB	Dense
InternLM2	1.8B Q4_K_M	1.2 GB	Dense
OLMo / OLMo2 / OLMoE	F16 / Q4_K_M / Q4_K_M	13.8 / 4.5 / 4.2 GB	Dense + MoE
Mistral3	3B Q4_K_M	2.1 GB	Dense
Qwen2-MoE	1.5B-2x Q4_K_S	2.3 GB	Small MoE
EXAONE / EXAONE4 / EXAONE-MoE	Q4_K_M / Q4_K_M / 236B pkg	1.6 / 0.8 GB / 143.5 GB (package)	EXAONE-MoE package-backed lane
Cohere2	7B Q4_K_M	5.1 GB	Dense
Qwen3.5 recurrent	4B Q4_K_M	3.0 GB	Recurrent variant — promote to Tier 1 if it enters prod
SmolLM3 / MiniCPM / MiniCPM3 / Plamo3	Q4_K_M	~1.9 / ~1.2 / ~1.8 / ~1.2 GB	Dense, low traffic
DeepSeek-OCR	Q8 + mmproj	~2.7 GB	Multimodal

Tier 3 — DROPPED

No backward compatibility. Dropped families stop loading on the next release with a clean "unsupported family" error. Rationale per row below; none have meaningful serving traffic (verify against telemetry before merge, but do not keep rows out of politeness).

Family	Cert artifact	File size	Why dropped
GPT2	gpt2 Q4_K_M	113 MB	Legacy arch, superseded many times over
Bloom	bloomz-560m Q4_K_M	561 MB	Dead lineage (training discontinued)
GPT-NeoX	pythia-70m Q4_K_M	49 MB	Research-only lineage
MPT	mpt-7b-chat Q4_K_M	4.3 GB	Dead lineage
CodeShell	7B Q4_K_M	~4 GB	Niche, no traffic
OpenELM	270M Q4_K_M	~160 MB	Research-only (Apple never shipped it widely)
Refact	1.6B fim Q4_K_M	~1 GB	Niche FIM model
StableLM	zephyr-3b Q4_K_M	1.7 GB	Discontinued
PLM	1.3B Q4_K_M	~0.8 GB	Niche (42dot)
Arcee	AFM-4.5B Q4_K_M	~2.8 GB	Niche
Deci	DeciLM-6b Q4_K_M	~3.6 GB	Niche
ChatGLM	chatglm3-6b Q4_K_M	~3.7 GB	Superseded by GLM4 rows
Baichuan	Baichuan2-7B Q4_K_M	4.6 GB	Superseded, minimal traffic
Granite-MoE	tiny-random toy GGUF	5 MB	Layout-parity toy only, not a real artifact
SmallThinker	3B preview	~1.9 GB	Preview toy
Maincoder	1B Q4_K_M	~0.6 GB	Niche
XVerse	7B Q4_K_M	~4.2 GB	Niche regional model
StarCoder2	3B Q4_K_M	~1.8 GB	Niche code model (revisit if traffic appears)

Removal must include: graph edits from the llama patch series (the real rebase surface), family_capability.rs expectation/infer entries, and a hard "unsupported family" error at load — no inference-ladder fall-through into a wrong-but-plausible config. A release-notes breaking-change entry; no deprecation window.

Build once, test many

• One builder job per run: patched llama.cpp + skippy release binary, upload-artifact keyed on source SHA + llama pin.
• Test jobs download the artifact and run just family-certify — zero downstream compiles.
• Reuse existing rails: configure-sccache-gha + capture-sccache-stats + trusted sccache seed; extend the mesh-llm-<os>-skippy-ci-smoke-models-* model-cache key pattern to tier artifact sets; cache corpora on hashFiles. Bump .github/cache-version.txt once on landing.

Workflow wiring (3 jobs per bump)

• llama-upstream-canary.yml: add hybrid parity + boundary sweep (every cut offset mod interleaving period, small artifact) and gate the auto pin-commit on parity passing (today it commits on cargo check alone — the silent-drift hole). Path-trigger on third_party/llama.cpp/patches/** so patch-series edits also run it.
• nightly-stability.yml: repeat-determinism lane (same model, ~5 runs, varying threads/batch mixes, bit-exact) to catch the B2 async-free class.
• skippy-radix-cache-benchmark.yml: wire kv-recurrent gate + a concurrent same-prompt record test (H1) as required checks.
• static-abi-artifact.yml: add C-side static_assert mirror of Rust #[repr(C)] layout for skippy_iteration_request and friends.

New test coverage required

• Hybrid stage whose first layer file is attention-only must be detected as recurrent (the B1 shape — nothing tests it today).
• Synthetic tiny hybrid GGUF fixture with known attention/recurrent interleaving, exercised at attention-first boundaries.
• End-to-end KvRecurrent restore (KV page + recurrent state) through the radix cache.
• Concurrent duplicate resident record (H1 leak shape).

Out of scope (tracked separately)

• The bug fixes themselves: B1 probe, async sync restore, k_idx export, 0016 n_embd > n_embd_inp overflow + partial-failure divergence, feat(skippy): replace flat prefix caches with unified radix #1429 insert/replace ref-awareness.
• TSAN/ASAN/fuzz lanes (add when a real failure demands it).

[ndizazzo]

@i386 Yeah this makes sense... totally agree we need to implement some kind of regression validation like this.

One suggestion I think that could make sense is to maintain a manifest matrix for CI instead (like we do for our release packaging repo) so moving a family between tiers, or adding one is a line in some JSON config somewhere.