models+gemm: Phi-4 and Granite-4-hybrid families, DiffusionGemma e2e (parity+coherent), gemma4 batched prefill (5.1-5.5x), fp16-GEMM row-grid fixes - #99
Merged
Conversation
…ry; lane-divergence unresolved — no cross-lane bleed per perm test, A/B both degenerate on diverging real-prompt lanes)
…d fallback + generic gemm API) — gemm_fp16 tiles BM=32; M>32 skipped rows 32..63 per block
…nly + one-time phi3 GGUF repack phi3 arch fits the core exactly (head_dim 128, full rotary per phi3.rope.dimension_count=128, plain NeoX rope theta=5e5, GQA 40/10, untied Q6_K head, no qkv bias / qk-norm / sliding window). The only structural gap is artifact-side: phi3 GGUFs fuse attn_qkv ([Q;K;V]) and gate_up (as ffn_up, 2*n_ff rows); repack_phi3_gguf.py splits them offline by row-range byte copy (K-quant rows independent -> bit-exact, no requant), keeping the loader and kernels untouched. Gates on amelia (M4 16GB, CLT-only runtime compile, colima resident): load+config from GGUF header PASS; greedy coherence PASS (fluent <think> reasoning, correct unit math, finite logits); Qwen3-1.7B-Q8 32-tok A/B vs pristine origin/main TOKEN-IDENTICAL (also regression-checks the bf05198 fp16-GEMM row-grid fix); decode median 10.49 tok/s pure / 9.79 generate (non-canonical host); batched N=4 lockstep smoke PASS.
… (config-only + one-time phi3 GGUF repack; 10.49 tok/s decode amelia)
…s + 2 PLE-inject GEMMs) — gemm_fp16 tiles BM=32, /64 grid skipped rows at M>32 (audit per deepseek bug class)
…root-caused as inherited seq>1-vs-seq=1 attention numerics STATUS + gate drivers + artifacts. Headline: N=8 T=128 serving TTFT 5.11x (single chunk) / 5.54x (chunk=64) on derek M4 16GB, decode aggregate 1.0000, old paths byte-identical (final build, batch=1 + batch=8). Lane correctness: chunk-split==single-chunk 8/8; identical-prompt and bitwise lane-permutation invariants hold on random/raw-fluent/chat prompt sets; first token 8/8 vs BOTH references (token-by-token lockstep AND single-stream forward(T)) in all regimes; 32-tok continuations 7/8 vs the single-stream seq>1 reference across different decode engines. The non-8/8 continuation-vs-token-by-token gate (6/8 chat, 3/8 raw, 2/8 random) is pre-existing: zero-new-code base-dylib demo shows the EXISTING single-stream prefill diverges from token-by-token at the same rates on the same prompts (6/8 chat sharing prompt 1; 4/8 raw, one at index 0) — the q_seq==1 split-Bc decode fast path vs q_seq>1 sequential online-softmax orderings differ in bf16 rounding. Earlier 'degenerate baseline' mystery was the harness tiling prompts x10. Row-grid audit table in STATUS: 6 BUG sites (committed in c75ea97), all dormant for 12B-unified (no PLE, Q4K body, Q8 embed head); latent repairs for PLE E-variants + bf16 fallbacks.
…ill (5.11-5.54x N=8 TTFT) + 6-site bf16-GEMM row-grid fix (latent PLE/bf16-fallback repairs)
… bidirectional canvas), sampler spec (entropy-bound + SC), byte budget (experts 14.09/15.64 GiB), 2-host fit plan, staged port
…B), 2.0% (14B); projections own prefill (3.31 ms/tok linear term)
… oracle, Metal unified forward), op tests green
… (compare_logits, compare_dumps, run_sk)
…-out QK^T + f32-in softmax (kq range parity with ggml PREC_F32), EOF-safe mmap windows
…er resident growth of touched no-copy windows around layer 26)
…py windows triggered system swap storms via CoW-broken GPU mappings); non-fatal dump writes
…d every MTLBuffer proxy ~0.85GB/layer; the actual leak behind both box takedowns)
…lloc churn outside rss swap-stormed the host even with eviction+pools; steady state is now pure memcpy)
…fill; resident file-cache growth was inflating rss and swap pressure
…anvas argmax 100%; llama.cpp FA-flip self-agreement 71%/61% bounds the achievable bar); full STATUS + parity tables
…9937 (verified); fma-contracted exp-arg mirrored; synthetic-logit gate: trajectory token-identical, 2/1792 discarded multinomial picks flip on cum plateaus (reference itself flag-unstable there)
…raph + e2e generate driver in progress, ungated)
…iffusion); generate.py skips SC compute at step 0 (sc_use=0 is bit-identical to zero-SC, saves the 1.5 GB embT stream)
… 1 logits parity (mask-canvas argmax 100% vs PR#24423) + bit-exact EntropyBound sampler; e2e staged (Stage 2 in flight)
…ion 5.1% of TTFT @t=512, shrinks with size; TTFT owned by linear projection term)
…ically vs instrumented reference (sc_sig oracle cos 0.9999999 / GPU 0.9999998; SC-active GPU step-1 logits argmax 256/256 vs ref; use_sc=0 gate exact-zero confirmed)
…nce logits (instrumented cli, S=10 seed=1234 C=256; 7-step trajectory incl. adaptive stop, 0/70 decision fields differ, max|dH| 2.4e-7)
…anite multipliers) granite-4.0-h-1b: 36 mamba2 + 4 attention layers (per-layer type from GGUF head_count_kv), dense SwiGLU FFN every layer, tied Q8_0 head. Reuses mamba2 family kernels (conv1d_silu/_step, mamba2_ssd, gate_norm) and shared dense kernels (q8_0_matvec, mha_causal D=128, kv_cache_write, rmsnorm, silu_mul). New granite_ops.metal carries the granite scalar multipliers; Q is pre-scaled by attention_scale*sqrt(D) so mha_causal's hardcoded 1/sqrt(D) becomes the granite attention_multiplier. h-1b over h-micro because h-micro is head_dim=64 (SK dense attention is D=128-only).
… buffers) + early probs16 + in-place softcap — multi-forward generation swap-stormed the tight-disk 16GB host (swapfiles ate root disk, watchdog/jetsam kills); forward verified BIT-IDENTICAL pre/post (zero-SC and SC legs)
…es 1-2 GREEN, e2e run pending)
…G/90s polls; watchdog the gate-4 cli leg
… France' (12 steps adaptive stop, 0 mask tokens, finite logits, correct answer; swap flat ~1.1G over 12 forwards)
…ps) + p2 (full-canvas haiku w/ self-critique; trim heuristic false-cuts comma lists) coherent; cli cross-check agrees on p1 (Paris, 8 steps); baseline 49.5-50.4 s/step wall, fw 98% — CPU cli 29.85 s/step beats SK GPU 1.65x, Stage-3 surface is the forward
…T in SK (3/3 prompts, SC active, adaptive stop; cli cross-check agrees; baseline 49.5s/step, forward=98% of wall)
…GOLD vs llama.cpp CPU greedy; poem fp16 near-tie divergence @8, gap 0.125; qwen3-1.7b no-regression 32/32 vs pristine main; 51.2 tok/s median decode)
…amba2 + 4 NoPE-attention layers) — QA 48/48 token-GOLD vs llama.cpp, 51.2 tok/s decode on lexie
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.