Explore CommitLLM as an optional verified-inference backend

[i386]

Summary

Explore how mesh-llm could use lambdaclass/commitllm to add optional verifiable inference to the mesh.

The key framing is that commitllm is not a replacement for mesh-llm's routing/discovery/runtime model. It is a provenance layer for inference: serve normally, return a compact commitment/receipt, and later answer audit challenges that a verifier can check against the claimed weights, config, transcript state, and decode policy.

Why this is interesting

mesh-llm already solves distributed placement, routing, discovery, and client access across a peer mesh. commitllm could add a trust layer on top for cases where clients need evidence that a remote node actually ran the claimed model and serving policy.

Potential value:

• auditable inference for untrusted or mixed-operator meshes
• stronger model-identity guarantees for benchmarks/evals
• compliance-oriented or premium "verified inference" mode
• groundwork for receipts/challenges in marketplace-style or third-party compute settings

Current fit

There is a strong conceptual fit, but the current implementations do not line up directly.

mesh-llm today:

• routes requests through a QUIC mesh
• uses llama.cpp (llama-server + rpc-server) as the inference backend
• assumes the inference layer is mostly unmodified and reachable over HTTP/TCP tunnels

commitllm today:

• is implemented around vLLM, not llama.cpp
• relies on runtime instrumentation/capture hooks in the serving path
• returns a response plus a commitment/receipt and supports later audit openings
• still has llama.cpp plugin, OpenAI-compatible proxy, Streaming / SSE, and several production-serving pieces on its roadmap

This means commitllm is not currently a drop-in addition to the existing llama.cpp mesh path.

Recommended direction

Phase 1: optional verified backend

Add support for a new backend class in mesh-llm for verified inference, likely something like verified-vllm.

Scope:

• keep mesh-llm as the network/routing/control plane
• run verified nodes as an alternate backend rather than modifying the current llama.cpp path
• likely constrain verified nodes to solo serving initially rather than tensor-split serving
• allow the mesh to advertise that a given model/backend supports receipts and audits
• preserve current behavior for normal llama.cpp nodes

Initial API shape:

• inference response includes request_id and commitment/receipt metadata when the selected backend is verified
• add an audit endpoint or management endpoint that proxies audit requests to the serving backend
• expose verification capability in status/UI so clients know which nodes/models are auditable

Phase 2: verifier and UX surface

Add a verifier-facing flow to mesh-llm:

• local CLI or plugin command to verify an audit proof from a receipt/challenge
• console/API support for showing verification capability and challenge/audit status
• optional receipt persistence and lookup by request ID

Phase 3: deeper native integration

Longer term, investigate native llama.cpp capture rather than treating verified inference as an alternate backend.

That would likely require:

• retained-state capture from llama.cpp / rpc-server
• canonical sampler parity
• GGUF/ggml model identity binding compatible with the verifier story
• a protocol for receipts/audits in tensor-split or multi-node execution

This is a research/runtime project, not an adapter.

Main technical questions

• How should backend capability be modeled in node status, routing, and model advertisements?
• Should verified inference be a per-node backend choice, a per-model capability, or both?
• What is the smallest stable API extension for returning receipt metadata without breaking existing clients?
• Where should audit requests live: inference API, management API, or plugin surface?
• Do we want the first version to support only solo verified nodes?
• What is the long-term plan for llama.cpp compatibility vs running a parallel verified backend?

Implementation notes

A reasonable first milestone is not to modify the current llama.cpp serving path.

Instead:

1. add backend abstraction where needed so a model can be served by either llama.cpp or a verified backend
2. route verified requests through the existing mesh proxy/control plane
3. return receipt metadata to clients when present
4. add an audit passthrough endpoint
5. keep tensor-split and native llama.cpp verification out of scope for the first cut

Acceptance criteria for the first milestone

• a node can advertise a verified backend capability for at least one model
• routed inference to that model returns normal output plus receipt metadata
• a client can request an audit opening for a prior response using the returned request ID
• current non-verified llama.cpp flows are unchanged
• status/API/UI surfaces make verified capability discoverable

References

• mesh-llm: QUIC mesh + llama.cpp routing/control plane
• commitllm: commit-and-audit provenance layer for open-weight inference
• commitllm roadmap items worth tracking: llama.cpp plugin, OpenAI-compatible proxy, Streaming / SSE

[i386]

Follow-up from source inspection of lambdaclass/commitllm at current HEAD (055ba12, 2026-04-03): there does not appear to be any landed llama.cpp runtime integration yet.

What I found:

• There is a roadmap item for llama.cpp plugin — capture retained state from llama.cpp serving path.
• There are a few protocol/type hints that look ggml/Q8_0-aware, but not a runtime path.
• The actual implemented serving/runtime path is still strongly vLLM-specific:
  • VerifiedInferenceServer uses vllm.LLM
  • startup code patches vllm._custom_ops.cutlass_scaled_mm
  • sampler/capture hooks target vLLM internals
  • weight loading is safetensors-based, not GGUF-based

So for planning purposes, this should be treated as:

• Llama model-family support: yes
• llama.cpp runtime support: not yet

That reinforces the recommendation in this issue: if we explore this, the first practical step is an optional verified backend path rather than trying to bolt it directly onto the current llama.cpp mesh runtime.

[michaelneale]

Deep Technical Analysis: CommitLLM ↔ Mesh LLM Integration

I did a thorough code-level analysis of both repos. Here's what I found about compatibility, the real gaps, and a concrete integration path.

---

Compatibility Matrix

Dimension	CommitLLM	Mesh LLM	Compatible?
Language	Rust (verifier, core) + Python (prover sidecar)	Rust (mesh) + C++ (llama.cpp)	Both Rust workspaces ✅
Architecture	Prover → receipt → challenge → audit → verify	Node → response → proxy → client	Protocol fits naturally ✅
API shape	HTTP /chat + /audit (FastAPI)	HTTP /v1/chat/completions (OpenAI-compat proxy)	Mesh proxy can forward/attach ✅
Plugin model	N/A (standalone server)	Protobuf IPC subprocess plugins with mesh-wide forwarding	CommitLLM verifier fits as plugin ✅
Quantization	W8A8 (INT8 weight+activation via cutlass_scaled_mm)	GGUF Q4_K_M, Q5, Q8_0 etc. (llama.cpp k-quant)	❌ Major gap
Inference engine	vLLM (Python, PyTorch, CUDA)	llama.cpp (C++, Metal/CUDA/ROCm/Vulkan)	❌ Major gap
Models validated	Qwen2.5-7B-W8A8, Llama-3.1-8B-W8A8	Qwen2.5-32B, GLM-4.7, Qwen3-8B etc. (GGUF)	Same families, different quant ⚠️

The Two Hard Gaps (in detail)

1. Inference engine mismatch

CommitLLM's prover sidecar (sidecar/verilm/) hooks vLLM at the kernel level. Specifically:

• _startup.py installs an import hook that wraps vllm._custom_ops.cutlass_scaled_mm
• capture.py uses call-counting (4 calls per layer: qkv_proj → o_proj → gate_up_proj → down_proj) to identify which layer/projection each matmul belongs to
• hooks.py installs PyTorch forward hooks on embed_tokens, LogitsProcessor, and final RMSNorm for embeddings, logits, and residuals
• sampler.py injects a canonical ChaCha20-seeded sampler as a forward hook on LogitsProcessor

None of this transfers to llama.cpp. The capture mechanism is fundamentally vLLM-specific.

2. Quantization format mismatch

CommitLLM's Freivalds check operates on:

z_i32 = W_i8 @ x_i8    (INT8 × INT8 → INT32 accumulator)
v · x  =?  r^T · z      (check in F_p where p = 2^32 - 5)

llama.cpp's Q4_K_M does:

z_f32 = dequant(W_q4) @ x_f32    (dequant to FP16/FP32, then FP matmul)

The entire verification math — Freivalds modular arithmetic in verilm-core/src/freivalds.rs, bridge requantization in verilm-core/src/rmsnorm.rs, attention corridor bounds in verilm-verify/src/corridor.rs — is built around INT8. These aren't configuration options; they're structural to the protocol.

Lucky Breaks

Despite those gaps, several things align well:

1. llama.cpp already has a tensor-level eval callback: ggml_backend_sched_eval_callback fires for every tensor node during graph compute. The callback receives struct ggml_tensor * t with name, shape, type, and data. This is the hook point for capture — equivalent to CommitLLM's cutlass_scaled_mm wrapper.

2. The mesh plugin system is purpose-built for this: Subprocess isolation, framed protobuf IPC over Unix sockets, mesh-wide ChannelMessage forwarding via QUIC, BulkTransferMessage for larger payloads, MCP tool exposure, management API exposure. The blackboard plugin already demonstrates the full pattern.

3. The verifier is pure Rust, CPU-only, fast: verilm-verify and verilm-core have zero Python/GPU dependencies. Verification takes ~1.3ms per challenged token (routine audit) or ~10ms (full audit). These crates link directly into any Rust binary.

4. CommitLLM's roadmap explicitly calls out mesh-relevant items: llama.cpp plugin (Add end-to-end proxy robustness integration tests #72), OpenAI-compatible proxy (Add native MLX backend and backend benchmarks #57), MoE support (fix: Nvidia Orin AGX VRAM stats #29), pipeline parallelism (ci: integration smoke tests — real inference on Linux, macOS, Windows (docs) #47), marketplace integration (Upgrade GitHub Actions and enforce Node 24 #73).

---

Concrete Integration Plan

Phase 1: Verifier-side plugin (1-2 weeks, zero engine changes)

Build a commitllm mesh-llm plugin that runs the verifier. No changes to llama.cpp, no changes to CommitLLM's prover.

┌──────────────────────────────────────────────────┐
│ Mesh LLM node (client)                           │
│                                                   │
│  ┌──────────────────┐   ┌──────────────────────┐ │
│  │ mesh-llm host    │   │ commitllm plugin      │ │
│  │                  │◄──┤ (verifier)            │ │
│  │ • proxy          │   │                       │ │
│  │ • gossip         │   │ • verify receipts     │ │
│  │ • plugin IPC     │   │ • challenge endpoint  │ │
│  │                  │──►│ • score peers         │ │
│  └──────────────────┘   └──────────────────────┘ │
│           │                        │              │
│           │ mesh gossip            │ MCP          │
│           ▼                        ▼              │
│    other nodes               agent tools          │
└──────────────────────────────────────────────────┘

Steps:

1. Create commitllm-plugin binary crate linking verilm-verify + verilm-core as Cargo dependencies
2. Implement plugin protocol (use mesh-llm/src/plugins/example/ as template)
3. Register MCP tools: commitllm_verify_receipt, commitllm_challenge, commitllm_peer_trust
4. Define commitllm.v1 channel for receipt delivery and audit challenge/response between mesh peers
5. Receipt transport: X-CommitLLM-Receipt header on proxied responses, or /v1/receipts/{id} endpoint

Value: Any mesh client can verify receipts from CommitLLM-enabled servers (vLLM-based). Establishes the plugin infrastructure for later phases.

Phase 2a: Witness-mode capture in llama.cpp (1-2 weeks)

Skip Freivalds. Use the existing ggml_backend_sched_eval_callback to capture:

• Output logits per token (tensor name: result_output or similar)
• Model GGUF file hash (computed at startup)
• Decode params (temperature, top-k, etc.)

Receipts prove: "this GGUF file produced these tokens with these settings."

What witness mode catches:

• ✅ Model substitution (serving Qwen-3B while claiming Qwen-32B)
• ✅ Decode policy lies (claiming greedy while using temperature=1.0)
• ✅ Output tampering (modified tokens after generation)
• ✅ Quantization substitution (serving Q2 while claiming Q4_K_M)
• ❌ Per-layer weight manipulation (skipping layers for speed)
• ❌ Attention shortcutting

This is weaker than full Freivalds but catches the #1 economically motivated attack (model substitution) and ships fast.

Steps:

1. Add capture callback registration to mesh-llm's start_llama_server() in inference/launch.rs — or more practically, add a small C++ shim library that llama-server loads
2. Capture logits + token IDs per request via shared memory / Unix socket to the plugin
3. Hash GGUF file at model load time (R_W equivalent)
4. Plugin builds witness receipts: (model_hash, token_ids, logits_hash, decode_params, nonce)
5. Extend verilm-verify with a witness-mode verifier path (simpler than full Freivalds)

Phase 2b: Q8_0 Freivalds (research, 2-4 weeks)

llama.cpp's Q8_0 format is per-block INT8 with a single f16 scale per 32 elements — structurally closest to W8A8. The Freivalds check could be adapted to block-quantized format.

Why Q8_0 specifically:

• Q8_0 is native INT8 with known scales → Freivalds modular arithmetic applies within blocks
• llama.cpp has a Q8_0 compute kernel (ggml_compute_forward_mul_mat with GGML_TYPE_Q8_0)
• Q8_0 models are ~2× larger than Q4_K_M but still much smaller than FP16
• Mesh LLM could support a --verified flag that forces Q8_0 for auditable inference

Open research questions:

• Does llama.cpp's Q8_0 matmul produce deterministic INT32-equivalent accumulators?
• What do corridor bounds (attention replay mismatch) look like on llama.cpp vs vLLM?
• Can per-block Freivalds be composed into a per-layer check without losing soundness?

Phase 3: Trust scoring (1 week after Phase 2)

1. Rolling verification score per peer, gossipped across the mesh
2. Routing preference: prefer verified nodes (extend existing RTT + VRAM scoring in network/router.rs)
3. Configurable audit sampling rate (e.g., challenge 5% of responses)
4. Visual feedback in web console (port 3131): green/yellow/red verification badges per node

Phase 4: Distributed verification (research)

Pipeline parallel: Each node produces receipts for its layer range. The coordinating node (llama-server host) stitches per-range receipts. Verifier checks contiguity + completeness. Maps to CommitLLM roadmap #47.

MoE expert sharding: Each node commits its expert subset identity + weight hashes. Router commits routing decisions. Verifier checks: correct routing → correct expert selection → correct expert weights. Maps to CommitLLM roadmap #29. Particularly interesting because mesh-llm's MoE mode runs independent llama-servers per node — the "trust boundary" is at the expert routing level.

---

Key Design Decisions

Q: Where should the verifier key come from?

Verifier keys contain Merkle roots over weights + secret Freivalds vectors. For the mesh:

• Witness mode: just needs model hash (GGUF SHA-256), no Freivalds vectors → trivial
• Full Freivalds: keys computed from public weights by any node, distributed via gossip or bundled with model catalog entries
• verilm-keygen would need a GGUF parser (currently only supports safetensors)

Q: Who does the verification?

Client nodes. The verifier is CPU-only and fast. A mesh client running mesh-llm --client (no GPU) can verify every response it receives. The commitllm plugin runs on the client side.

Q: What's the overhead budget?

CommitLLM measures 12-14% online overhead on vLLM/CUDA. On llama.cpp:

• Witness mode: logit capture only → near-zero (logits are already computed, just copy them)
• Q8_0 Freivalds: eval callback fires for every tensor → need filtering. Only capture ~8 tensors per layer per token out of potentially hundreds of intermediate tensors. With filtering, should be similar order.
• The mesh already tolerates higher latency for split models (21 tok/s 2-node vs 68 solo). A 10-15% overhead for verification is within budget.

Q: Enforce Q8_0 for verified inference?

Probably yes, for the Freivalds path. Q4_K_M's dequant→FP→matmul path fundamentally doesn't produce the INT32 accumulators Freivalds needs. Practical answer: mesh-llm --model X --verified forces Q8_0 quant. Witness mode works with any quant.

---

Summary

Phase	What	Closes	Timeline
1	Verifier plugin	Clients verify receipts from any CommitLLM server	1-2 weeks
2a	Witness-mode capture	Model substitution detection on llama.cpp nodes	1-2 weeks
2b	Q8_0 Freivalds	Full algebraic verification on llama.cpp (Q8_0 only)	2-4 weeks (research)
3	Trust scoring	Verification affects mesh routing	1 week
4	Distributed verification	Pipeline parallel + MoE receipts	Research

The fastest path to real value: Phase 1 + Phase 2a. Together they give you a mesh where clients can detect model substitution — the dominant economic attack in untrusted compute — without solving the hard Freivalds-on-GGUF problem. Phase 2b adds the strong cryptographic guarantee for nodes willing to serve Q8_0.

[michaelneale]

Correction: Q8_0 Freivalds is already implemented in CommitLLM

Digging deeper into the code, I found that CommitLLM already has Q8_0 block-aware Freivalds implemented and tested in verilm-core/src/freivalds.rs. This is not a research item — the math is done.

Specifically, verilm-core contains:

• Q8_0_BLOCK_SIZE = 32 matching llama.cpp's Q8_0 format (32 elements per block)
• Phase A: check_q8_blocks() — batched block Freivalds check. The prover sends per-block sumi accumulators (exact INT8×INT8→INT32 within each block of 32 elements). The verifier checks all blocks in one batched algebraic test using verifier-secret random coefficients. Soundness is Schwartz-Zippel, same as the W8A8 path.
• Phase B: check_q8_assembly() — verifies that the final f32 output was correctly assembled from the verified block accumulators using the public per-block scales (d_w, d_x). This handles the Q8_0-specific scale multiplication: output[row] = Σ_b (d_w[row,b] · d_x[b] · sumi_b[row])
• derive_block_coefficients() — derives secret batching coefficients from the verifier key seed, strictly stronger than Fiat-Shamir (prover has zero information)
• Full test suite covering correct checks, wrong-sumi rejection, assembly tolerance

This maps directly to how llama.cpp computes Q8_0 matmuls internally:

1. For each block of 32 columns: multiply INT8 weights × INT8 input → INT32 sumi
2. Scale each block's sumi by d_w * d_x (the f16 block scales, converted to f32)
3. Sum across blocks to get the final f32 output

The Freivalds check verifies step 1 algebraically (exact, information-theoretically sound) and step 2-3 by canonical replay with tolerance.

What this means for the integration plan: Phase 2b ("Q8_0 Freivalds research, 2-4 weeks") shrinks dramatically. The verification math is done. What remains is:

1. Capture: hooking llama.cpp's Q8_0 compute path to extract per-block sumi arrays, block scales d_w/d_x, and input vectors — via the existing ggml_backend_sched_eval_callback
2. Keygen: extending verilm-keygen to parse GGUF Q8_0 weights and generate verifier keys (Merkle root + Freivalds vectors over the INT8 weight blocks)
3. Bridge/corridor: measuring the attention corridor bounds on llama.cpp's attention implementation (may differ from vLLM/CUDA)

The core crates (verilm-core, verilm-verify, verilm-prover) are also completely engine-agnostic — zero references to vLLM, PyTorch, CUDA, or any specific inference engine. The entire vLLM dependency is isolated in the Python sidecar (sidecar/verilm/). The Rust protocol, commitment, and verification layers are pure math.

So the real integration surface is:

• Protocol/verification: ready, engine-agnostic ✅
• Q8_0 Freivalds math: implemented and tested ✅
• Capture mechanism: needs a new llama.cpp-native capture layer (the only real gap)
• GGUF keygen: needs a GGUF parser for weight hashing + Freivalds precomputation

[michaelneale]

Deep dive: llama.cpp capture path and backend reality

After reading through the llama.cpp fork, ggml internals, and the CommitLLM paper/article, here is where things actually stand for making this integration real.

---

The eval callback — what exists today

llama.cpp has ggml_backend_sched_eval_callback — fires for every named tensor during graph compute. For a Llama model you get ~20 named tensors per layer:

Tensor	What
attn_norm-{il}	Post-RMSNorm input to attention
Qcur-{il}, Kcur-{il}, Vcur-{il}	Projection outputs (post-RoPE)
kq-{il}, kq_soft_max-{il}	Attention scores
kqv_out-{il}	Attention output
attn_out-{il}	Post-Wo projection
ffn_norm-{il}	Post-RMSNorm input to FFN
ffn_gate-{il}, ffn_up-{il}, ffn_down-{il}	FFN projections
l_out-{il}	Layer output
result_norm, result_output	Final norm + logits

The callback is already wired: llama_cparams.cb_eval flows through to ggml_backend_sched_set_eval_callback() in llama-context.cpp:1197. The llama-server launched by mesh does not set it (nullptr). The examples/eval-callback/ example shows it working end-to-end.

The callback gives you FP32 output tensors. This is enough to hash layer outputs and prove what model ran, but it does not give you the per-block INT32 accumulators that Freivalds needs.

---

The gap: sumi lives inside the dot product

CommitLLM's Freivalds check (check_q8_blocks) needs sumi_b — the per-block accumulator before scale multiplication. This value only exists transiently inside the dot product kernel:

// ggml_vec_dot_q8_0_q8_0_generic (ggml/src/ggml-cpu/quants.c:345)
for (; ib < nb; ++ib) {
    int sumi = 0;
    for (int j = 0; j < qk; j++) {
        sumi += x[ib].qs[j]*y[ib].qs[j];  // ← this is what Freivalds needs
    }
    sumf += sumi*(d_x * d_w);  // scale assembly consumes it immediately
}

sumi is a local variable. Never exposed. Extracting it requires modifying the dot product.

---

Backend math divergence — the critical finding

Backend	Weight format	Input quantized to	Dot product	Accumulator	Freivalds?
CPU generic	Q8_0	Q8_0 (on-the-fly)	INT8 × INT8	INT32	✅
CPU AVX2	Q8_0	Q8_0 (on-the-fly)	INT8 × INT8	INT32 (SIMD)	✅
CPU ARM NEON	Q8_0	Q8_0 (on-the-fly)	INT8 × INT8	INT32 (SIMD)	✅
CUDA dp4a	Q8_0	Q8_1 (on-the-fly)	INT8 × INT8	INT32 (dp4a)	⚠️ Close — Q8_1 has extra sum field
Metal	Q8_0	F32 (no quant)	INT8 × F32	F32	❌ No

Metal does qs[i] * yl[i] where yl is float. There is no INT32 block accumulator. The Metal kernel at ggml-metal.metal:3373 (kernel_mul_mv_q8_0_f32_impl) reads INT8 weights but multiplies against FP32 activations, accumulating into FP32 sums.

CommitLLM's Freivalds check assumes sumi_b[row] = Σ w_q8[j] · x_q8[j] (INT8 × INT8 → INT32). That math literally does not happen on Metal.

From the CommitLLM paper: "The key insight is that INT8 quantized inference consists of operations that are either deterministic or canonically recomputable, and can be verified exactly." The entire security model rests on integer arithmetic being exact. FP32 accumulation has rounding that compounds across the block — you can't get algebraic equality, only approximate bounds, which a cheating provider could exploit.

There is no known alternative in the literature. Freivalds works because r·(Wx) = (r^T W)·x is exact in a finite field. In FP32, floating-point multiplication is not associative and this identity doesn't hold. SafetyNets (Ghodsi et al. 2017) and zkCNN (Liu et al. 2021) — the closest related work cited by CommitLLM — also operate over integer arithmetic / arithmetic circuits.

MLX (remotes/origin/mlx-rs-native branch) uses mlx_rs::ops::quantized_matmul — need to investigate whether MLX's quantized matmul uses integer accumulation internally. This could be a path forward for Apple Silicon if MLX accumulates in INT32 before scaling.

---

What this means for the integration

CPU and CUDA backends: Freivalds can work. Requires modifying the dot product to write out per-block sumi values. Files to touch:

• ggml/src/ggml-cpu/quants.c (generic)
• ggml/src/ggml-cpu/arch/arm/quants.c (NEON)
• ggml/src/ggml-cpu/arch/x86_64/quants.c (AVX2/512)
• ggml/src/ggml-cuda/vecdotq.cuh (CUDA dp4a, with Q8_1 adjustment)
• ggml/include/ggml-backend.h (new capture mechanism)
• ggml/src/ggml-cpu/ggml-cpu.c (threading capture buffer through mul_mat dispatch)

Metal backend: No path to Freivalds. Options:

1. Force CPU backend for audited matmuls (massive perf hit)
2. Accept that Metal nodes can verify others' receipts but can't produce Freivalds-verifiable receipts themselves
3. Investigate MLX — if its quantized_matmul uses INT32 accumulation, MLX becomes the verified inference backend for Apple Silicon

All backends: The eval callback works everywhere for capturing FP32 layer outputs. This gives you model identity (GGUF hash) and the ability to hash/commit intermediate state, but without Freivalds binding those to specific weights, it's not the CommitLLM security guarantee.

---

Plan

This work should happen on a separate clone/branch of the llama.cpp fork — not touching the production fork.

1. Investigate MLX quantized_matmul internals — if it uses INT32 accumulation, this is the path for Apple Silicon verified inference
2. Prototype sumi capture on CPU backend — modify ggml_vec_dot_q8_0_q8_0_generic to optionally write block accumulators to a capture buffer, controlled by a flag
3. Wire eval callback in llama-server — add a --capture flag or similar that enables the callback for commitment/receipt generation
4. GGUF keygen — extend verilm-keygen to parse GGUF Q8_0 weights for Merkle roots and Freivalds vector precomputation
5. Measure attention corridor on llama.cpp — may differ from vLLM/CUDA measurements in the paper

Draft PR with the verifier-side plugin (correctly scoped but waiting on the prover side): #149

Full capture plan written up in CommitLLM repo: LLAMA_CPP_CAPTURE_PLAN.md

[michaelneale]

Update: capture hook built, hit the repack wall

Built the sumi capture hook on a separate llama.cpp branch: michaelneale/llama.cpp#1

What's there:

• ggml/include/ggml-capture.h — public C API: ggml_capture_set_hook(), ggml_capture_enable(), callback struct with per-block sumi, d_w, d_x
• Instrumented the generic and ARM NEON/MMLA ggml_vec_dot_q8_0_q8_0 paths
• Zero overhead when capture is disabled (single atomic load fast path)
• Test binary that loads a Q8_0 GGUF and reports capture statistics

What I found:

On Apple Silicon with i8mm (M1 Pro and later), ggml never calls the Q8_0 vec_dot function at all for matmul. Instead it uses repacked GEMM kernels:

ggml_gemm_q8_0_4x8_q8_0()   // in ggml/src/ggml-cpu/arch/arm/repack.cpp
ggml_gemv_q8_0_4x8_q8_0()   // single-token decode path

These use SMMLA (signed matrix multiply-accumulate) instructions with a repacked 4x8 weight layout. Weights are reorganized at load time for cache efficiency. The vec_dot path is dead code on modern Apple Silicon.

The same may apply to x86 with AMX, which also has its own repacked GEMM paths (ggml-cpu/amx/).

So the instrumentation points are:

Platform	Matmul path	Capture status
Generic C fallback	ggml_vec_dot_q8_0_q8_0_generic	✅ Instrumented
ARM without i8mm	ggml_vec_dot_q8_0_q8_0 (NEON dotprod)	✅ Instrumented
ARM with i8mm (M1+)	ggml_gemm_q8_0_4x8_q8_0 (repack.cpp)	❌ Not yet
x86 AVX2/AVX512	ggml_vec_dot_q8_0_q8_0	Needs instrumentation
x86 AMX	amx/mmq.cpp	Needs instrumentation

Next step: instrument ggml_gemm_q8_0_4x8_q8_0 in repack.cpp. The SMMLA kernel accumulates INT32 internally (that's what SMMLA does — 2×2 INT8 matrix multiply into INT32), so the Freivalds math still applies. Just need to extract the accumulator before the scale-and-reduce step.

[michaelneale]

Update: sumi capture working on Apple Silicon ✅

PR: michaelneale/llama.cpp#1

It works. 7.4M per-block INT32 sumi captures from a 2-token prompt on SmolLM2-135M Q8_0.

The key discovery was that Apple Silicon ggml uses repacked GEMM/GEMV kernels (repack.cpp) for Q8_0 matmul, not the vec_dot functions. The repack system:

1. Converts weights to interleaved block_q8_0x4 format at load time
2. Quantizes F32 activations to Q8_0 at runtime
3. Calls tiled GEMM (vdotq/vmmlaq → INT32) or GEMV per chunk

The INT32 accumulators are extracted after the NEON dot product accumulation but before the FP32 scale-and-reduce step. Each capture fires per (block, row, column) in the GEMM tile with:

• sumi: INT32 accumulator (the exact value Freivalds verifies)
• d_w, d_x: FP32 block scales
• result: per-block FP32 contribution

Overhead is zero when capture is disabled — single atomic_load fast path check.

The 4x4 (dotprod) and 4x8 (i8mm/SMMLA) variants are both instrumented for GEMM and GEMV paths. The test binary forces CPU with -ngl 0 and reports statistics.

Still TODO: x86 repack paths (AVX2/AVX512), CUDA dp4a, and wiring the capture output to verilm-prover to generate actual Freivalds receipts.