Q2_0 group 64: Metal backend

[khosravipasha]

DRAFT PR for tesintg and reviews.

[Copilot]

Pull request overview

Adds Metal backend support for the Q2_0 quantization format (group size 64) so quantized weights can participate in common Metal paths (copy/dequant, get_rows, mul_mv, mul_mm, and mul_mat dispatch).

Changes:

• Implements Q2_0 quantize/dequant routines in the Metal shader library and wires them into existing generic kernels (cpy/get_rows/mul_mm).
• Adds a Q2_0-specific dot-product routine and a dedicated kernel_mul_mv_q2_0_f32 path.
• Enables GGML_TYPE_Q2_0 in Metal op dispatch / device capability checks and sets up pipeline params for mul_mv/mul_mv_id.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
ggml/src/ggml-metal/ggml-metal.metal	Adds Q2_0 quantize/dequant, dot-product helper, and kernel/template instantiations for copy/get_rows/mul_mm/mul_mv.
ggml/src/ggml-metal/ggml-metal-ops.cpp	Allows Q2_0 to use the small-batch mul-mv-ext path in MUL_MAT when applicable.
ggml/src/ggml-metal/ggml-metal-impl.h	Introduces N_R0_Q2_0 / N_SG_Q2_0 constants for pipeline configuration.
ggml/src/ggml-metal/ggml-metal-device.m	Updates Metal device op support checks to include Q2_0 for relevant copy/dup/cont paths.
ggml/src/ggml-metal/ggml-metal-device.cpp	Adds Q2_0 case for mul_mv and mul_mv_id pipeline configuration (nsg/nr0).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[khosravipasha]

metal: Q2_0 backend

Overview

Follow up PR after merging of Q2_0 CPU PR[# 24448].
This PR adds the Metal backend support for Q2_0.

Supported Models

Supported Models: Ternary-Bonsai-{8B, 4B, 1.7B}-Q2_0_g64.gguf, more coming very soon.

Note: In our huggingface repos ggufs ending with _Q2_0_64.gguf are packed for this PR, Q2_0.gguf ones are older packing from our fork that have group 128, after all backends merge will rename and replace with the official Q2_0(group size 64)).

# downloads to models/Ternary-Bonsai-<size>-Q2_0_g64.gguf
hf download prism-ml/Ternary-Bonsai-1.7B-gguf Ternary-Bonsai-1.7B-Q2_0_g64.gguf --local-dir models
hf download prism-ml/Ternary-Bonsai-4B-gguf  Ternary-Bonsai-4B-Q2_0_g64.gguf  --local-dir models
hf download prism-ml/Ternary-Bonsai-8B-gguf  Ternary-Bonsai-8B-Q2_0_g64.gguf  --local-dir models

Correctness: Metal vs CPU logits KL

The ensure correctness we run the model twice: 1) Q2_0 CPU 2) Q2_0 Metal. We store the logits for both and compare the KL divergence. Using the standard llama-perplexity tool.

# reference logits from the CPU kernel
./build/bin/llama-perplexity -m models/Ternary-Bonsai-<size>-Q2_0_g64.gguf \
  -f datasets/wikitext-2-raw/wiki.test.raw -c 512 --chunks 20 -ngl 0 --save-all-logits cpu.bin
# Metal vs CPU
./build/bin/llama-perplexity -m models/Ternary-Bonsai-<size>-Q2_0_g64.gguf \
  -f datasets/wikitext-2-raw/wiki.test.raw -c 512 --chunks 20 -ngl 99 \
  --kl-divergence --kl-divergence-base cpu.bin

size	Mean KLD	Maximum KLD	Same top-1
1.7B	0.000000 ± 0.000000	0.000062	99.980 %
4B	0.000000 ± 0.000000	0.000067	99.980 %
8B	0.000000 ± 0.000000	0.000056	99.980 %

Metal Q2_0 matches the CPU reference (Mean KLD ~0, top-1 ≥99.9%).

Speeds: llama-bench Metal

./build/bin/llama-bench -m models/Ternary-Bonsai-<size>-Q2_0_g64.gguf -ngl 99 -p 512 -n 128 -r 5

| model           |       size |     params | backend  | ngl |  test |               t/s |
| --------------- | ---------: | ---------: | -------- | --: | ----: | ----------------: |
| qwen3 1.7B Q2_0 | 461.79 MiB |     1.72 B | MTL,BLAS |  99 | pp512 |  2148.81 ±  4.82  |
| qwen3 1.7B Q2_0 | 461.79 MiB |     1.72 B | MTL,BLAS |  99 | tg128 |   246.01 ±  0.57  |
| qwen3 4B Q2_0   |   1.05 GiB |     4.02 B | MTL,BLAS |  99 | pp512 |   856.51 ±  0.74  |
| qwen3 4B Q2_0   |   1.05 GiB |     4.02 B | MTL,BLAS |  99 | tg128 |   124.05 ±  1.44  |
| qwen3 8B Q2_0   |   2.15 GiB |     8.19 B | MTL,BLAS |  99 | pp512 |   457.85 ±  1.45  |
| qwen3 8B Q2_0   |   2.15 GiB |     8.19 B | MTL,BLAS |  99 | tg128 |    77.31 ±  0.37  |

Requirements

• I have read and agree with the contributing guidelines: Yes
• AI usage disclosure: Initial boilerplate code for Metal kernels were generated with AI for Q2_0 group size 64, manually reviewed all changes, and verified correctness using KL-test as above. Have been using the packed models and its working well as expected. Tested again after rebasing with new changes to make sure model is behaving as expected.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.