Skip to content

Use a 32-row block in qmm_t_nax when one block covers all of M - #4171

Open
dwijenpatel wants to merge 1 commit into
ml-explore:mainfrom
dwijenpatel:tune-qmm-nax-small-m
Open

Use a 32-row block in qmm_t_nax when one block covers all of M#4171
dwijenpatel wants to merge 1 commit into
ml-explore:mainfrom
dwijenpatel:tune-qmm-nax-small-m

Conversation

@dwijenpatel

Copy link
Copy Markdown
Contributor

A small follow-up to #3925 and #4023, which tuned the block size for
gather_qmm_rhs_nax: the same question applied to the plain quantized
matmul. qmm_t_nax always runs 64-row blocks; for M <= 32 we measured
modest gains from a 32-row block.

Only wide matrices reach this kernel at small M: below the qmv batch
limit the vector kernel serves the product, and up to 256 column tiles
the split-K path does, so N > 8192 is what arrives here with M <= 32.
In practice that is vocabulary heads and wide MLP projections during
speculative-decode verification, batch serving, and short prompts.

Unlike the gather case, a shorter block here is not free: every row
block reads all the weight columns it touches, so BM=32 pays only while
one block still covers all of M (forcing it at M=128 measures 13%
slower). The dispatch shrinks the block exactly when it stays a single
block.

Measured on a base M5 (10-core GPU), fp16, 4-bit gs64, against merged
main (5ec30ac), both arms built from one tree, alternating in one
session, fresh process per cell, three passes with the last recorded
(drift on the repeated first cell: 0.7%):

M K N main this branch
14 5120 13824 1011.0 us 833.1 us 1.21x
32 5120 13824 980.7 us 804.2 us 1.22x
48 5120 13824 same path in both builds

BM=16 was also measured and dropped: 6% over BM=32 at M=14 on the one
shape where it showed, not worth its instantiations.

Two notes for review:

  1. The fp_quantized_nax.metal instantiation macros did not forward
    their tile-size arguments into the template, so every fp tile landed
    on the defaults; a new tile would have compiled to a mis-labeled
    64-row kernel. They now forward bm/bk/bn/wm/wn. No behavior change
    for existing instantiations, which all pass the defaults.

  2. The new instantiations grow the metallib by 4.02%; half of that is
    the batch_1 and alN_false variants, which I can drop behind a
    dispatch guard if size matters more.

Tests: the existing test_qmm_large_dims shapes (16 and 33 rows at
N=32840) already land on either side of the new dispatch, so one added
row pins the boundary itself (M=32). A small new test covers the two
variants no existing shape reaches, batched and fp-mode; the fp row is
what exercises the macro fix.

Repro:

python -c "
import time
import mlx.core as mx
M, K, N = 32, 5120, 13824
x = (mx.random.normal((M, K)) / K**0.5).astype(mx.float16)
w = (mx.random.normal((N, K)) / K**0.5).astype(mx.float16)
wq = mx.quantize(w, group_size=64, bits=4)
def f(): mx.eval(mx.quantized_matmul(x, *wq, transpose=True, group_size=64, bits=4))
for _ in range(30): f()
t0 = time.perf_counter()
for _ in range(300): f()
print(f'{(time.perf_counter()-t0)/300*1e6:.1f} us per matmul')"

The plain quantized matmul reaches this kernel at small M only for wide
weight matrices: below the qmv batch limit the vector kernel serves the
product, and up to 256 column tiles the split-K path does. What arrives
here with M <= 32 is therefore vocabulary heads and wide MLP projections:
speculative-decode verification, batch serving, and short prompts. At
those sizes the fixed 64-row block computes rows that do not exist, and
on wide matrices that arithmetic is the bottleneck.

Unlike gather_qmm_rhs_nax, a shorter block here is not free: every row
block reads all the weight columns it touches, so shrinking BM only pays
when it does not add row blocks. The dispatch uses BM=32 exactly when one
block still covers M, and a same-binary sweep confirms the boundary: at
M=128 forcing BM=32 is 13% slower (529 us to 608), while inside the
window it wins with no case worse than parity.

Measured on a base M5 (10-core GPU), fp16, 4-bit gs64, alternating
configurations in one session, fresh process per cell:

  M    K      N       BM=64      BM=32
  14   5120   13824   996.8 us   835.8 us   1.19x
  32   5120   13824   975.4 us   785.0 us   1.24x
  48   5120   13824   (same path in both builds)

A 16-row block was also measured and dropped: 6% over BM=32 at M=14 on
the one shape where it showed at all, not worth its instantiations.

The fp_quantized_nax.metal instantiation macros dropped their tile-size
arguments, so every fp tile landed on the template defaults and a new
tile would have compiled to a mis-labeled 64-row kernel. They now forward
bm/bk/bn/wm/wn; no behavior change for existing instantiations, which all
pass the defaults.

The new instantiations grow the metallib by 4.01%. Half of that is the
batch_1 and alN_false variants; dropping them in exchange for a dispatch
guard is a straightforward trim if the size matters more.

test_qmm_small_m_block covers both block heights, the unaligned-N and
batched variants, and all four quantization families, at shapes verified
by dispatch logging to actually reach them; the existing test_qmm shapes
cannot, because at N <= 256 the qmv limit exceeds every M below 33.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@zcbenz zcbenz added the await verification This pull request is non-trivial and requires a human expert to verify its correctness. label Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

await verification This pull request is non-trivial and requires a human expert to verify its correctness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants