Use a 32-row block in qmm_t_nax when one block covers all of M - #4171
Open
dwijenpatel wants to merge 1 commit into
Open
Use a 32-row block in qmm_t_nax when one block covers all of M#4171dwijenpatel wants to merge 1 commit into
dwijenpatel wants to merge 1 commit into
Conversation
The plain quantized matmul reaches this kernel at small M only for wide weight matrices: below the qmv batch limit the vector kernel serves the product, and up to 256 column tiles the split-K path does. What arrives here with M <= 32 is therefore vocabulary heads and wide MLP projections: speculative-decode verification, batch serving, and short prompts. At those sizes the fixed 64-row block computes rows that do not exist, and on wide matrices that arithmetic is the bottleneck. Unlike gather_qmm_rhs_nax, a shorter block here is not free: every row block reads all the weight columns it touches, so shrinking BM only pays when it does not add row blocks. The dispatch uses BM=32 exactly when one block still covers M, and a same-binary sweep confirms the boundary: at M=128 forcing BM=32 is 13% slower (529 us to 608), while inside the window it wins with no case worse than parity. Measured on a base M5 (10-core GPU), fp16, 4-bit gs64, alternating configurations in one session, fresh process per cell: M K N BM=64 BM=32 14 5120 13824 996.8 us 835.8 us 1.19x 32 5120 13824 975.4 us 785.0 us 1.24x 48 5120 13824 (same path in both builds) A 16-row block was also measured and dropped: 6% over BM=32 at M=14 on the one shape where it showed at all, not worth its instantiations. The fp_quantized_nax.metal instantiation macros dropped their tile-size arguments, so every fp tile landed on the template defaults and a new tile would have compiled to a mis-labeled 64-row kernel. They now forward bm/bk/bn/wm/wn; no behavior change for existing instantiations, which all pass the defaults. The new instantiations grow the metallib by 4.01%. Half of that is the batch_1 and alN_false variants; dropping them in exchange for a dispatch guard is a straightforward trim if the size matters more. test_qmm_small_m_block covers both block heights, the unaligned-N and batched variants, and all four quantization families, at shapes verified by dispatch logging to actually reach them; the existing test_qmm shapes cannot, because at N <= 256 the qmv limit exceeds every M below 33. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A small follow-up to #3925 and #4023, which tuned the block size for
gather_qmm_rhs_nax: the same question applied to the plain quantizedmatmul.
qmm_t_naxalways runs 64-row blocks; for M <= 32 we measuredmodest gains from a 32-row block.
Only wide matrices reach this kernel at small M: below the qmv batch
limit the vector kernel serves the product, and up to 256 column tiles
the split-K path does, so N > 8192 is what arrives here with M <= 32.
In practice that is vocabulary heads and wide MLP projections during
speculative-decode verification, batch serving, and short prompts.
Unlike the gather case, a shorter block here is not free: every row
block reads all the weight columns it touches, so BM=32 pays only while
one block still covers all of M (forcing it at M=128 measures 13%
slower). The dispatch shrinks the block exactly when it stays a single
block.
Measured on a base M5 (10-core GPU), fp16, 4-bit gs64, against merged
main (5ec30ac), both arms built from one tree, alternating in one
session, fresh process per cell, three passes with the last recorded
(drift on the repeated first cell: 0.7%):
BM=16 was also measured and dropped: 6% over BM=32 at M=14 on the one
shape where it showed, not worth its instantiations.
Two notes for review:
The
fp_quantized_nax.metalinstantiation macros did not forwardtheir tile-size arguments into the template, so every fp tile landed
on the defaults; a new tile would have compiled to a mis-labeled
64-row kernel. They now forward bm/bk/bn/wm/wn. No behavior change
for existing instantiations, which all pass the defaults.
The new instantiations grow the metallib by 4.02%; half of that is
the batch_1 and alN_false variants, which I can drop behind a
dispatch guard if size matters more.
Tests: the existing test_qmm_large_dims shapes (16 and 33 rows at
N=32840) already land on either side of the new dispatch, so one added
row pins the boundary itself (M=32). A small new test covers the two
variants no existing shape reaches, batched and fp-mode; the fp row is
what exercises the macro fix.
Repro: