Skip to content

Swizzle scales - #2979

Merged
awni merged 9 commits into
ml-explore:mainfrom
nastya236:swizzle-scales
Jan 10, 2026
Merged

Swizzle scales#2979
awni merged 9 commits into
ml-explore:mainfrom
nastya236:swizzle-scales

Conversation

@nastya236

Copy link
Copy Markdown
Collaborator

Improvements to scales swizzling kernel.

Overview:

  • Each thread loads 4 rows × 4 bytes = 16 bytes with stride 32, so a warp covers one full 128×4 tile.
  • If K % 4 == 0: vectorized int32 loads
  • If not: byte-by-byte loads with zero-padding ( reduction dim is typically divisible by group_size x 4, so we won't be there often).
  • load: global-> registers -> shared
  • coalesced store: shared -> global
  • XOR swizzle to avoid bank conflicts

Some numbers for qqmm with new swizzling (variance is <0.5%):

M N K Improvement (%)
16384 4096 11008 10.19
32768 4096 11008 9.40
65536 4096 11008 9.41
16384 11008 4096 13.54
32768 11008 4096 9.47
65536 11008 4096 13.79
16384 4096 12288 5.59
32768 4096 12288 5.06
65536 4096 12288 6.44
16384 12288 4096 13.48
32768 12288 4096 11.86
65536 12288 4096 15.02
16384 5120 27648 7.59
32768 5120 27648 5.12
65536 5120 27648 6.09
16384 27648 5120 13.18
32768 27648 5120 12.42
65536 27648 5120 11.34

Probably can be improved further!

Comment thread mlx/backend/cuda/quantized/qqmm_utils.cu Outdated
Comment thread mlx/backend/cuda/quantized/qqmm_utils.cu Outdated
Comment thread mlx/backend/cuda/quantized/qqmm_utils.cu Outdated

@awni awni left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Left a couple more nits. Please take a look then we can merge.

As a side-note I'd like to find a way to test the QQ ops in CI 🤔 but I'm drawing blanks. It might at some point make sense to implement slow fallbacks for the sm < 10.0, since we just need the matmuls. Otherwise it could be a while before CI has blackwell available.

@nastya236

nastya236 commented Jan 10, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the review! Regarding tests, this is a very good point. I tested it with examples/python/qqmm.py on B200. I’ll implement a fallback so we can test correctness

@nastya236 nastya236 mentioned this pull request Jan 10, 2026
4 tasks
@awni
awni merged commit 43341e8 into ml-explore:main Jan 10, 2026
15 checks passed
jzdziarski pushed a commit to jzdziarski/mlx that referenced this pull request Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants