Summary
Tracking issue for 1-bit affine weight support, needed to serve prism-ml/Bonsai-27B-mlx-1bit (binary g128, dense Qwen3_5ForConditionalGeneration). Currently blocked upstream — no released MLX has 1-bit affine kernels.
Current behavior
rmlx info parses the config (affine bits=1 g128), and the model graph loads (~2 s). Generation produces zero tokens:
WARN prefill: exit_prefill quantization failed
error=mlx: Array::eval: [metal::Device] Unable to load kernel affine_dequantize_bfloat16_t_gs_128_b_1
# → HTTP 503 "generation produced zero tokens"
This is a weight dequant failure (KV-mode-independent — reproduces with --kv-quant none): stock MLX has no bits=1 affine dequant Metal kernel.
Upstream status (verified 2026-07)
- Bare MLX 0.31.2 and 0.32.0 (latest release) both reject
bits=1 at mx.quantize: "supported bits are 2, 3, 4, 5, 6, 8." Apple's own mlx_lm generate fails identically.
- 1-bit affine support lives in ml-explore/mlx#3161 ("Add 1-bit affine quantization, Metal+CPU") — open / unmerged / unreleased. Known edge cases:
qmv_quad_impl inner-dim < 128, group_size=32 occupancy; no CUDA.
The good news — packing is standard
The 1-bit safetensors uses standard MLX affine bits=1 layout (weight U32 = 32 weights/u32, scales+biases F16 group 128, __metadata__: {format: mlx}, bit0→−s_g / bit1→+s_g). That is exactly what mlx#3161's kernel consumes — no custom rMLX packing needed.
Recommended path (do NOT build a custom MSL kernel)
rMLX delegates all weight quant to mlx-c (ops/matmul.rs → mlx_dequantize / mlx_quantized_matmul); it owns no custom weight-dequant MSL kernel (only KV codecs). A custom 1-bit fused quantized_matmul MSL kernel would duplicate mlx#3161 and add a maintenance burden.
Preferred: wait for mlx#3161 to merge + ship in a release → brew upgrade mlx mlx-c → rebuild rMLX. The standard packing means it should then work with zero rMLX code.
ROI note: 1-bit is the phone / tight-memory operating point (~3.9 GB, fits iPhone 17 Pro Max). On a 128 GB Apple Silicon dev box the 2-bit ternary build is higher quality (94.6% vs 89.5% of FP16) at similar speed — so low urgency.
Acceptance
Summary
Tracking issue for 1-bit affine weight support, needed to serve
prism-ml/Bonsai-27B-mlx-1bit(binary g128, denseQwen3_5ForConditionalGeneration). Currently blocked upstream — no released MLX has 1-bit affine kernels.Current behavior
rmlx infoparses the config (affine bits=1 g128), and the model graph loads (~2 s). Generation produces zero tokens:This is a weight dequant failure (KV-mode-independent — reproduces with
--kv-quant none): stock MLX has nobits=1affine dequant Metal kernel.Upstream status (verified 2026-07)
bits=1atmx.quantize: "supported bits are 2, 3, 4, 5, 6, 8." Apple's ownmlx_lm generatefails identically.qmv_quad_implinner-dim < 128, group_size=32 occupancy; no CUDA.The good news — packing is standard
The 1-bit safetensors uses standard MLX affine bits=1 layout (
weightU32 = 32 weights/u32,scales+biasesF16 group 128,__metadata__: {format: mlx}, bit0→−s_g / bit1→+s_g). That is exactly what mlx#3161's kernel consumes — no custom rMLX packing needed.Recommended path (do NOT build a custom MSL kernel)
rMLX delegates all weight quant to mlx-c (
ops/matmul.rs→mlx_dequantize/mlx_quantized_matmul); it owns no custom weight-dequant MSL kernel (only KV codecs). A custom 1-bit fusedquantized_matmulMSL kernel would duplicate mlx#3161 and add a maintenance burden.Preferred: wait for mlx#3161 to merge + ship in a release →
brew upgrade mlx mlx-c→ rebuild rMLX. The standard packing means it should then work with zero rMLX code.ROI note: 1-bit is the phone / tight-memory operating point (~3.9 GB, fits iPhone 17 Pro Max). On a 128 GB Apple Silicon dev box the 2-bit ternary build is higher quality (94.6% vs 89.5% of FP16) at similar speed — so low urgency.
Acceptance
brew upgrade+ rebuild →Bonsai-27B-mlx-1bitserves coherent text