Issues with nvfp4 model (tool usage, weird nn and no message reply)

[kuanjames]

When using nvfp4 version of Qwen3.5-122B-A10B, the tool use fails and also sometimes it does not generate a response:

2026-03-19 21:47:33,724 - omlx.server - DEBUG - [-] - Chat completion request received: model=Qwen3.5-122B-A10B-mlx-nvfp4, messages=7, stream=True, max_tokens=None, temp=None
2026-03-19 21:47:33,798 - omlx.server - DEBUG - [-] - Sampling params: temperature=1.0, top_p=0.95, top_k=20, repetition_penalty=1.0, min_p=0.0, presence_penalty=1.5, frequency_penalty=0.0, max_tokens=30000 (model: Qwen3.5-122B-A10B-mlx-nvfp4)
2026-03-19 21:47:33,826 - omlx.cache.prefix_cache - DEBUG - [-] - Cache hit for b236b38c-cb45-45c6-9da0-595f9801d39d: 13 blocks, 26624 tokens
2026-03-19 21:47:33,837 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: ec6e50a4d93648d1...
2026-03-19 21:47:33,837 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: f01dab21a6bff316...
2026-03-19 21:47:33,838 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: 618e56f593164b36...
2026-03-19 21:47:33,839 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: fc505bae58b178aa...
2026-03-19 21:47:33,839 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: 0d79b532bfc04c2e...
2026-03-19 21:47:33,840 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: e04fddd90ef56f43...
2026-03-19 21:47:33,840 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: 27fce56ec6b3bd03...
2026-03-19 21:47:33,841 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: 71e3e9dc16124f47...
2026-03-19 21:47:33,842 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: 932d6b7b76d3719e...
2026-03-19 21:47:33,842 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: cc214b9867f64ee6...
2026-03-19 21:47:33,843 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: 0b02cf4434ac11a6...
2026-03-19 21:47:33,843 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: e12977339cb1ecf0...
2026-03-19 21:47:33,844 - omlx.cache.paged_ssd_cache - DEBUG - [-] - Loaded block with metadata from SSD cache: 48983f26cf0cc778...
2026-03-19 21:47:33,844 - omlx.cache.prefix_cache - DEBUG - [-] - Reconstructed cache from tiered cache: 48 layers, 26624 tokens from 13 blocks
2026-03-19 21:47:33,844 - omlx.scheduler - DEBUG - [-] - Request b236b38c-cb45-45c6-9da0-595f9801d39d: paged cache hit, 26624 tokens in 13 blocks, 1971 tokens remaining, cache reconstructed
2026-03-19 21:47:33,844 - omlx.scheduler - DEBUG - [-] - Added request b236b38c-cb45-45c6-9da0-595f9801d39d with 28595 prompt tokens
2026-03-19 21:47:33,845 - omlx.scheduler - DEBUG - [-] - Scheduled request b236b38c-cb45-45c6-9da0-595f9801d39d (uid=0) with 1971 tokens to process (28595 total), 26624 cached, with cache
2026-03-19 21:47:45,633 - omlx.scheduler - INFO - [-] - Enabled boundary cache snapshots for stateful non-sliceable cache layers
2026-03-19 21:47:45,658 - omlx.scheduler - DEBUG - [-] - Captured boundary cache snapshot for b236b38c-cb45-45c6-9da0-595f9801d39d at 28672 tokens
2026-03-19 21:47:49,527 - omlx.scheduler - DEBUG - [-] - Extracted 48 layer states for request b236b38c-cb45-45c6-9da0-595f9801d39d
2026-03-19 21:47:49,527 - omlx.scheduler - DEBUG - [-] - Request b236b38c-cb45-45c6-9da0-595f9801d39d finished: stop, 239 tokens
2026-03-19 21:47:49,527 - omlx.cache.prefix_cache - DEBUG - [-] - Skipping trailing partial block for b236b38c-cb45-45c6-9da0-595f9801d39d: 1971 token(s) not persisted (block_size=2048, needs +77 token(s) to fill next block)
2026-03-19 21:47:49,527 - omlx.cache.prefix_cache - DEBUG - [-] - Stored cache for b236b38c-cb45-45c6-9da0-595f9801d39d: 13 blocks (0 saved to tiered cache), 26624 tokens
2026-03-19 21:47:49,527 - omlx.scheduler - DEBUG - [-] - Stored paged cache for request b236b38c-cb45-45c6-9da0-595f9801d39d (28595 tokens stored, 28834 total: 28595 prompt + 239 output)
2026-03-19 21:47:49,527 - omlx.scheduler - DEBUG - [-] - Released 13 blocks for eviction (request b236b38c-cb45-45c6-9da0-595f9801d39d)
2026-03-19 21:47:49,527 - omlx.cache.prefix_cache - DEBUG - [-] - Cleared request entry for b236b38c-cb45-45c6-9da0-595f9801d39d (blocks retained)

And if there is a response, it gives weird formating like this:
n * In this clue, 8 is at Position 1 (Wrong) and 4 is at Position 4 (Wrong).n * Since we know 1 is at Position 4, this confirms 4 is not at Position 4.n5. Analyze 1 9 3 7: (Two numbers are correct but in the wrong positions).n * 1 is already identified.

or:

--\n\n## Reference\n\n[1]

[kuanjames]

This is the model link on Huggingface: https://huggingface.co/ayxl/Qwen3.5-122B-A10B-mlx-nvfp4

[jundot]

Thanks for the detailed report and logs.

I looked into this and the root cause is a known limitation in MLX's nvfp4 implementation on the Metal backend (which is what Apple Silicon uses).

The NVIDIA nvfp4 spec uses two-level scaling: a per-block E4M3 scale + a per-tensor FP32 global scale. MLX added tensor-level scale support in PR ml-explore/mlx#3022, but only for CUDA and CPU — Metal still doesn't have it. Without the tensor scale, the effective dynamic range is limited to ~2688x, which isn't enough for a 122B MoE model with 256 experts. When tensor magnitudes exceed that range, scale factors saturate and the logit distribution gets distorted.

This explains all three symptoms:

• Literal \n in output: the model picks escape sequence tokens instead of actual newline tokens due to distorted logits
• Tool calling failures: same distortion causes malformed tool call markup that the parser can't recognize
• Occasional empty responses: logit noise can cause premature EOS generation

Standard mlx 4-bit (affine int4, group_size=64) doesn't have this issue because it uses a different scaling scheme with better dynamic range characteristics.

For now i'd recommend using a standard mlx 4-bit quantized version of the model instead. Once MLX adds Metal support for nvfp4 tensor-level scaling, nvfp4 models should work better on Apple Silicon.

The relevant upstream issue: ml-explore/mlx#2962