How to do disk offloading?

[TomLucidor]

Assuming that a whole model can fit on memory, BUT that the KV cache generated is likely not able to fit along side it, is it possible to use the disk as a kind of backup to help add capacity?

[rajanshxrma]

MLX doesn't have a live disk-backed KV cache today, and the reason is architectural rather than a missing feature: on Apple Silicon, CPU and GPU share unified memory, so there's no separate "host RAM" vs "device RAM" boundary the way CUDA-based frameworks have — which is normally what a disk-offload tier gets built underneath.

What mlx-lm does give you for KV-cache memory pressure:

1. --max-kv-size N — uses a RotatingKVCache that caps the cache to a fixed window of N tokens (evicting the oldest ones beyond it), so memory stops growing instead of blowing up on long contexts. Quality trades off on very long sequences since old tokens fall out of context.

2. --kv-bits N (with --kv-group-size, --quantized-kv-start) — quantizes the KV cache in place (e.g. down to 4 or 8 bits via QuantizedKVCache), shrinking its memory footprint several-fold. This is usually the better lever if you want to keep the full context rather than rotate it out.

3. save_prompt_cache() / load_prompt_cache() (mlx_lm.cache_prompt, or --prompt-cache-file on mlx_lm.generate/server) — persists a computed KV cache to a .safetensors file on disk so you don't have to recompute a big system prompt's cache across separate runs. That is disk-backed, but only between sessions — you still have to load the whole thing back into RAM to resume, it's not a live overflow mechanism mid-generation.

Combining --max-kv-size and --kv-bits is the practical way to fit a KV cache that would otherwise not fit alongside the model.