1. Expert Double-Buffered(TPS x2.3) 2. Remove KV cache 3. Support Local Model Path

[ccckblaze]

Compared to main branch with the following improvements:

1. Expert Double-Buffered

• Implemented double buffering mechanism (Buffer A/B) in PredictiveExpertCache
• Supports async expert prefetching (prefetch_async) and buffer swapping (swap_buffers)
• Prefetches next-layer experts to optimize inference throughput

TPS from ~ 12-15 tok/s to ~28 tok/s(up to 2.3x) on my M4 mac

notice: 8-bit model not 4-bit

2. Removed KV Cache

• Removed server-side KV cache mechanism (kv_cache_slots parameter)
• Reason: KV cache can cause context confusion in certain conditions
• Related changes:
  • cli.py: Removed --kv-cache-slots parameter
  • server.py: Removed KV cache related code (_put_kv_cache, _kv_cache dict)

3. Support Local Model Path

• Added local path support in generate.py, allowing direct model directory paths
• Supports loading models from local filesystem without requiring HuggingFace IDs

4. Other Improvements

• pyproject.toml: Lowered Python version requirement to 3.12, upgraded mlx/mlx-lm version dependencies
• Cache path fix: Properly handles cache directory naming for local paths
• Dynamic prefetch: Added logic in dynamic_cache_update to predict next-layer experts based on current layer requests