mlx_lm.server: [metal::malloc] Resource limit (499000) exceeded in BatchGenerator._generate wedges the server invisibly (HTTP layer stays up, generation thread dies) — recurrence of #831/#1185/#1332's signature in continuous batched serving

[athal7]

Context

Same Resource limit (499000) signature as #831 (closed, distributed mlx.launch --backend jaccl serving), #1185 (open, LoRA training), and #1332 (open, DeepSeek-V4 decode — root cause there: un-detached per-decode-step Metal buffer references in the attention/KV cache implementation, "resource_limit is a count of live resident Metal buffers, not bytes", with a fix PR at Blaizzy#25 scoped to that model's cache code). This report is a fourth trigger path: single-node mlx_lm.server continuous batched serving (not distributed, not training, not DeepSeek-V4), and documents an operational consequence none of the three cover — the crash is invisible to standard process supervision.

Also related to my own #1666 (same server, same hardware, different failure mode — kernel panic vs. this uncaught RuntimeError) and to #1493 (silent hang after prompt processing on large requests, also described as "invisible to KeepAlive-style supervision" — same operational blind spot, different trigger).

Environment

• mlx-lm 0.31.3, mlx 0.32.0
• macOS 26.5.2, Apple M4 Pro, 48GB unified memory
• Model: mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit
• mlx_lm.server --decode-concurrency 1 --prompt-concurrency 1 --prompt-cache-size 4 --prompt-cache-bytes 10737418240

Trigger

A client session had been running on a different (non-mlx) backend with a much larger context window, accumulated ~145K tokens of conversation, then fell back to this server (contextWindow: 128000 as configured on the client) after its primary backend hit a transient error. The very next request sent that ~145K-token history to mlx_lm.server.

Crash

Exception in thread Thread-2 (_generate):
Traceback (most recent call last):
  File ".../threading.py", line 1075, in _bootstrap_inner
    self.run()
  File ".../threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File ".../mlx_lm/server.py", line 853, in _generate
    prompt_responses, gen_responses = batch_generator.next()
  File ".../mlx_lm/generate.py", line 1855, in next
    return self._next()
  File ".../mlx_lm/generate.py", line 1775, in _next
    generation_responses = self._generation_batch.next()
  File ".../mlx_lm/generate.py", line 1415, in next
    tokens, logprobs = self._step()
  File ".../mlx_lm/generate.py", line 1369, in _step
    mx.async_eval(self._next_tokens, self._next_logprobs, token_context)
RuntimeError: [metal::malloc] Resource limit (499000) exceeded.

Operational consequence (the actual bug report, beyond the already-documented root cause)

This exception is uncaught in the background _generate thread. The thread dies; the process does not. mlx_lm.server's HTTP layer keeps answering cheap endpoints (GET /v1/models kept returning 200 in <100ms for the ~9 minutes I observed after the crash), so:

• Standard health checks (anything hitting /v1/models or similar metadata endpoint) report healthy.
• launchd's KeepAlive (and presumably systemd's analogous restart-on-exit) never fires, since the process never exits.
• Every subsequent /v1/chat/completions request — from any client, not just the one that triggered the crash — hangs forever under --*-concurrency 1, since there is exactly one generation thread and it is now dead.

Confirmed directly: after the crash, a trivial 5-token completion request against a brand-new conversation hung for a 30s client timeout with zero response, while /v1/models kept responding the whole time.

What would make this actionable

• Given DeepSeek-V4 (Flash/Pro) on Apple Silicon: unbounded Metal residency growth during decode → [metal::malloc] Resource limit (499000) exceeded after ~11K tokens #1332's root-cause analysis, does Qwen3.6's cache implementation in mlx-lm share the same non-detached-per-step-buffer pattern, or is this trigger specific to continuous-batching's per-conversation prompt-cache handling (--prompt-cache-size/--prompt-cache-bytes)? I don't have visibility into Qwen3.6's specific cache code to check either way.
• Independent of root cause: _generate's thread body should catch fatal exceptions and mark the in-flight batch/queue as failed (returning a clean 5xx to waiting HTTP clients) rather than letting the thread die silently forever. This is the same ask as generate() crashes on Metal OOM instead of recovering gracefully #1015 ("generate() crashes on Metal OOM instead of recovering gracefully"), scoped specifically to the server's background thread.
• A /health-style endpoint that actually exercises the generation path (not just static metadata) would let external supervisors detect this without needing to know the exact crash signature in advance.

Mitigation

No server-side fix available yet. Deployed a workaround on my end: a watchdog that tails mlx_lm.server's stderr for this exact RuntimeError signature and force-restarts the process, since no other signal (process exit, health check) can detect the wedge.