Summary
When --moe-slot-bank N equals the model's expert count (256 for Qwen3.6-35B), the server produces degenerate/looping output after multiple sequential server restarts within a macOS session. Setting --moe-slot-bank 255 (one below n_experts) on the same machine with the same model works correctly.
Reproduction
Minimal reproducer (run on a machine that has already started/stopped the 35B server several times in the current session):
# Broken: slot_bank == n_experts
llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.streammoe.gguf \
--moe-sidecar <same> --moe-mode slot-bank --moe-slot-bank 256 \
--moe-slot-path persistent --moe-topk 4 --ctx-size 4096 --no-warmup &
# After ready:
curl .../completion -d '{"prompt":"1+1=","n_predict":4,"temperature":0}'
# Returns: "1+1=" (echoes prompt, no generation)
# Working: slot_bank = n_experts - 1
# Same command with --moe-slot-bank 255
# Returns: "2" (correct)Both --moe-slot-path persistent and --moe-slot-path legacy reproduce the failure identically at sb=256, ruling out corrupted on-disk state.
Behaviour
- sb=256 (broken):
/completion echoes the prompt back with no new tokens; reasoning_content from /v1/chat/completions loops on fragments of the input indefinitely
- sb=255 (working): correct generation, full quality (measured 100% SWE-bench, 100% LiveCodeBench pass@1)
- Not affected:
--moe-coalesced-sync flag — degenerate output occurs with or without it at sb=256
- Session-scoped: at the start of the same session, sb=256 functioned correctly (82 TPS, byte-identical quality). The failure appeared after approximately 15–20 sequential server restarts across different benchmark configurations
Root Cause Hypothesis
When slot_bank >= n_experts, the server enters the "dense GPU mode" (all experts loaded at boot). After many restarts, the Metal command queue or MPS heap for the full-expert allocation reaches a state where the model's logit computation produces near-uniform distributions, causing greedy decoding to loop on the same or similar tokens. The sb=255 path uses the streaming callback code path, which constructs the expert FFN differently and is not affected.
Specific code areas to investigate:
common/streammoe-slot-runtime.cpp: slot bank boot load when slot_bank >= n_expertssrc/llama-graph.cpp: build_moe_ffn GPU vs legacy slot_ids path selectionsrc/llama-context.cpp: coalesced_gpu_path auto-selection condition
Value of the Dense GPU Mode
This is the highest-throughput configuration for the 35B model:
| Mode |
TPS (ctx=16384) |
RSS |
sb=256 GPU coalesced (--moe-coalesced-sync on) |
~66–72 |
~21 GiB |
| sb=256 legacy callback |
~11–17 |
~21 GiB |
| sb=181 streaming |
~27–37 |
~13 GiB |
The GPU coalesced path (Option 3) eliminates 40 per-layer CPU↔GPU syncs by using a single ggml_get_rows kernel over a GPU-resident slot map. This ~4–6× TPS improvement over the legacy path is the primary production throughput configuration for the 35B model.
Fix Priority
High. This is the primary production path for 35B inference. The sb=255 workaround restores quality but loses the GPU coalesced TPS benefit entirely (falls back to streaming callback at ~27 TPS vs ~72 TPS). A Metal context reset between server restarts, or explicit cleanup of the slot-bank Metal buffers on server shutdown, would likely fix this.
Environment
- Platform: macOS 25.5.0, Apple Silicon (M-series, 48 GB unified memory)
- Branch:
streammoe/moe-engine
- Model:
Qwen3.6-35B-A3B-UD-Q4_K_XL.streammoe.gguf (unified format, E29)
- Confirmed not a fresh-boot issue — sb=256 works correctly on first start of a session
Summary
When
--moe-slot-bank Nequals the model's expert count (256 for Qwen3.6-35B), the server produces degenerate/looping output after multiple sequential server restarts within a macOS session. Setting--moe-slot-bank 255(one below n_experts) on the same machine with the same model works correctly.Reproduction
Minimal reproducer (run on a machine that has already started/stopped the 35B server several times in the current session):
Both
--moe-slot-path persistentand--moe-slot-path legacyreproduce the failure identically at sb=256, ruling out corrupted on-disk state.Behaviour
/completionechoes the prompt back with no new tokens;reasoning_contentfrom/v1/chat/completionsloops on fragments of the input indefinitely--moe-coalesced-syncflag — degenerate output occurs with or without it at sb=256Root Cause Hypothesis
When
slot_bank >= n_experts, the server enters the "dense GPU mode" (all experts loaded at boot). After many restarts, the Metal command queue or MPS heap for the full-expert allocation reaches a state where the model's logit computation produces near-uniform distributions, causing greedy decoding to loop on the same or similar tokens. The sb=255 path uses the streaming callback code path, which constructs the expert FFN differently and is not affected.Specific code areas to investigate:
common/streammoe-slot-runtime.cpp: slot bank boot load whenslot_bank >= n_expertssrc/llama-graph.cpp:build_moe_ffnGPU vs legacyslot_idspath selectionsrc/llama-context.cpp:coalesced_gpu_pathauto-selection conditionValue of the Dense GPU Mode
This is the highest-throughput configuration for the 35B model:
--moe-coalesced-sync on)The GPU coalesced path (Option 3) eliminates 40 per-layer CPU↔GPU syncs by using a single
ggml_get_rowskernel over a GPU-resident slot map. This ~4–6× TPS improvement over the legacy path is the primary production throughput configuration for the 35B model.Fix Priority
High. This is the primary production path for 35B inference. The sb=255 workaround restores quality but loses the GPU coalesced TPS benefit entirely (falls back to streaming callback at ~27 TPS vs ~72 TPS). A Metal context reset between server restarts, or explicit cleanup of the slot-bank Metal buffers on server shutdown, would likely fix this.
Environment
streammoe/moe-engineQwen3.6-35B-A3B-UD-Q4_K_XL.streammoe.gguf(unified format, E29)