Session bank silently stops caching above ~35k tokens, and three settings paths fail without a message

[Cyb3rb1ade]

Summary

On a 48 GB Mac, MTPLXApp's auto session-cache sizing yields a 3.00 GiB
per-session budget. A 40 540-token KV snapshot measures 3.92 GiB, so
session_bank.py:625 drops it as skipped_oversized_snapshot. Nothing is
stored, so every following turn is a full re-prefill: 130 s TTFT instead of
3.2 s, on every turn, with no error, no warning, and no indication in the UI.

Trying to fix it by hand hits three further silent failures, described below.
Each of them looks like success.

Impact, measured

Same session, ~43 k tokens of context, before and after raising the budget:

	before	after
cached_tokens on follow-up	0	43 101
new_prefill_tokens	43 103	166
cache_source	none	ram
TTFT	126.4 s	3.24 s

Over a 100-request agent trail: 316 012 tokens re-prefilled, 581.9 s of pure
prefill stall against 3 244.5 s of decode.

/health evidence before the fix:

"recent_evictions": [
  { "reason": "skipped_oversized_snapshot",
    "session_id": "anon-b1a6f1dd6a4f45e6",
    "prefix_len": 40540,
    "nbytes": 4208962560,
    "budget": 3221225472 }
]

The SSD tier is dead too — writes_enqueued: 0, restore_hits: 0,
restore_misses: 21 — because nothing ever reaches it from the RAM tier.

Root cause

ram_session_cache_*_size = "auto" resolves to 3 GiB per session on this
machine. The heuristic budgets a share of post-model RAM but does not scale
with KV footprint × context length, so it under-provisions exactly at long
context. Below ~35 k tokens the cache works; above it, it stops, silently.

Three silent failure paths when correcting it by hand

1. The allocation policy discards the explicit sizes.
apps/MTPLXApp/Sources/MTPLXAppCore/Services/MTPLXCommandBuilder.swift:938:

guard configuration.ramSessionCachePolicy != "target-default" else {
    return [:]
}

While the policy is target-default (the default),
ram_session_cache_per_session_max_size, ..._max_size and ..._max_entries
are dropped entirely when building the environment. Setting all three correctly
in settings.json changes nothing and produces no diagnostic. The UI hides the
size fields in that state, so this only bites users editing the file — but it
bites them invisibly.

2. The byte parser accepts only K/M/G/T and falls back without a word.
mtplx/engine_session.py → _bank_bytes_from_env, measured against the
installed build:

input	result
8GB	default (discarded)
16GB	default (discarded)
8G	8 589 934 592
16G	17 179 869 184
8589934592	8 589 934 592

This was the hardest of the three to catch: "8GB" fell back to
DEFAULT_PER_SESSION_MAX_BYTES, which happens to be exactly 8 GiB — the very
value intended. /health then reported “8.00 GiB” and looked like a success.
It only surfaced because max_bytes simultaneously reported 24 GiB instead of
the 16 GiB set — the other default. With one setting instead of two, the
mistake would have gone unnoticed.

3. paged_kv_quantization cannot be turned off from settings.
"paged_kv_quantization": "off" is a valid mode per /health
(kv_quant_policy.modes: ["off","q8","q4"]), but the daemon still reports q8
after a restart — the model-family defaults in MTPLXCommandBuilder appear to
overwrite the user value.

Reproduction

1. On a 48 GB Mac, launch via MTPLXApp with ram_session_cache_policy at its
   default and a dense 27B model.
2. Send a ~43 000-token request, then a short follow-up in the same session.
3. Observe cache_miss_reason: ssd_prefix_miss on the follow-up and
   skipped_oversized_snapshot in /health → session_bank.recent_evictions.
4. Set ram_session_cache_per_session_max_size to 8G and
   ram_session_cache_policy to bounded, restart, repeat: the follow-up now
   restores from ram.

mtplx doctor --json

All diagnostic checks pass (os.macos_version, python.native_arm64,
python.version and the rest report status: pass). The condition is invisible
to doctor, which is part of the problem — nothing in the supported diagnostic
surface reports that the session cache has stopped storing.

Exact command

Launched by MTPLXApp:

python -m mtplx.server.openai \
  --model ~/.mtplx/models/Youssofal--Qwen3.6-27B-MTPLX-Optimized-Speed \
  --backend-id qwen3_next --host 127.0.0.1 --port 18085 \
  --depth 3 --generation-mode mtp --profile sustained \
  --verify-strategy capture_commit --verify-core linear-gdn-from-conv-tape \
  --paged-kv-quantization q8 --ssd-session-cache on \
  --adaptive-policy expected_value

with MTPLX_SESSION_BANK_PER_SESSION_BYTES=3G,
MTPLX_SESSION_BANK_MAX_BYTES=8G, MTPLX_SESSION_BANK_MAX_ENTRIES=6.

Model path or repo id

Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed

Chip, RAM, macOS version

Apple M5 Pro · 48 GB unified memory · macOS 27.0 (26A5388g) · MTPLX 2.4.2

Suggested fixes

1. Report the drop. skipped_oversized_snapshot should surface at least
   once per session in the UI or as a server warning. Today it is only visible
   to someone who thinks to read recent_evictions.
2. Size the budget from KV footprint × context window, not from a flat
   share of post-model RAM.
3. Fail loudly on unparseable sizes. _bank_bytes_from_env should log the
   rejected value rather than returning the default, since the default can
   coincide with the intended value.
4. Do not silently discard explicit settings. Either honour the size fields
   regardless of policy, or reject the combination visibly.

Side note

scripts/session_cache_followup_qa.py sends no Authorization header and dies
with HTTP 401 against a key-protected daemon. A local patch adding
--api-key (with an $MTPLX_API_KEY fallback, redacted from the JSONL trail)
is available if useful.