KV cache to disk not working

[m5kro]

Hi!

Thanks for making ds4 more usable on the Spark! But I've noticed that KV caching to disk isn't working. Nothing is getting written to disk, and it's causing a lot of prompt reprocessing.

The upstream version is saving to disk, but the overall speed just isn't comparable.

Thanks

[fabiopili]

I ran into this too and tracked down a cause. Short version: the empty disk cache and the constant prompt reprocessing are the same bug, and it only affects thinking mode.

What I see

On a DGX Spark (GB10) with --kv-disk-dir configured at a 64 GiB budget, a coding agent with a 29,234 token preamblere-prefilled its entire prompt on every single turn:

cont admit bank=1 pos_base=0 suffix=29234 wall=41.44s
cont admit bank=1 pos_base=0 suffix=29308 wall=40.57s
cont admit bank=1 pos_base=0 suffix=31022 wall=43.10s

pos_base=0 every time means nothing was reused, even though the conversation only grew by a few tokens per turn. Over a full day of traffic the disk cache directory stayed completely empty, across four clean shutdowns.

The same client, same prompts, same box, pointed at the non-thinking deepseek-chat alias instead:

partial fork admit src=0 dst=1 cut=29123 suffix=145
cont admit bank=1 pos_base=29056 suffix=212 wall=0.80s

40.86s versus 0.80s for turn two, with only the thinking mode differing. And on that run the banks did get written to disk (145 MiB and 146 MiB at shutdown).

So if you are seeing "nothing written to disk", check whether your client is using thinking mode. That is very likely the trigger.

Why they are the same bug

cont_warm_retire() in ds4_server.c returned early for every thinking row, so no warm record was ever built for the bank. Two consequences follow from that one thing:

1. cont_warm_pick() has no record to match, so every turn is a cold admit.
2. A bank with no record never becomes reuse-trustworthy, so it is never written to the disk cache, not even on a clean shutdown.

That is why it looks like disk caching is broken when the disk tier itself is fine. I confirmed the tier works by watching a non-thinking bank persist normally while a thinking bank in the neighbouring slot was silently dropped at the same shutdown.

Worth noting for anyone bisecting: this is not a recent regression. The early return is present from v0.1.0 onward. The session path already handles this correctly with visible-live checkpoints; the continuous batching path, added later, never inherited that mechanism.

Fix

I have a patch that wires the existing mechanism into the continuous path:
Entrpi/ds4#4

It keys the warm record on the transcript the client will actually replay, while the payload stays the bank's committed frontier, so the hidden reasoning stays in KV and is never re-prefilled. make test passes in full and all three previously broken thinking shapes (no tools, with tools, reasoning echoed back) now reuse.

Workaround until then

Point the client at the non-thinking model id (deepseek-chat, or thinking={"type":"disabled"} / think=false). Caching works normally there. It costs you thinking, which may or may not be an acceptable trade for your workload, but for prefill-heavy agent sessions the difference is large.

[Entrpi]

Fixed in ds4 v0.4.2, and the root cause turned out to be upstream of the
disk tier entirely. Thinking-mode conversations never built warm records
on the continuous batching path, and a bank without a warm record is
never considered reuse-trustworthy, so it was never written to the disk
cache. Disk KV itself was working the whole time; thinking traffic just
never produced anything eligible to store.

The fix came in as a community contribution:
Entrpi/ds4#4 by @fabiopili, who diagnosed both
symptoms (every thinking turn cold re-prefills, and the empty disk dir
you hit) back to that single cause. I verified it with a behavioral gate
on a DGX Spark: after a clean stop, thinking banks now persist (12 bank
persist lines, 6 files in the disk dir, previously always empty), and
after a reboot the next turn restores from disk and answers correctly
(bank restore hit in 66.9 ms vs a full re-prefill).

To pick it up: DS4_REF=v0.4.2 ./install.sh (or git pull + rebuild if
you already have the fork checked out). Thanks for the report, this one
pointed at a real hole in the serving path.