fix(dsh): make extraction drain resilient to LLM stalls and long batches

[evilh2019]

Problem

The DSH adapter's extractPending loop (in dsh.ts) drains unextracted session messages into the knowledge graph in fixed-size batches:

const messages = getUnextracted(db, sid, 50);

with no length bound on the batch, an unbounded existingNames hint list (getBySession(db, sid).map(n => n.name) — can reach tens of thousands of chars), and a complete() LLM call that had no stream timeout at all.

On any error the whole drain stops and the batch is deliberately left unextracted ("so a later turn/restart can retry"). In practice that means:

1. A burst of long tool results builds a request that exceeds the model context / stalls the stream.
2. The stalled stream never emits a finish/error chunk → extractPending hangs (no timeout), or with a provider-side timeout the batch fails.
3. The failing batch is retried identically on every restart → the backlog is pinned forever; extraction effectively never completes and the DB keeps every raw message with extracted=0.

Observed on a real deployment: a 47K-message session with a backlog stuck at extracted=0 across many restarts; batches of ~18.7K chars timed out while an ~18.6K batch succeeded, confirming the failure is content/stall-driven, not a simple fixed threshold.

Fix

Bounded, resilient drain with guaranteed progress:

• Size-bounded batches: pull the oldest unextracted messages in order, normalized, capped by accumulated characters (8K) then count (15). A single long message always fits, so the drain always progresses; order is preserved so the markExtracted(upToTurn) max-turn marking stays correct.
• Bounded existingNames: cap at 150 entries / 3K chars so the hint list cannot blow the prompt.
• Stream timeout: hard-bound complete() at 180s (previously unbounded — a stall could pin the session's extraction chain forever). AbortController-style race with timer cleanup.
• Retry → bisect → skip ladder (drainBatch):
  • retry transient failures with backoff (5s then 15s),
  • if still failing, bisect the batch so one poison message cannot sink the rest,
  • last resort: mark a lone failing message extracted (raw message stays in gm_messages) so the drain can never deadlock.

Validation

• npm test → 129 tests pass (one pre-existing file-level setup failure on node:sqlite typings, unrelated to this change).
• Real deployment: the previously-stuck backlog (never completing across restarts) now drains to zero with visible retry → split → DONE sequences in logs; node/edge counts grow steadily.

Small, self-contained change: dsh.ts + rebuilt dist/dsh.js (the repo tracks dist/).

[evilh2019]

Follow-up commit adds three adapter tests (test/dsh-adapter.test.ts) that drive the real backfill → scheduleExtract → drainBatch path against a file-backed store:

• transient LLM failure → retried → backlog drains to zero (no SKIP)
• permanently failing batch → bisect → each singleton marked extracted → drain still progresses (no deadlock), verified via gm_messages extracted=0 count
• oversized message → batch split by accumulated normalized length; every extraction request stays under the size cap

Also exposed / as adapter config so hosts can tune the resilience knobs.

Full suite: 132 tests passed.

[adoresever]

Thank you @evilh2019 and @penggaolai for finding the extraction-backlog stall and contributing the initial bounded batching, retry, timeout and bisect work.

Maintainer integration #97 preserved both of your commits and their original authorship, then addressed the review blockers found with official DSH and real model testing: lossless single-message segmentation, durable quarantine instead of false extracted flags, exact acknowledgements, cancellation, retry tooling, retention safety, optional Headless services, and the awaited agent/turn-stopping lifecycle drain.

#97 has passed Node 22/24 CI and is merged. I am closing this PR as superseded by that integration, not as a rejection of the contribution. Thanks again.