You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I understand this issue needs status:approved before a PR can be opened
📝 Bug Description
This is a follow-up on the architectural decision in #271 (fixed in v1.14.6). That fix correctly enforced db.SetMaxOpenConns(1) so the busy_timeout=5000 pragma always applies to every connection in the pool. The single-connection decision is sound for SQLite single-writer correctness.
However, the decision has an undocumented trade-off that amplifies the severity of #477 (stale WAL-index mmap in long-running processes): with a single connection, there is zero concurrency headroom. When the WAL-index mmap goes stale (per #477), the single connection is the ONLY connection, so every read AND write operation fails or hangs. There is no secondary path, and no code path triggers a reopen.
Concretely: /health continues to respond (trivial handler, no DB access), while every DB-backed endpoint (/search, /observations, writes) silently fails. The process cannot self-recover; it requires an external kill + restart.
This is not a duplicate of #477 or #206. It is the architectural gap between them:
Observe: /health returns ok, mem_doctor reports 4/4 checks passing (including checkpoint_log: 890), but /search hangs and returns SQLITE_BUSY after busy_timeout=5000.
Confirm via lsof -p $(pgrep -f 'engram serve') | grep db-shm that the daemon holds a stale inode versus a fresh process.
The daemon cannot recover without a manual kill + restart.
Reproduced on Linux (Ubuntu via Homebrew), Engram 1.16.0, single daemon, ~1.7 days uptime.
✅ Expected Behavior
A single-connection pool should be paired with a mechanism that detects WAL-index staleness and reopens the connection transparently (fsnotify on -shm/-wal, or inode verification before each transaction), as proposed in #206. With that companion, #271's single-conn decision becomes both correct AND resilient.
Additionally, /health should distinguish "process alive" from "DB reachable", so monitoring can detect the silent divergence instead of relying on mem_doctor (which checks store integrity, not handler liveness).
❌ Actual Behavior
The single connection becomes a single point of failure when the WAL-index goes stale.
/health reports ok while DB-backed operations fail.
No code path triggers a connection reopen; recovery requires external intervention (process restart).
🖥️ Environment
Operating System: Linux (Ubuntu/Debian)
Engram Version: 1.16.0
Agent / Client: Other (Pi via gentle-engram plugin)
📋 Relevant Logs
# mem_search failure surface (gentle-engram side):
gentle-engram could not reach the Engram HTTP server at http://127.0.0.1:7437.
The Pi-native mem_* tools are registered, but the native memory provider is not currently responding.
# Yet the trivial endpoint still responds:
$ curl http://127.0.0.1:7437/health
{"service":"engram","status":"ok","version":"0.1.0"}
# And mem_doctor reports a healthy store:
sqlite_lock_contention: ok (journal_mode=wal, busy_timeout_ms=5000, checkpoint_log=890)
💡 Additional Context
Related issues (this issue is the connective tissue between them):
Option B from the original #271 report (DSN pragmas via ?_pragma=busy_timeout(5000) plus a multi-connection read pool) was not chosen. That option would have provided read concurrency headroom, making staleness less catastrophic (reads could still succeed on a fresh connection). Either Option B is worth reconsidering, OR #206 should be prioritized as the necessary companion to the chosen Option A.
Downstream impact: the gentle-engram Pi plugin reports this failure as "could not reach Engram HTTP server", which is misleading (the server is reachable, only the DB-backed handlers are poisoned). Improving that error message would help users debug, but the underlying recovery gap remains.
Happy to contribute a fix in either direction (fsnotify-based reload aligned with #206, or a /healthz endpoint that probes the DB), pending maintainer guidance on which direction is preferred.
📋 Pre-flight Checks
status:approvedbefore a PR can be opened📝 Bug Description
This is a follow-up on the architectural decision in #271 (fixed in v1.14.6). That fix correctly enforced
db.SetMaxOpenConns(1)so thebusy_timeout=5000pragma always applies to every connection in the pool. The single-connection decision is sound for SQLite single-writer correctness.However, the decision has an undocumented trade-off that amplifies the severity of #477 (stale WAL-index mmap in long-running processes): with a single connection, there is zero concurrency headroom. When the WAL-index mmap goes stale (per #477), the single connection is the ONLY connection, so every read AND write operation fails or hangs. There is no secondary path, and no code path triggers a reopen.
Concretely:
/healthcontinues to respond (trivial handler, no DB access), while every DB-backed endpoint (/search,/observations, writes) silently fails. The process cannot self-recover; it requires an externalkill+ restart.This is not a duplicate of #477 or #206. It is the architectural gap between them:
🔄 Steps to Reproduce
engram serveas a long-lived daemon (no cloud sync, no MCP needed to trigger this).-shm/-walfiles on disk (corruption recovery, external checkpoint, or the natural staleness in bug(store): long-running processes keep stale WAL-index (-shm) mmap after file replacement — silent write loss + constraint failed (1555) on mem_session_summary #477) leaves the daemon operating on a divergent WAL view./healthreturnsok,mem_doctorreports 4/4 checks passing (includingcheckpoint_log: 890), but/searchhangs and returnsSQLITE_BUSYafterbusy_timeout=5000.lsof -p $(pgrep -f 'engram serve') | grep db-shmthat the daemon holds a stale inode versus a fresh process.kill+ restart.Reproduced on Linux (Ubuntu via Homebrew), Engram 1.16.0, single daemon, ~1.7 days uptime.
✅ Expected Behavior
A single-connection pool should be paired with a mechanism that detects WAL-index staleness and reopens the connection transparently (fsnotify on
-shm/-wal, or inode verification before each transaction), as proposed in #206. With that companion, #271's single-conn decision becomes both correct AND resilient.Additionally,
/healthshould distinguish "process alive" from "DB reachable", so monitoring can detect the silent divergence instead of relying onmem_doctor(which checks store integrity, not handler liveness).❌ Actual Behavior
/healthreports ok while DB-backed operations fail.mem_doctorreports all checks passing while writes are silently lost (per bug(store): long-running processes keep stale WAL-index (-shm) mmap after file replacement — silent write loss + constraint failed (1555) on mem_session_summary #477).🖥️ Environment
📋 Relevant Logs
💡 Additional Context
Related issues (this issue is the connective tissue between them):
SetMaxOpenConns(1). This is the architectural choice in question.Option B from the original #271 report (DSN pragmas via
?_pragma=busy_timeout(5000)plus a multi-connection read pool) was not chosen. That option would have provided read concurrency headroom, making staleness less catastrophic (reads could still succeed on a fresh connection). Either Option B is worth reconsidering, OR #206 should be prioritized as the necessary companion to the chosen Option A.Downstream impact: the
gentle-engramPi plugin reports this failure as"could not reach Engram HTTP server", which is misleading (the server is reachable, only the DB-backed handlers are poisoned). Improving that error message would help users debug, but the underlying recovery gap remains.Happy to contribute a fix in either direction (fsnotify-based reload aligned with #206, or a
/healthzendpoint that probes the DB), pending maintainer guidance on which direction is preferred.