[Feature] Pluggable agent memory: a generic backend seam with causal-memory / mem0 as reference implementations

[idling11]

Background

Codewhale's agent memory is currently a hardwired, single-implementation native feature with no pluggable third-party backend entry:

• The MemoryBackend enum has only Native / Off variants (crates/tui/src/config.rs:1985) — no Custom/External/MCP.
• NativeMemoryStore (crates/tui/src/native_memory.rs:51) is a concrete struct (Markdown source + SQLite FTS5 index) with no trait, so a third party cannot supply their own store.
• The plugin activation capability list PluginActivationCapability (crates/tui/src/plugins/activation.rs:26) has no Memory category, so a plugin cannot declare "I provide a memory backend".
• docs/MEMORY.md states that native is the only memory system, the planned Moraine MCP backend was removed, and backend accepts only native / off.

Today a third-party memory system can only be side-mounted through the generic MCP channel (most external memory systems expose an MCP server). It can be wired in, but it is not a first-class backend recognized by [memory] — # quick-add, the remember tool, and the /memory command all remain pinned to native.

Goal

Upgrade memory from "single, hardwired implementation" to a generic, pluggable, optional seam:

• Keep native as the default, zero-dependency local backend.
• Define a stable backend seam that any third-party memory system can implement — not one tailored to a particular vendor.
• Make [memory] backend a first-class config choice, with the existing remember / recall / /memory lifecycle routing to whichever backend is selected; a backend only implements "store" and "retrieve".
• Ship causal-memory and mem0 as the first two reference implementations to prove the seam is generic across very different backend shapes — not as the endpoint of the work.

The seam is open to any memory system

The abstraction is deliberately transport- and vendor-agnostic, so the following (and others) can all be implemented behind the same interface:

• causal-memory — causal/decision memory (local, MCP stdio/HTTP).
• mem0 — fact/preference recall (remote service, API/SDK + MCP).
• Zep, Letta (MemGPT), LangMem, MemOS, Mem0 — any agent-memory platform exposing a service API or MCP.
• A custom in-house store — implement the trait directly.

The only requirement is that a backend can answer "store an entry" and "retrieve relevant entries for a query".

Reference implementation #1 (focus): causal-memory (developed in-house)

causal-memory is our open-source agent memory system (Rust, Apache-2.0, JingxuanC/causal-memory). Its core differentiator is a causal core + inhibition modeling: facts, temporal state, and decision → outcome causal edges live on a single SQLite store, driven by a hippocampus-style engine (typed spreading activation with excitatory and inhibitory edges, Hebbian co-occurrence reinforcement, Q-value dynamics, immutable SWR consolidation). Agents recall not only what happened and when it was true, but why it worked and what would happen if they acted differently.

Why it is the first backend we build:

1. Causal information is the most fragile type under text compaction. The causal table lives outside the agent's context window, so compaction cannot touch it — in a real-LLM compaction benchmark causal-memory retains 100% causal-table recall while mem0 drops to 45%. This directly patches the "re-hits the same bug the same wrong way" hole after long-session compaction.
2. Clearly ahead of mem0 on causal capabilities (CausalEval): C7 belief update 100% vs 80%, C2 intervention prediction 75% vs 40%, C4 inhibition distinction 80% vs 50%.
3. Integration surface is ready: 14 MCP tools (stdio + HTTP transports), a Python Memory facade, and an existing DSH plugin shape — wiring into Codewhale's MCP/plugin channel is nearly zero new code.
4. The honest caveat: pure fact recall (LoCoMo 79.1% vs mem0 91.6%) is not its strong suit — that is exactly mem0's strength. The two are complementary, which is why a generic seam beats "pick one".

Reference implementation #2: mem0

mem0 is the fact-recall specialist (LoCoMo 91.6%) and is chosen as the second reference implementation precisely because its shape is the opposite of causal-memory's: a remote hosted service with an API/SDK, rather than a local store. If the seam can absorb both, it is general enough for the rest of the ecosystem.

Plan (phased)

• Phase 0 — baseline (already landed): native as the only backend, [memory] opt-in, # quick-add, remember/recall tools. Landed in v0.9.x.
• Phase 1 — define the generic backend seam (core change): introduce a MemoryBackend trait with a minimal, transport-agnostic method set (store / search / prompt_block, plus a health/status probe); add an External/Custom variant to the MemoryBackend enum; make [memory] backend resolve a backend by kind + connection parameters (process/stdio command, HTTP service URL, or MCP server name) rather than hardcoding vendor names. Native becomes a default impl of the trait; Off stays as-is. Add a Memory capability to PluginActivationCapability so a backend can ship as a plugin bundle.
• Phase 2 — two reference implementations to prove genericity: land causal-memory first (MCP stdio + HTTP), then mem0 (remote API/SDK). Success criterion: the trait needed no vendor-specific escape hatches to absorb two very different backend shapes.
• Phase 3 — bring-your-own-backend: document the driver/adaptor contract so any third party (Zep, Letta, LangMem, a custom store, …) can add a backend — ideally as an external plugin bundle, without a core change.
• Phase 4 — wrap-up: update docs/MEMORY.md, config.example.toml, and the [memory] config contract; add backend selection + fallback (degrade to off or native when a backend is unavailable); document the KV-cache effect (the boundary of memory injected into the system-prompt prefix is unchanged).

Integration surface

• Reuse the existing MCP/plugin channel: causal-memory is already an MCP server and can land first as a plugin bundle (McpStdio/McpRemote) as the Phase 2 bootstrap — no need to wait for the trait abstraction before trying it out. Any MCP-speaking memory system gets the same path for free.
• Add a native trait backend: make [memory] backend = "causal-memory" (or any registered backend) a first-class config option, with the remember tool, /memory command, and # quick-add all routing to the selected backend.
• Open the plugin capability: a Memory activation capability lets a third party ship a backend as a reviewed plugin, exactly like they ship skills, MCP servers, or hooks today.

References

• Current memory implementation: crates/tui/src/native_memory.rs, crates/tui/src/config.rs (MemoryBackend / MemoryConfig)
• Memory docs: docs/MEMORY.md
• Plugin activation capabilities: crates/tui/src/plugins/activation.rs
• causal-memory: https://github.com/JingxuanC/causal-memory (README includes the CausalEval / compaction-survival benchmarks)
• Original memory EPIC: EPIC: Project + user memory system (Claude Code-style) #489

[Hmbown]

Thanks for separating persistence from pluggable backends. I audited the live native_memory and remember paths for 0.9.13: opt-in native memory already persists Markdown notes across sessions, with workspace/global scope and a rebuildable SQLite FTS5 search index. Enable [memory] enabled = true; remember, memory_search, memory_get, and /memory native use that store.

Backend selection currently supports only native and off. A memory service can expose its own tools through existing MCP/plugin support, but that does not replace the native commands or quick-add path. A first-class backend needs one complete capture/search/correction/deletion/scope/error contract across those entry points. Volatile recall must append to history, preserving the frozen prompt prefix described in docs/CACHE.md.

The bounded 0.9.13 slice is documentation of that existing behavior and its privacy boundary: recalled notes go to the selected model provider. I am leaving this issue open for the complete backend work; no causal-memory or mem0 integration is being claimed in this release.