token-goat-mem (the mem CLI) is a local-first, SQLite-backed store of discrete facts --
preferences, decisions, project facts, corrections -- that AI coding agents write to and read from
across sessions. This document is a map of the source tree plus the load-bearing shape decisions
that aren't obvious from any single file.
There are two live recall paths, and both converge on the same ranking function, retrieve() in
src/retrieval.ts. Plain mem recall reaches it through src/cli.ts. The TGMEM/2 wire-format path
reaches it through buildHintFormatUnsafe in src/integration-seam.ts, which is what the
SessionStart and UserPromptSubmit hooks installed by mem init claude-code invoke on every
prompt. Neither path re-implements ranking, trust classification, or contradiction resolution --
retrieve() is the single place all of that happens.
Both paths load the entire facts table with no WHERE clause and filter and rank in JS. That
is a documented, deliberate tradeoff, not an oversight: retrieve() resolves contradictions across
its whole input pool before any filter runs, and a pre-filtered pool is a partial one -- narrowing in
SQL surfaced a genuinely contested fact as clean ground truth, because the other side of the
contradiction had already been filtered out before resolution ever saw it.
Ranking is hybrid: BM25 over Porter-stemmed, stopword-filtered text, fused by reciprocal rank fusion
(RRF) with rank lists from embeddings, entity overlap, usefulness feedback, and graph propagation.
An auxiliary list contributes only when it is non-empty, and the BM25 list only when some score in
it is above zero: an all-zero BM25 list is a ranking, but not a signal -- its sort has fallen through
to the captured_at tie-break, and feeding that pure recency order into RRF let recency outvote a
real embedding match. Every ranking sort ends on the same two keys, newest captured_at and then
fact id (plain code-unit order, compareFactIds), so facts with equal scores recall in one order
whatever order the store lists them in.
The graph signal (src/factgraph.ts) propagates score across the co-occurrence edges between facts
that share entity vocabulary, so a fact the query never mentions can surface on the strength of its
neighbours. Like every other signal it reaches retrieve() as precomputed data -- the caller runs
getGraphScoresForQuery against the database and passes the resulting map in.
Structured facet extraction (src/facets.ts) exists because BM25's tokenizer destroys exactly the
tokens a query is most likely to be about: identifiers, paths, and version strings collapse into
unrelated stems or fragments once lowercased, split, and Porter-stemmed. Facets preserve those tokens
verbatim. The path into ranking is indirect, and deliberately so: src/storage.ts extracts a fact's
facets at write time into the fact_terms table, then extracts a query's facets at read time to
join against it, handing retrieve() a finished entity-overlap map. src/retrieval.ts never imports
facets.ts -- the dependency runs one way only (facets -> retrieval, for the shared tokenizer), which
is what keeps the ranking module free of any knowledge of storage or facet extraction.
Trust (ground-truth, hint, or withheld) is computed per query and never stored. Anchors --
read-only predicates attached to a Fact -- are re-evaluated on every recall under a time budget; a
budget-exhausted check reads as unverified, which classifies as hint rather than being suppressed
outright, so a slow filesystem never silently drops a fact from view. Anchors are evaluated in rank
order (pinned facts first), so when the shared deadline runs out mid-loop the facts cut short to
unverified are the lowest-ranked ones. An unverified verdict also carries a reason, one of
no-anchor, root-missing, out-of-scope, undecidable or out-of-time (UnverifiedReason in src/anchors.ts),
which mem show and mem review print and mem show --json exposes as freshnessReason.
Verdicts persist in anchor_cache, keyed by (root, anchor) and carrying a witness (an mtime, a
hash, a git ref) so a later evaluation can treat a mismatch as a miss rather than trusting a stale
affirmed. The cache is what keeps the budget from being spent twice on the same predicate across
sessions. Which roots get prefetched is not a free choice: anchorRootsFor derives them from the
same anchorRootFor the evaluation itself uses, because a root computed any other way produces a
key the lookup never hits -- a silent, verdict-correct miss that re-evaluates every anchor while
every test still passes. mem epoch --gc deletes cache rows older than 90 days or stamped more
than a day in the future; a deleted verdict costs one re-evaluation, never a wrong answer, since a row is reused only
when its witness still matches.
Facts of kind preference decay on a 180-day half-life computed fresh from captured_at on every
read and never persisted back to storage. Decay can demote a fact from ground-truth to hint, and
never further than that -- decay alone cannot withhold a fact.
Storage is SQLite via better-sqlite3. src/db.ts owns the facts, audit_log, and meta tables,
and with them the whole-database PRAGMA user_version counter: openDb runs src/migrations.ts on
every connection open. src/storage.ts owns sources, recall_log, fact_terms, anchor_cache,
and fact_links but carries no schema logic of its own -- every table and column any module needs is
one ordered step in the single migration list.
fact_links holds the sub-threshold relations mem consolidate --related discovers: pairs similar
enough to share topic vocabulary but not similar enough to merge. Recording the relation instead of
collapsing the pair is the point -- a merge destroys one of the two facts, and the similarity that
prompted it is frequently a shared subject rather than a shared claim. Writing follows this command's
usual contract: --related lists, --related --apply persists, and nothing lands until a human has
seen the listing. Pairs are stored in canonical
order (fact_id_a < fact_id_b), enforced by a CHECK and not merely by convention, so an unordered
pair cannot be written once per side and double every count of how many links exist. Rows are not
status-aware: the cascade fires on delete, not on supersede, so the first consumer that cares must
join facts and filter on status itself.
Each step is individually idempotent, guarded by a PRAGMA table_info column check rather than by
catching SQLite's "duplicate column" error text, because a store written by an older build arrives
with every column already present and user_version still at 0. Replaying the baseline against it
has to be a silent no-op, so a fresh database and a years-old one converge on the same shape the
first time a newer build opens either.
src/integration-seam.ts publishes TGMEM/2, a line-oriented wire format with a versioned header, one
line per fact, and an optional trailing footer line. This module never imports or reads token-goat's
own state -- it is a one-directional, pull-based, pure-CLI contract: it shapes CLI output for a
caller like token-goat to consume via mem recall --hint-format. Line shape, field order, separator,
and the closed tag/verdict sets are version-committed; the footer's prose is explicitly not, so it can
change without bumping the wire version.
A hint run spends its soft budget in two parts: the anchor checks get all of it but a fixed
POST_ANCHOR_RESERVE_MS (25 ms), which is kept for ranking, formatting and recording the result. A
run that still runs out of time emits only the pinned reserve followed by its own footer line
(BUDGET_EXHAUSTED_PINNED_FOOTER_LINE), or the older empty-set footer when nothing is pinned, and
keeps the anchor verdicts it had already computed for the next run. TGMEM/1 has no footer line, so
it still gets nothing.
Each hint run ends by appending one JSON line to hook-outcomes.log beside the store
(src/hookOutcomes.ts): hook, outcome (ok, out-of-time, store-unreadable, internal-error),
duration and the number of anchor checks the budget cut short. The log rotates to .log.1 at 256 KiB
and every failure to write it is swallowed, since a diagnostic must not cost a hook its budget or
change its output. mem doctor summarizes both files from the file system rather than the store, so
it still counts the runs that met an unreadable store.
eval/ is a ranking-quality harness, not a test: npm run eval runs it, and npm test deliberately
does not. It scores the same synthetic corpus under four configurations -- three reduced rankers in
eval/rank.ts (recency, query-without-stemming, query-with-stemming) and "pipeline", which drives
the real retrieve() -- and reports precision@k, nDCG@k, duplicate-subject pairs, tokens, and the
share of surfaced anchored facts whose anchor re-affirmed. The baselines exist to make the last
number meaningful in both directions: they report n/a for it, because rankFacts has no anchor
stage at all.
Two properties make the numbers mean anything. The corpus is seeded, so a diff in the output is a
diff in the ranker; and "pipeline" runs against a real temp filesystem (eval/pipelineFixture.ts),
because an anchor verdict is a trust signal rather than a ranking one -- it moves neither precision
nor nDCG, so a corpus whose anchors all fail to resolve scores identically to one whose anchors all
hold. That was not hypothetical: the fixture's anchors were unevaluable for their whole existence and
every ranking metric was unmoved.
The "a query ranks, it never filters" invariant is checked by re-running each no-match scenario
against the same configuration with an empty query and comparing pool sizes. Comparing against a
scope-eligible count instead charges the result limit, unconditional superseded-exclusion, and
restrictToRoot binding to the query, and reports a correct pipeline as a violation.
eval/ sits outside .arch-doc-sync.json's roots, so the generated table below does not cover it --
which is exactly why it is described here.
npm run eval -- --check compares the four configurations' means against the committed
eval/expected.json and fails on a regression (an improvement only hints to refresh the file);
npm run eval -- --write-expected regenerates it after an intended ranking change. CI runs the check.
The table below the marker comment is generated, not hand-maintained. scripts/sync-arch-docs.mjs
discovers every module in src/ via git ls-files, classifies it into a layer from
.arch-doc-sync.json, and extracts a Role (from the module's own doc-comment) and its key exports.
npm run arch:writeregenerates the table in place.npm run arch:checkis read-only and fails (exit 1) on any drift -- a new or removed module, a changed export list, or a module that matches no declared layer. It runs as part of the guards tier (tests/guards/arch-docs.test.ts), which the pre-commit hook and CI both enforce.- A Role you edit by hand is preserved as curated: as long as it differs from what fresh extraction
would produce for that path,
--writenever overwrites it, and a rename or split is followed via git history so a curated Role survives agit mvtoo.
| Module | Layer | Role | Key exports |
|---|---|---|---|
src/backupCommands.ts |
Entry | mem backup and mem restore: the hand-driven side of the snapshots src/backup.ts takes on its own |
registerBackupCommands |
src/cli.ts |
Entry | Commander-based CLI wiring for mem (design plan Sections 3/4/5/6, AGENTS.md's command list) |
describeHookHealthRefusal, buildProgram, run |
src/cliRuntime.ts |
Entry | The runtime every mem command action shares: the exit-code contract, the error-to-exit-code mapping, the guard wrapp |
EXIT_SUCCESS, EXIT_USER_ERROR, EXIT_INTERNAL_ERROR, UsageError, exitCodeForError |
src/doctor.ts |
Entry | mem doctor: the read-only environment and store health check |
DOCTOR_CHECKS, DoctorCheck, FindingStatus, Finding, LAST_SURFACED_WARN_AFTER_MS |
src/doctorWiring.ts |
Entry | mem doctor's wiring checks, kept out of doctor.ts: whether each supported tool's mem block is present and current (`wi |
wiringHomeFromEnv, PROBE_TIMEOUT_ENV, probeTimeoutFromEnv, ProbeBudget, createProbeBudget |
src/hookCommand.ts |
Entry | mem hook <event>: the one entry point installed hook commands call |
MEM_HOOK_IMPLIED_ARGS, isMemHookSlug, deriveHookRoot, buildHookArgv, HookCommandOptions |
src/index.ts |
Entry | Library entry point | — |
src/main.ts |
Entry | Package executable | — |
src/reflect.ts |
Entry | mem reflect: the end-of-session worklist that turns pending suggestions into decisions, update-before-create |
registerReflectCommand |
src/timeline.ts |
Entry | mem log: the store-wide audit timeline, newest first |
formatAuditLine, resolveLoggedFactId, registerLogCommand |
src/anchors.ts |
Retrieval | Anchor evaluation (design plan P3, Section 3, review S1/S4) | AnchorVerdict, AnchorCacheStore, clearAnchorCaches, _clearAnchorMemoForTests, isGenuineAbsence |
src/contradiction.ts |
Retrieval | Deterministic subject+value contradiction detection (design plan P4, Section 6, review S5/S8) | ContradictionGroup, FactStatusUpdate, ContradictionDetectionResult, computeProjectIdentityGroups, computeContradictionBucketGroups |
src/embeddings.ts |
Retrieval | A concrete EmbeddingBackend (retrieval.ts) speaking the OpenAI /v1/embeddings wire shape, plus the config plumbing t |
EMBED_URL_ENV, EMBED_MODEL_ENV, EMBED_API_KEY_ENV, DEFAULT_EMBED_REQUEST_TIMEOUT_MS, EmbeddingConfig |
src/facets.ts |
Retrieval | Structured facet extraction: the tokens BM25's tokenizer destroys, preserved verbatim | FactFacets, MAX_ENTITIES_PER_FACT, MAX_TOPICS_PER_FACT, normalizeTermKey, extractFacets |
src/factgraph.ts |
Retrieval | The co-occurrence fact graph and score propagation over it (design plan Section 3 extension, "Retrieval" layer) | FactGraphOptions, PropagateOptions, neighbours, propagate, getGraphScoresForQuery |
src/related.ts |
Retrieval | Which live facts share the most entities and topics with a given fact -- the one similarity lookup behind `mem show --re | RelatedFact, findRelatedFacts |
src/retrieval.ts |
Retrieval | Hybrid retrieval (design plan Section 3 "Retrieval", P8, review S10) | EmbeddingBackend, EmbeddingBackendLoader, TrustLevel, ContradictionOutcome, RetrievalOptions |
src/capture.ts |
Capture | The two-mode capture pipeline (design plan Section 3, principles P1/P7, review findings S7/S9) | MAX_SOURCE_EXCERPT_LENGTH, CaptureValidationError, InvalidAnchorError, SecretMatchSummary, SecretDetectedError |
src/consolidate.ts |
Capture | Near-duplicate clustering and stale-fact detection -- the two halves of mem consolidate |
DEFAULT_DUPLICATE_THRESHOLD, DEFAULT_STALE_AGE_DAYS, DuplicateMember, DuplicateCluster, jaccard |
src/dream.ts |
Capture | Cross-fact inference ("dreaming"): propose facts that follow from several stored facts but that nobody stated outright | DREAM_URL_ENV, DREAM_MODEL_ENV, DREAM_API_KEY_ENV, MAX_DREAM_FACTS, DreamConfig |
src/exportImport.ts |
Capture | mem import --from-json <path> -- the full-fidelity counterpart to mem import --from-md (src/import.ts) |
JSON_EXPORT_SCHEMA_VERSION, MAX_IMPORT_FILE_SIZE_BYTES, JsonImportError, planImportFromJson, ImportFromJsonOptions |
src/import.ts |
Capture | mem import --from-md <path> -- the mem-side half of the "advisory CLAUDE.md->mem migration probe" (the other half, `to |
MarkdownImportError, MarkdownBullet, extractMarkdownBullets, ImportFromMarkdownOptions, ImportCandidate |
src/reviewActions.ts |
Capture | The status transitions a human drives through mem review (promote, reject, undo a rejection), and the two primitives e |
setStatusWithAudit, REVIEW_RESOLVABLE_STATUSES, reconcileContradictions, PromotionOutcome, PromotionKey |
src/sessionCapture.ts |
Capture | Files a session transcript's durable-statement candidates as pending suggestions -- the store side of mem scan-session |
transcriptSourceRefPrefix, FileTranscriptOptions, fileTranscriptSuggestions |
src/sessionScan.ts |
Capture | Deterministic extraction of durable-preference candidates from a session transcript | MAX_SCANNED_TURNS, MAX_CANDIDATE_LENGTH, Candidate, userTurnText, extractCandidates |
src/backup.ts |
Storage | Snapshots of the mem store, kept outside the mem home so they outlive it (resolveBackupDir in src/db.ts: `~/.mem-backu |
SnapshotReason, Snapshot, AUTO_SNAPSHOT_INTERVAL_MS, AUTO_SNAPSHOTS_KEPT, AUTO_SNAPSHOT_CLAIM |
src/db.ts |
Storage | SQLite connection and schema for the facts table (design plan Section 3), plus two small infra tables every write path |
resolveMemHome, resolveDbPath, BACKUP_DIR_ENV, resolveBackupDir, OpenDbOptions |
src/epoch.ts |
Storage | The write epoch's read and write primitives (design plan Section 4: "every write bumps it") | getEpoch, writeEpoch |
src/hookOutcomes.ts |
Storage | Best-effort JSONL log of how each hook run ended (ok, out of time, store unreadable, internal error), kept beside the store and summarized by mem doctor |
HOOK_OUTCOME_HOOKS, HOOK_OUTCOME_KINDS, HookOutcomeHook, HookOutcomeKind, HookOutcomeRecord |
src/migrations.ts |
Storage | Ordered schema migrations for the mem database, keyed on SQLite's own PRAGMA user_version |
MigrationStep, MigrationResult, hasColumn, addColumn, MIGRATIONS |
src/promptBlockState.ts |
Storage | The UserPromptSubmit memory-block marker: one empty file per session beside the store, present while the block already w | promptBlockMarkerPath, promptBlockSent, markPromptBlockSent, resetPromptBlock |
src/restore.ts |
Storage | mem restore's engine: replaces the live store's contents with a snapshot's, without ever handing the live file to anyt |
UnusableSnapshotError, RestoreOptions, RestoreResult, restoreStore |
src/storage.ts |
Storage | Storage layer: schema and typed CRUD for the sources table plus a write epoch, and typed CRUD for the facts table (d |
ensureStorageSchema, openStorage, createAnchorCacheStore, clearAnchorCacheStore, BufferedAnchorVerdict |
src/hook-envelope.ts |
Integration | Parsing for the JSON envelope a coding tool's hook hands mem recall --hook-stdin on stdin |
HOOK_PROMPT_KEYS, HOOK_SESSION_KEY, HOOK_TRANSCRIPT_KEY, HOOK_STOP_ACTIVE_KEY, HookEnvelope |
src/integration-seam.ts |
Integration | The token-goat integration seam (design plan Section 4) | TGMEM_PROTOCOL_VERSION, TGMEM_HEADER, FOLLOW_UP_SHOW_DETAIL, FOLLOW_UP_REVIEW, TGMEM_FOOTER_LINE |
src/probeSpawn.ts |
Integration | Synchronous launch of one probe of the mem binary that never leaves a live process behind when the probe budget expire |
ProbeSpawnResult, PROBE_HELPER_BUDGET_ARG_INDEX, isSafeProbeToken, PROBE_REFUSED_ERROR_CODE, ProbeSpawnOptions |
src/wiring.ts |
Integration | Automates what docs/integrations/*.md currently ask a human to hand-copy: install() writes exactly the config snippets |
WiringOpts, WiringFileAction, WiringChange, WiringResult, WiringPlanEntry |
src/budgetClock.ts |
Support | The clock every retrieval deadline reads: the anchor time budget, the embedding timer budget and the hint seam's soft bu | budgetNow |
src/budgetTimer.ts |
Support | The timers that enforce a retrieval deadline on the budget clock: the race around an embedding backend and the abort sig | BudgetTimerOptions, BudgetTimeout, armBudgetTimer, startBudgetTimeout |
src/factText.ts |
Support | The normalized-text key storage.ts and migrations.ts both need for fact deduplication |
normalizeFactText, hashFactText |
src/fileUtils.ts |
Support | Shared filesystem error handling for imports | MEM_HOME_MODE, MEM_DB_MODE, restrictPermissions, formatBytes, readFileWithErrorMapping |
src/pathUtils.ts |
Support | Case-folds a path for comparison on filesystems that ignore case | normalizePath, isInsideOrEqual |
src/processAlive.ts |
Support | Whether a process with pid exists: signal 0 delivers nothing and fails with ESRCH for a pid no process holds, while EP |
processIsAlive |
src/projectIdentity.ts |
Support | Repository-relative identity for a project root, so a project-scoped fact survives the path it was captured at | PROJECT_IDENTITY_ENV, clearProjectIdentityCache, normalizeRemoteUrl, resolveProjectIdentity, normalizeProjectIdentity |
src/timeUtils.ts |
Support | Day arithmetic shared by every age-based rule in mem: recall's --age-days window, preference decay, pin reconfirmation |
MS_PER_DAY, daysAgoIso, parseStrictIsoTimestamp, ageInDays, formatAge |
src/types.ts |
Support | Shared domain types for token-goat-mem | FACT_KINDS, FactKind, factNounPhrase, FACT_SCOPES, FactScope |