feat(reindex): two-tier index — engine, write-time budgets, lane routing, skill

.gitignore

@@ -7,3 +7,5 @@ COMMIT_EDITMSG
 .cache/
 # internal-only: marketing, launch drafts, positioning notes — not shipped OSS
 internal/
+# codegraph local index + daemon state — rebuildable, machine-local
+.codegraph/

CLAUDE.md

@@ -75,6 +75,8 @@ okfmem backfill  --dry-run    # stamp decay frontmatter on existing pages
 okfmem init      --dry-run    # per-repo memory link + pointers + registry wiring
 okfmem consolidate --dry-run  # decay + archive stale pages + push
 okfmem sync [-m "<msg>"]      # commit + pull-rebase + push the store (prompts for the message if -m omitted)
+okfmem reindex --report       # read-only: auto-loaded bytes vs ceiling + per-section breakdown
+okfmem reindex --verify       # read-only: link integrity across every MEMORY*.md; exits 1 on dangling/orphans
 
 python3 scripts/check-leaks.py   # leak gate (also runs first in CI)
 ruff check .                     # lint (advisory in CI today)

README.md

@@ -146,7 +146,7 @@ the inverse, so I built the smallest thing that does it.
 
 Storage uses [Google's Open Knowledge Format (OKF) v0.1][okf] — one markdown page per topic, YAML frontmatter, plain-markdown links. No database, no server.
 
-> A page read 3× survives ~3× longer; the always-loaded index stays ≤200 lines. Decay does the forgetting so the signal never gets buried mid-context.
+> A page read 3× survives ~3× longer; the always-loaded index stays under an 8KB byte budget. Decay does the forgetting so the signal never gets buried mid-context.
 
 ## Architecture: Engine ⇄ Store Split
 
@@ -178,7 +178,7 @@ Once installed, the memory system works transparently with your AI agent.
 ### 1. Auto-Loading Context (Start of Session)
 When the AI starts, it automatically reads two files per project:
 *   **`STATE.md` (Active State):** A bounded snapshot of current work, priorities, and context. Overwritten every session.
-*   **`MEMORY.md` (Durable Knowledge):** A 200-line index of one-line pointers to deeper knowledge.
+*   **`MEMORY.md` (Durable Knowledge):** An index of one-line pointers to deeper knowledge, kept under an ~8KB auto-load budget — a page's pointer routes to a topic-specific lane index by default, with `MEMORY.md` itself reserved for cross-cutting facts and a routing map to the lanes.
 
 ### 2. On-Demand Retrieval (During Session)
 If the AI needs more context, it `grep`s the durable `<slug>.md` pages referenced in `MEMORY.md`.
@@ -233,7 +233,7 @@ Scans your system for supported harnesses (Claude Code, Antigravity) and writes
 An idempotent tool that stamps required YAML frontmatter (like `importance`, `pinned`, `created`) onto all durable pages. (The `install.sh` script runs this automatically).
 
 ### 4. Status Check (`okfmem status`)
-Run this anytime to view the wiring status, detected harnesses, and whether your store has any uncommitted changes. It also prints a per-project inventory — page and archive counts, `MEMORY.md` line count, and `STATE.md` presence — marking the project your current directory maps to (`*`) and flagging any project whose `MEMORY.md` has grown past the 200-line auto-load limit (a `/okfmem-curate` candidate), plus the decay epoch. The default view collapses to the current project and any over-limit project; add `--all` to list every project, or `--project <name>` for one.
+Run this anytime to view the wiring status, detected harnesses, and whether your store has any uncommitted changes. It also prints a per-project inventory — page and archive counts, `MEMORY.md` size in bytes, and `STATE.md` presence — marking the project your current directory maps to (`*`) and flagging any project whose `MEMORY.md` has grown past the 8KB auto-load byte ceiling (a `/okfmem-reindex` candidate, recommended remedy a lane split), plus the decay epoch. The default view collapses to the current project and any over-ceiling project; add `--all` to list every project, or `--project <name>` for one.
 
 ### 5. Session Search (`okfmem search`)
 An opt-in plugin that builds a local SQLite FTS5 index over your agent's past conversation transcripts (e.g., Claude Code or Antigravity logs). This allows your agent to perform deep full-text searches across historical sessions to recover details not currently in `MEMORY.md`. The `.db` is purely a derived local cache—gitignored and rebuildable anytime via `okfmem index`.
@@ -257,6 +257,24 @@ okfmem graduate my-slug             # [y/N] confirm, then apply
 
 It distills the source page's body into the target `CLAUDE.md` (default: the project-root file; `--to <dir>/CLAUDE.md` targets a lane-scoped file, seeded on first write), mirrors the same insertion into a sibling `AGENTS.md` **only when it's a real file** — a symlinked `AGENTS.md` (e.g. `AGENTS.md -> CLAUDE.md`) already resolves through, so it's left alone — and then **archives, never deletes**, the source page: moved to `projects/<proj>/archive/`, its `MEMORY.md` line dropped, and its frontmatter stamped with `graduated_to:` (target file + heading anchor + date/PR) so provenance survives and a later curate pass never re-flags or hard-deletes it. Writing outside the store is a rung-2 op, so it sits behind a `[y/N]` confirmation — skippable non-interactively, printing the exact manual command to run later.
 
+### 8. Reindex measurement (`okfmem reindex`)
+Once an index grows past one file, a few questions decide every restructuring — and none is answerable by looking at file sizes. Every mode is strictly read-only; none writes to the store.
+
+```bash
+okfmem reindex --report          # where the auto-loaded bytes actually sit
+okfmem reindex --verify          # did that move break anything?  exit 1 if so
+okfmem reindex --verify --json   # same, machine-readable
+okfmem reindex --budget-check    # pointer lines over the per-line char budget
+```
+
+**`--report`** leads with the only two files a harness auto-loads — `MEMORY.md` and `STATE.md` — measured against their byte ceiling, then breaks `MEMORY.md` down **per section**. That last number is the one that matters: a 18 KB index where a single flat block holds 83% of the bytes needs that *lane split out*, while the same 18 KB spread evenly needs its hooks tightened. Total file size can't tell those apart. Pages on disk are reported too and explicitly labelled non-context: several hundred pages and a few MB contribute exactly zero at session start, so page count is never by itself a reason to prune.
+
+**`--verify`** walks **every** `MEMORY*.md` and accepts **both** pointer syntaxes — `[title](slug.md)` and the bare `- slug.md — hook` a lane index uses — reporting each dangling pointer *named with the index file it came from*, plus any page in no index at all. It exits non-zero on either, so it can gate a reindex.
+
+**`--budget-check`** counts index lines over the per-line pointer budget (150 characters), across every index and in both syntaxes. **Characters, not bytes** — the pointer convention's em-dash is one character and three bytes, and a byte count over-reports every line that carries one. It is advisory and always exits 0; the byte ceiling `--report` measures is the number that gates anything.
+
+The parsing is anchored on the link **target**, at both ends of the line: a pointer whose *title* starts with a filename (`- [CLAUDE.md subdir lanes](real-slug.md)`) resolves to `real-slug.md`, and a filename named in the *hook* stays prose. Retired `ck_*.md` snapshots are skipped as orphan candidates, the same way `consolidate` and `backfill` skip them — they were never indexed by design, and counting them would make `--verify` fail on most real stores for a known-benign reason. This lives in Python rather than shell on purpose — the obvious `grep | sed` version of the same check uses a GNU-only BRE that BSD `sed` ignores, so on macOS it silently reported nothing, and the `awk` version of the budget count was blind to bare pointers *and* counting bytes. A checker that fails open is worse than no checker.
+
 ```mermaid
 flowchart TD
     subgraph Harness [AI Coding Agent]

memory_consolidate.py

@@ -41,6 +41,10 @@
 # Shared git commit+push path (pull-rebase + lock) lives beside this script.
 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
 from memory_sync import sync_store  # noqa: E402
+# Index enumeration: since #53's write-time lane routing, a page's pointer may
+# live in any `MEMORY*.md`, not just the root one. One shared enumerator so
+# this pass and `okfmem reindex --verify` can never disagree about the file set.
+from memory_reindex import index_files  # noqa: E402
 
 SKIP_NAMES = {"MEMORY.md", "STATE.md", "CONTEXT.md"}
 DECAY_EXEMPT_TYPES = {"user", "feedback"}
@@ -773,12 +777,20 @@ def main():
         archive_page(c, today, args.dry_run)
         per_proj_slugs.setdefault(pdir, []).append(c["slug"])
 
+    # Drop the archived page's pointer from EVERY index that carries it, not
+    # just the root `MEMORY.md`. With #53's lane routing a new page's pointer
+    # is written straight into a lane index, so a root-only drop leaves a
+    # dangling pointer behind -- and this pass runs unattended from the Stop
+    # hook, so the damage accumulates silently until `--verify` starts failing
+    # with no user action. `drop_memory_lines`'s two patterns already handle
+    # both pointer syntaxes; only the file set was wrong.
     dropped_total = 0
     for pdir, slugs in per_proj_slugs.items():
-        dropped_total += drop_memory_lines(
-            os.path.join(pdir, "MEMORY.md"), slugs, args.dry_run)
+        for idx in index_files(pdir):
+            dropped_total += drop_memory_lines(
+                os.path.join(pdir, idx), slugs, args.dry_run)
     if to_archive:
-        print(f"MEMORY.md lines dropped: {dropped_total}")
+        print(f"index lines dropped: {dropped_total}")
 
     print(f"mode: {'DRY-RUN' if args.dry_run else 'APPLY'}")
 

memory_graduate.py

@@ -52,6 +52,7 @@
     update_fields,
 )
 from memory_init import _current_git_root, _load_registry, _prompt_yes_no  # noqa: E402
+from memory_reindex import index_files  # noqa: E402
 
 DEFAULT_STORE = os.environ.get("OKFMEM_STORE", os.path.expanduser("~/okfmem-store"))
 
@@ -254,7 +255,7 @@ def render_plan(plan):
                         os.path.basename(plan["src"]))
     lines.append("")
     lines.append(f"archive:  {plan['src']}")
-    lines.append(f"      ->  {dest}  (MEMORY.md pointer dropped)")
+    lines.append(f"      ->  {dest}  (index pointer dropped)")
     return "\n".join(lines)
 
 
@@ -275,11 +276,18 @@ def apply_plan(plan, store, today):
     # split-brain (rule in both CLAUDE.md and the page, or source live beside a
     # duplicate archive copy) survives a throw.
     proj_dir = _project_dir_of(plan["src"])
-    memory_path = os.path.join(proj_dir, "MEMORY.md")
-    memory_before = None
-    if os.path.isfile(memory_path):
-        with open(memory_path, "r", encoding="utf-8", newline="") as f:
-            memory_before = f.read()
+    # EVERY index, not just the root one: since #53's lane routing a page's
+    # pointer may live in any `MEMORY*.md`, so a root-only drop would leave a
+    # dangling pointer behind. The rollback snapshot therefore has to cover the
+    # same set — snapshotting one file while writing several would turn a
+    # failed graduate into a half-modified store, which is worse than the bug
+    # being fixed.
+    index_paths = [os.path.join(proj_dir, n) for n in index_files(proj_dir)]
+    memory_before = {}
+    for p in index_paths:
+        if os.path.isfile(p):
+            with open(p, "r", encoding="utf-8", newline="") as f:
+                memory_before[p] = f.read()
 
     # The archive destination is deterministic (same path archive_page would
     # return), computed up front so the internal-phase rollback can clean up a
@@ -309,9 +317,11 @@ def apply_plan(plan, store, today):
         with open(dest, "w", encoding="utf-8", newline="") as f:
             f.write(dest_text)
 
-        dropped = drop_memory_lines(memory_path, [plan["slug"]], dry_run=False)
+        dropped = 0
+        for p in index_paths:
+            dropped += drop_memory_lines(p, [plan["slug"]], dry_run=False)
     except Exception:
-        _rollback_internal(plan, dest, memory_path, memory_before)
+        _rollback_internal(plan, dest, memory_before)
         raise
 
     # --- outward, non-atomic side (rolled back on failure) ---
@@ -325,19 +335,23 @@ def apply_plan(plan, store, today):
             with open(plan["agents_path"], "w", encoding="utf-8") as f:
                 f.write(plan["agents_after"])
     except Exception:
-        _rollback_apply(plan, dest, memory_path, memory_before, target_written)
+        _rollback_apply(plan, dest, memory_before, target_written)
         raise
 
     return dest, dropped
 
 
-def _rollback_internal(plan, dest, memory_path, memory_before):
+def _rollback_internal(plan, dest, memory_before):
     """Best-effort undo of the store-internal phase: drop the archived copy,
-    restore the live source page, and put MEMORY.md back. Remove the archive
-    copy FIRST, then restore the source — so a partial rollback can never leave
-    BOTH (the source-live-beside-a-duplicate-archive split-brain we most want to
-    avoid). Each undo step is guarded independently; a failure in one must not
-    skip the others or mask the original exception about to be re-raised."""
+    restore the live source page, and put **every** index we snapshotted back.
+    Remove the archive copy FIRST, then restore the source — so a partial
+    rollback can never leave BOTH (the source-live-beside-a-duplicate-archive
+    split-brain we most want to avoid). Each undo step is guarded
+    independently; a failure in one must not skip the others or mask the
+    original exception about to be re-raised — which is also why the index
+    restore loops with a per-file guard rather than one try around the loop.
+
+    `memory_before` maps index path -> its content before this run."""
     try:
         if os.path.isfile(dest):
             os.remove(dest)
@@ -348,23 +362,23 @@ def _rollback_internal(plan, dest, memory_path, memory_before):
             f.write(plan["src_text"])
     except Exception:
         pass
-    try:
-        if memory_before is not None:
-            with open(memory_path, "w", encoding="utf-8", newline="") as f:
-                f.write(memory_before)
-    except Exception:
-        pass
+    for path, text in (memory_before or {}).items():
+        try:
+            with open(path, "w", encoding="utf-8", newline="") as f:
+                f.write(text)
+        except Exception:
+            pass
 
 
-def _rollback_apply(plan, dest, memory_path, memory_before, target_written):
+def _rollback_apply(plan, dest, memory_before, target_written):
     """Best-effort undo of a partially-applied graduate: restore the live
-    source page, drop the archived copy, put MEMORY.md back, and revert any
+    source page, drop the archived copy, put every touched index back, and revert any
     outward CLAUDE.md/AGENTS.md write already made — so a failed apply leaves
     the world as if graduate never ran (no split-brain, no data loss). Each
     step is guarded independently; a rollback failure must not mask the
     original error that is about to be re-raised."""
     # Store-internal: un-archive the source.
-    _rollback_internal(plan, dest, memory_path, memory_before)
+    _rollback_internal(plan, dest, memory_before)
     # Outward: revert CLAUDE.md (and a mirrored AGENTS.md) if we managed to
     # write it before failing. The mirror only runs after CLAUDE.md succeeds,
     # so it's only in play once target_written is True.
@@ -423,7 +437,7 @@ def cmd_graduate(args):
     today = datetime.now(timezone.utc).date()
     dest, dropped = apply_plan(plan, store, today)
     print(f"\narchived: {dest}")
-    print(f"MEMORY.md lines dropped: {dropped}")
+    print(f"index lines dropped: {dropped}")
     return 0