diff --git a/.gitignore b/.gitignore index 7d0515c..5f67d26 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,5 @@ COMMIT_EDITMSG .cache/ # internal-only: marketing, launch drafts, positioning notes — not shipped OSS internal/ +# codegraph local index + daemon state — rebuildable, machine-local +.codegraph/ diff --git a/CLAUDE.md b/CLAUDE.md index 2b92d90..ee282cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -75,6 +75,8 @@ okfmem backfill --dry-run # stamp decay frontmatter on existing pages okfmem init --dry-run # per-repo memory link + pointers + registry wiring okfmem consolidate --dry-run # decay + archive stale pages + push okfmem sync [-m ""] # commit + pull-rebase + push the store (prompts for the message if -m omitted) +okfmem reindex --report # read-only: auto-loaded bytes vs ceiling + per-section breakdown +okfmem reindex --verify # read-only: link integrity across every MEMORY*.md; exits 1 on dangling/orphans python3 scripts/check-leaks.py # leak gate (also runs first in CI) ruff check . # lint (advisory in CI today) diff --git a/README.md b/README.md index 013b9ae..5aa2480 100644 --- a/README.md +++ b/README.md @@ -146,7 +146,7 @@ the inverse, so I built the smallest thing that does it. Storage uses [Google's Open Knowledge Format (OKF) v0.1][okf] — one markdown page per topic, YAML frontmatter, plain-markdown links. No database, no server. -> A page read 3× survives ~3× longer; the always-loaded index stays ≤200 lines. Decay does the forgetting so the signal never gets buried mid-context. +> A page read 3× survives ~3× longer; the always-loaded index stays under an 8KB byte budget. Decay does the forgetting so the signal never gets buried mid-context. ## Architecture: Engine ⇄ Store Split @@ -178,7 +178,7 @@ Once installed, the memory system works transparently with your AI agent. ### 1. Auto-Loading Context (Start of Session) When the AI starts, it automatically reads two files per project: * **`STATE.md` (Active State):** A bounded snapshot of current work, priorities, and context. Overwritten every session. -* **`MEMORY.md` (Durable Knowledge):** A 200-line index of one-line pointers to deeper knowledge. +* **`MEMORY.md` (Durable Knowledge):** An index of one-line pointers to deeper knowledge, kept under an ~8KB auto-load budget — a page's pointer routes to a topic-specific lane index by default, with `MEMORY.md` itself reserved for cross-cutting facts and a routing map to the lanes. ### 2. On-Demand Retrieval (During Session) If the AI needs more context, it `grep`s the durable `.md` pages referenced in `MEMORY.md`. @@ -233,7 +233,7 @@ Scans your system for supported harnesses (Claude Code, Antigravity) and writes An idempotent tool that stamps required YAML frontmatter (like `importance`, `pinned`, `created`) onto all durable pages. (The `install.sh` script runs this automatically). ### 4. Status Check (`okfmem status`) -Run this anytime to view the wiring status, detected harnesses, and whether your store has any uncommitted changes. It also prints a per-project inventory — page and archive counts, `MEMORY.md` line count, and `STATE.md` presence — marking the project your current directory maps to (`*`) and flagging any project whose `MEMORY.md` has grown past the 200-line auto-load limit (a `/okfmem-curate` candidate), plus the decay epoch. The default view collapses to the current project and any over-limit project; add `--all` to list every project, or `--project ` for one. +Run this anytime to view the wiring status, detected harnesses, and whether your store has any uncommitted changes. It also prints a per-project inventory — page and archive counts, `MEMORY.md` size in bytes, and `STATE.md` presence — marking the project your current directory maps to (`*`) and flagging any project whose `MEMORY.md` has grown past the 8KB auto-load byte ceiling (a `/okfmem-reindex` candidate, recommended remedy a lane split), plus the decay epoch. The default view collapses to the current project and any over-ceiling project; add `--all` to list every project, or `--project ` for one. ### 5. Session Search (`okfmem search`) An opt-in plugin that builds a local SQLite FTS5 index over your agent's past conversation transcripts (e.g., Claude Code or Antigravity logs). This allows your agent to perform deep full-text searches across historical sessions to recover details not currently in `MEMORY.md`. The `.db` is purely a derived local cache—gitignored and rebuildable anytime via `okfmem index`. @@ -257,6 +257,24 @@ okfmem graduate my-slug # [y/N] confirm, then apply It distills the source page's body into the target `CLAUDE.md` (default: the project-root file; `--to /CLAUDE.md` targets a lane-scoped file, seeded on first write), mirrors the same insertion into a sibling `AGENTS.md` **only when it's a real file** — a symlinked `AGENTS.md` (e.g. `AGENTS.md -> CLAUDE.md`) already resolves through, so it's left alone — and then **archives, never deletes**, the source page: moved to `projects//archive/`, its `MEMORY.md` line dropped, and its frontmatter stamped with `graduated_to:` (target file + heading anchor + date/PR) so provenance survives and a later curate pass never re-flags or hard-deletes it. Writing outside the store is a rung-2 op, so it sits behind a `[y/N]` confirmation — skippable non-interactively, printing the exact manual command to run later. +### 8. Reindex measurement (`okfmem reindex`) +Once an index grows past one file, a few questions decide every restructuring — and none is answerable by looking at file sizes. Every mode is strictly read-only; none writes to the store. + +```bash +okfmem reindex --report # where the auto-loaded bytes actually sit +okfmem reindex --verify # did that move break anything? exit 1 if so +okfmem reindex --verify --json # same, machine-readable +okfmem reindex --budget-check # pointer lines over the per-line char budget +``` + +**`--report`** leads with the only two files a harness auto-loads — `MEMORY.md` and `STATE.md` — measured against their byte ceiling, then breaks `MEMORY.md` down **per section**. That last number is the one that matters: a 18 KB index where a single flat block holds 83% of the bytes needs that *lane split out*, while the same 18 KB spread evenly needs its hooks tightened. Total file size can't tell those apart. Pages on disk are reported too and explicitly labelled non-context: several hundred pages and a few MB contribute exactly zero at session start, so page count is never by itself a reason to prune. + +**`--verify`** walks **every** `MEMORY*.md` and accepts **both** pointer syntaxes — `[title](slug.md)` and the bare `- slug.md — hook` a lane index uses — reporting each dangling pointer *named with the index file it came from*, plus any page in no index at all. It exits non-zero on either, so it can gate a reindex. + +**`--budget-check`** counts index lines over the per-line pointer budget (150 characters), across every index and in both syntaxes. **Characters, not bytes** — the pointer convention's em-dash is one character and three bytes, and a byte count over-reports every line that carries one. It is advisory and always exits 0; the byte ceiling `--report` measures is the number that gates anything. + +The parsing is anchored on the link **target**, at both ends of the line: a pointer whose *title* starts with a filename (`- [CLAUDE.md subdir lanes](real-slug.md)`) resolves to `real-slug.md`, and a filename named in the *hook* stays prose. Retired `ck_*.md` snapshots are skipped as orphan candidates, the same way `consolidate` and `backfill` skip them — they were never indexed by design, and counting them would make `--verify` fail on most real stores for a known-benign reason. This lives in Python rather than shell on purpose — the obvious `grep | sed` version of the same check uses a GNU-only BRE that BSD `sed` ignores, so on macOS it silently reported nothing, and the `awk` version of the budget count was blind to bare pointers *and* counting bytes. A checker that fails open is worse than no checker. + ```mermaid flowchart TD subgraph Harness [AI Coding Agent] diff --git a/memory_consolidate.py b/memory_consolidate.py index 7396d05..58923dc 100644 --- a/memory_consolidate.py +++ b/memory_consolidate.py @@ -41,6 +41,10 @@ # Shared git commit+push path (pull-rebase + lock) lives beside this script. sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from memory_sync import sync_store # noqa: E402 +# Index enumeration: since #53's write-time lane routing, a page's pointer may +# live in any `MEMORY*.md`, not just the root one. One shared enumerator so +# this pass and `okfmem reindex --verify` can never disagree about the file set. +from memory_reindex import index_files # noqa: E402 SKIP_NAMES = {"MEMORY.md", "STATE.md", "CONTEXT.md"} DECAY_EXEMPT_TYPES = {"user", "feedback"} @@ -773,12 +777,20 @@ def main(): archive_page(c, today, args.dry_run) per_proj_slugs.setdefault(pdir, []).append(c["slug"]) + # Drop the archived page's pointer from EVERY index that carries it, not + # just the root `MEMORY.md`. With #53's lane routing a new page's pointer + # is written straight into a lane index, so a root-only drop leaves a + # dangling pointer behind -- and this pass runs unattended from the Stop + # hook, so the damage accumulates silently until `--verify` starts failing + # with no user action. `drop_memory_lines`'s two patterns already handle + # both pointer syntaxes; only the file set was wrong. dropped_total = 0 for pdir, slugs in per_proj_slugs.items(): - dropped_total += drop_memory_lines( - os.path.join(pdir, "MEMORY.md"), slugs, args.dry_run) + for idx in index_files(pdir): + dropped_total += drop_memory_lines( + os.path.join(pdir, idx), slugs, args.dry_run) if to_archive: - print(f"MEMORY.md lines dropped: {dropped_total}") + print(f"index lines dropped: {dropped_total}") print(f"mode: {'DRY-RUN' if args.dry_run else 'APPLY'}") diff --git a/memory_graduate.py b/memory_graduate.py index 9219680..6878b78 100644 --- a/memory_graduate.py +++ b/memory_graduate.py @@ -52,6 +52,7 @@ update_fields, ) from memory_init import _current_git_root, _load_registry, _prompt_yes_no # noqa: E402 +from memory_reindex import index_files # noqa: E402 DEFAULT_STORE = os.environ.get("OKFMEM_STORE", os.path.expanduser("~/okfmem-store")) @@ -254,7 +255,7 @@ def render_plan(plan): os.path.basename(plan["src"])) lines.append("") lines.append(f"archive: {plan['src']}") - lines.append(f" -> {dest} (MEMORY.md pointer dropped)") + lines.append(f" -> {dest} (index pointer dropped)") return "\n".join(lines) @@ -275,11 +276,18 @@ def apply_plan(plan, store, today): # split-brain (rule in both CLAUDE.md and the page, or source live beside a # duplicate archive copy) survives a throw. proj_dir = _project_dir_of(plan["src"]) - memory_path = os.path.join(proj_dir, "MEMORY.md") - memory_before = None - if os.path.isfile(memory_path): - with open(memory_path, "r", encoding="utf-8", newline="") as f: - memory_before = f.read() + # EVERY index, not just the root one: since #53's lane routing a page's + # pointer may live in any `MEMORY*.md`, so a root-only drop would leave a + # dangling pointer behind. The rollback snapshot therefore has to cover the + # same set — snapshotting one file while writing several would turn a + # failed graduate into a half-modified store, which is worse than the bug + # being fixed. + index_paths = [os.path.join(proj_dir, n) for n in index_files(proj_dir)] + memory_before = {} + for p in index_paths: + if os.path.isfile(p): + with open(p, "r", encoding="utf-8", newline="") as f: + memory_before[p] = f.read() # The archive destination is deterministic (same path archive_page would # return), computed up front so the internal-phase rollback can clean up a @@ -309,9 +317,11 @@ def apply_plan(plan, store, today): with open(dest, "w", encoding="utf-8", newline="") as f: f.write(dest_text) - dropped = drop_memory_lines(memory_path, [plan["slug"]], dry_run=False) + dropped = 0 + for p in index_paths: + dropped += drop_memory_lines(p, [plan["slug"]], dry_run=False) except Exception: - _rollback_internal(plan, dest, memory_path, memory_before) + _rollback_internal(plan, dest, memory_before) raise # --- outward, non-atomic side (rolled back on failure) --- @@ -325,19 +335,23 @@ def apply_plan(plan, store, today): with open(plan["agents_path"], "w", encoding="utf-8") as f: f.write(plan["agents_after"]) except Exception: - _rollback_apply(plan, dest, memory_path, memory_before, target_written) + _rollback_apply(plan, dest, memory_before, target_written) raise return dest, dropped -def _rollback_internal(plan, dest, memory_path, memory_before): +def _rollback_internal(plan, dest, memory_before): """Best-effort undo of the store-internal phase: drop the archived copy, - restore the live source page, and put MEMORY.md back. Remove the archive - copy FIRST, then restore the source — so a partial rollback can never leave - BOTH (the source-live-beside-a-duplicate-archive split-brain we most want to - avoid). Each undo step is guarded independently; a failure in one must not - skip the others or mask the original exception about to be re-raised.""" + restore the live source page, and put **every** index we snapshotted back. + Remove the archive copy FIRST, then restore the source — so a partial + rollback can never leave BOTH (the source-live-beside-a-duplicate-archive + split-brain we most want to avoid). Each undo step is guarded + independently; a failure in one must not skip the others or mask the + original exception about to be re-raised — which is also why the index + restore loops with a per-file guard rather than one try around the loop. + + `memory_before` maps index path -> its content before this run.""" try: if os.path.isfile(dest): os.remove(dest) @@ -348,23 +362,23 @@ def _rollback_internal(plan, dest, memory_path, memory_before): f.write(plan["src_text"]) except Exception: pass - try: - if memory_before is not None: - with open(memory_path, "w", encoding="utf-8", newline="") as f: - f.write(memory_before) - except Exception: - pass + for path, text in (memory_before or {}).items(): + try: + with open(path, "w", encoding="utf-8", newline="") as f: + f.write(text) + except Exception: + pass -def _rollback_apply(plan, dest, memory_path, memory_before, target_written): +def _rollback_apply(plan, dest, memory_before, target_written): """Best-effort undo of a partially-applied graduate: restore the live - source page, drop the archived copy, put MEMORY.md back, and revert any + source page, drop the archived copy, put every touched index back, and revert any outward CLAUDE.md/AGENTS.md write already made — so a failed apply leaves the world as if graduate never ran (no split-brain, no data loss). Each step is guarded independently; a rollback failure must not mask the original error that is about to be re-raised.""" # Store-internal: un-archive the source. - _rollback_internal(plan, dest, memory_path, memory_before) + _rollback_internal(plan, dest, memory_before) # Outward: revert CLAUDE.md (and a mirrored AGENTS.md) if we managed to # write it before failing. The mirror only runs after CLAUDE.md succeeds, # so it's only in play once target_written is True. @@ -423,7 +437,7 @@ def cmd_graduate(args): today = datetime.now(timezone.utc).date() dest, dropped = apply_plan(plan, store, today) print(f"\narchived: {dest}") - print(f"MEMORY.md lines dropped: {dropped}") + print(f"index lines dropped: {dropped}") return 0 diff --git a/memory_init.py b/memory_init.py index c5386c5..ea75175 100755 --- a/memory_init.py +++ b/memory_init.py @@ -44,6 +44,13 @@ import sys import time +# The auto-load byte ceiling is defined once, in memory_reindex (issue #54), +# and reused here for the status trigger (issue #53) rather than restated as +# a second number. memory_reindex only imports memory_init lazily inside a +# function, so this top-level import does not create a cycle. +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from memory_reindex import MEMORY_BUDGET_BYTES, page_files # noqa: E402 + # --------------------------------------------------------------------------- # Output formatting — TTY-gated color + status glyphs, ASCII-safe when piped # --------------------------------------------------------------------------- @@ -2256,10 +2263,11 @@ def update_nudge(): # --------------------------------------------------------------------------- # Per-project inventory (content half of `okfmem status`) # --------------------------------------------------------------------------- -# `MEMORY.md`'s first this-many lines auto-load into the agent; pointers past it -# silently stop reaching the model, so a project over the cap is the one signal -# `okfmem status` must surface (a `/okfmem-curate` candidate). Named, not a -# literal, so the threshold has one home. +# Historical line-count trigger (issue #40). Superseded by the byte-based +# trigger below (issue #53): a store can sit well under this line count while +# well over the byte ceiling once pointers run long, so `cmd_status` no longer +# compares against it. Left defined -- not reused for a second threshold -- +# in case another caller still wants a line-count read. MEMORY_AUTOLOAD_LINES = 200 @@ -2267,15 +2275,23 @@ def project_inventory(store): """Per-project content inventory of the store. Returns a list of - ``(name, pages, archived, memory_lines, has_state, has_archive_dir)`` + ``(name, pages, archived, memory_bytes, has_state, has_archive_dir)`` tuples, one per directory under ``/projects/``, sorted by name. Counting rules (must match the skill's intent): - * ``MEMORY.md`` and ``STATE.md`` are index/state files, NOT pages. + * Pages come from ``memory_reindex.page_files``: every index file + (``MEMORY.md`` and each ``MEMORY-.md``), ``STATE.md``, + ``CONTEXT.md`` and the retired ``ck_*.md`` snapshots are excluded. + Lane indexes are index files, not pages -- counting them inflates the + page total on exactly the stores #53's lane routing produces. * ``archived`` counts ``archive/*.md``; a MISSING ``archive/`` reports 0 but stays distinguishable via ``has_archive_dir`` (the shell version conflated the two and errored on the unquoted glob). - * ``memory_lines`` is the ``MEMORY.md`` line count (0 when absent). + * ``memory_bytes`` is the ``MEMORY.md`` size on disk (0 when absent) -- + the auto-load cost, and the byte-based restructure trigger (#53) + reads it against ``memory_reindex.MEMORY_BUDGET_BYTES``. A line count + undercounts a store where pointers run long, since bytes and lines + diverge whenever pointer length isn't uniform (#53). Pure stdlib, read-only. A missing ``projects/`` dir returns ``[]``. """ @@ -2287,27 +2303,22 @@ def project_inventory(store): d = os.path.join(root, name) if not os.path.isdir(d): continue - # glob.escape the directory so a project name containing a glob - # metacharacter can't corrupt the "*.md" match; the pattern tail stays - # literal on every platform (no shell = no unquoted-glob footgun). - pages = [ - f - for f in glob.glob(os.path.join(glob.escape(d), "*.md")) - if os.path.basename(f) not in ("MEMORY.md", "STATE.md") - ] + # Page enumeration is the engine's, not a second local rule: with #53's + # lane routing a store holds MEMORY-.md index files, and a + # root-only "not MEMORY.md/STATE.md" filter counts every one of them as + # a durable page. page_files() applies the whole rule (indexes, state, + # ck_ snapshots) that every sibling pass already uses. + pages = page_files(d) adir = os.path.join(d, "archive") archived = glob.glob(os.path.join(glob.escape(adir), "*.md")) mem = os.path.join(d, "MEMORY.md") - lines = 0 - if os.path.exists(mem): - with open(mem, "r", encoding="utf-8", errors="replace") as f: - lines = sum(1 for _ in f) + mem_bytes = os.path.getsize(mem) if os.path.exists(mem) else 0 out.append( ( name, len(pages), len(archived), - lines, + mem_bytes, os.path.exists(os.path.join(d, "STATE.md")), os.path.isdir(adir), ) @@ -2391,24 +2402,30 @@ def cmd_status(store, show_all=False, project_filter=None): shown = inv else: # Default view: the cwd's project plus any project over the auto-load - # cap (the only rows that need acting on); collapse the rest. + # byte ceiling (the only rows that need acting on); collapse the rest. + # Bytes, not lines (#53) -- a store can sit well under the old + # 200-line mark while over budget once pointers run long, since a + # long pointer costs many bytes but still just one line. shown = [ row for row in inv - if row[0] == cwd_project or row[3] > MEMORY_AUTOLOAD_LINES + if row[0] == cwd_project or row[3] > MEMORY_BUDGET_BYTES ] - for name, pages, archived, mem_lines, has_state, _has_arch in shown: + for name, pages, archived, mem_bytes, has_state, _has_arch in shown: marker = "*" if name == cwd_project else " " state = "yes" if has_state else "no" flag = "" - if mem_lines > MEMORY_AUTOLOAD_LINES: + if mem_bytes > MEMORY_BUDGET_BYTES: + # Remedy is a lane split, not tightening hooks: hook-tightening + # recovers a few hundred bytes, a lane split recovers thousands + # (issue #53). Once split, pointers are never re-flattened back. flag = ( - f" {glyph('warn')} over {MEMORY_AUTOLOAD_LINES}-line " - "auto-load limit" + f" {glyph('warn')} over {MEMORY_BUDGET_BYTES}-byte " + "auto-load ceiling -- split a lane index (/okfmem-reindex)" ) print( f" {marker} {name:20} pages:{pages:<4} " - f"MEMORY.md:{mem_lines:<4} archived:{archived:<4} " + f"MEMORY.md:{mem_bytes:<6}B archived:{archived:<4} " f"STATE:{state}{flag}" ) if project_filter is None and not show_all: diff --git a/memory_reindex.py b/memory_reindex.py new file mode 100644 index 0000000..3a0e00f --- /dev/null +++ b/memory_reindex.py @@ -0,0 +1,780 @@ +#!/usr/bin/env python3 +"""okfmem reindex -- deterministic reindex engine for a memory store project +(issue #54). + +Three read-only modes, no writes to the store (rung 1 on the confirmation +ladder -- see CLAUDE.md "Confirmation discipline"): + + okfmem reindex --report [TARGET] # where the auto-loaded bytes sit + okfmem reindex --verify [TARGET] # link integrity across MEMORY*.md + okfmem reindex --budget-check [TARGET] # pointer lines over the char budget + +`--report` leads with the only two files that reach the model at session start +(`MEMORY.md` and `STATE.md`) measured against their byte ceiling, then breaks +`MEMORY.md` down **per section** -- a single dominant block is the finding that +decides whether the remedy is "tighten a few hooks" or "split a lane", and total +file size alone never shows it. Pages on disk are reported too, explicitly +labelled non-context: they cost zero at session start. + +`--verify` walks **every** `MEMORY*.md`, accepts **both** pointer syntaxes, and +reports dangling pointers *named with the index they came from* plus pages in no +index at all. It exits non-zero when either is found, so it can gate a reindex. + +`--budget-check` counts pointer lines over the per-line character budget (#52) +across every index -- the after-the-fact check for what slipped past +`okfmem-save`'s write-time enforcement. Characters, never bytes: the pointer +convention's em-dash is one character and three bytes, so a byte count +over-reports every line that carries one. Advisory, so it always exits 0. + +Both syntaxes are first-class. The bare form is not legacy -- it is what a +graph-era lane index uses today: + + - [Human title](some-slug.md) -- hook # rich + - some-slug.md -- hook # bare + - one-slug.md + two-slug.md -- hook # bare, several under one hook + +The parsing is anchored on the **link target**, never on a loose `- \\[?` prefix, +and it is anchored at both ends of the line: + + * a rich pointer whose *title text* starts with a filename + (`- [CLAUDE.md subdir lanes](real-slug.md)`) resolves to `real-slug.md`, not + to a phantom `CLAUDE.md`; + * a filename named in the *hook* (`- real-slug.md -- MEMORY.md is the only + file loaded`) is prose, not a pointer. + +This lives in Python rather than shell precisely because the shell version of +the same check fails open on BSD `sed` (macOS). + +Usage: + okfmem reindex [--report] [--verify] [--budget-check] [TARGET] + [--project NAME] [--store PATH] [--json] [--budget BYTES] + + TARGET memory dir to inspect. Default: /projects//, + with from --project, else the cwd's git-root name + (the same rule `okfmem init`/`graduate` use). + --json machine-readable output instead of markdown (stable contract for + callers such as the /okfmem-reindex skill). + --budget override the auto-load byte ceiling for this run. + +Exit codes: + 0 clean (or --report/--budget-check, which never fail on content) + 1 --verify found dangling pointers and/or orphans + 2 usage error / target directory not found + +Pure stdlib, cross-platform, read-only. +""" +import argparse +import glob +import json +import os +import re +import sys +from collections import namedtuple + +DEFAULT_STORE = os.environ.get("OKFMEM_STORE", os.path.expanduser("~/okfmem-store")) + +# Index files: the root index plus any lane index split out of it. +INDEX_PREFIX = "MEMORY" +ROOT_INDEX = "MEMORY.md" +# Files that live in a project dir but are neither an index nor a durable page, +# so they are never orphan candidates. Same set the backfill/consolidate passes +# skip. +NON_PAGE_NAMES = {"STATE.md", "CONTEXT.md"} + +# Retired `ck_*.md` session snapshots. Every sibling pass already skips them by +# this exact prefix test (`memory_consolidate.scan_project`, +# `memory_backfill`), and curate flags them `ck_snapshot` cruft rather than +# durable pages -- so they are not orphan candidates here either. Counting them +# would make `--verify` fail on most real stores for a known-benign reason, +# and a gate nobody can turn on is not a gate. +CK_PREFIX = "ck_" + +# The only two files a harness auto-loads at session start, so the only two that +# cost context. Named here rather than as literals so the ceiling has one home +# (issue #53's byte-based restructure trigger reads from here too). 8 KiB is +# roughly 2k tokens per file -- a routing table, not a page list. +MEMORY_BUDGET_BYTES = 8192 +STATE_BUDGET_BYTES = 8192 + +# Per-line pointer budget (#52), in CHARACTERS. `okfmem-save` enforces it at +# write time; `--budget-check` is the after-the-fact sweep for what slipped +# through. One home for the number, so the skills can cite it instead of +# re-typing 150 into a shell one-liner that then drifts. +POINTER_BUDGET_CHARS = 150 + +# --- pointer syntax --------------------------------------------------------- +# Rich form. Captured loosely to the first `)` and normalized below: a slug +# never contains a paren, so the loose capture survives <>-wrapping, a +# `#fragment` and a link title without a regex that guesses at all three. +INLINE_LINK_RE = re.compile(r"\]\(([^)\n]*)\)") + +# Bare form: `- .md -- hook`. Anchored so the filename must be the FIRST +# token of the list item. `[` is deliberately not a legal first character, which +# is what stops `- [CLAUDE.md subdir lanes](real-slug.md)` from being read as a +# bare pointer to `CLAUDE.md` -- the false positive a greedy `- \[?` prefix +# match produces. +BARE_POINTER_RE = re.compile( + r"^[ \t]*[-*+][ \t]+([A-Za-z0-9][A-Za-z0-9._-]*\.md)(?=[ \t]|$)" +) + +# One list item may carry several slugs under a shared hook: +# `- slug-a.md + slug-b.md - hook`. Only `+` joins them, and only in the +# unbroken run at the head of the item -- deliberately NOT "any *.md token on +# the line". A hook routinely names a file in prose +# (`- index-is-the-only-loaded-file.md - MEMORY.md is the ONLY file loaded`), +# and reading that as a pointer is the same anchor-on-the-target failure as the +# `- \[?` prefix bug, just at the other end of the line. +BARE_CONTINUATION_RE = re.compile( + r"^[ \t]*\+[ \t]*([A-Za-z0-9][A-Za-z0-9._-]*\.md)(?=[ \t]|$)" +) + +# Reference-style link definition: `[label]: slug.md`. Spec-legal markdown, so +# a page reachable only this way must not read as an orphan. +REF_DEF_RE = re.compile(r"^[ \t]{0,3}\[[^\]]+\]:[ \t]*(.+)$") + +FENCE_RE = re.compile(r"^[ \t]{0,3}(`{3,}|~{3,})") +HEADING_RE = re.compile(r"^(#{1,6})[ \t]+(.*?)[ \t]*$") + +# target = same-dir `.md` filename; syntax = bare|inline|external; line = 1-based +Pointer = namedtuple("Pointer", "target syntax line") +Section = namedtuple("Section", "heading level bytes pointers") +Dangling = namedtuple("Dangling", "index target line syntax") + + +class ReindexError(Exception): + pass + + +# --------------------------------------------------------------------------- +# reading +# --------------------------------------------------------------------------- +def read_lines(path): + """Return ``(lines, costs)`` for a file. + + ``lines`` are decoded, newline-stripped source lines; ``costs[i]`` is the + exact byte cost of line ``i`` including its terminator, so ``sum(costs)`` + equals the file size on disk. Splitting on b"\\n" (rather than + ``str.splitlines``) keeps the two lists index-aligned -- ``splitlines`` + also breaks on \\x0b/\\x0c/U+2028, which would silently desync a pointer's + line number from the byte accounting. + """ + with open(path, "rb") as f: + raw = f.read() + segs = raw.split(b"\n") + costs = [len(s) + 1 for s in segs] + if costs: + costs[-1] -= 1 # the last segment has no trailing newline + lines = [s.decode("utf-8", "replace").rstrip("\r") for s in segs] + return lines, costs + + +def fence_mask(lines): + """``True`` for every line inside a fenced code block, delimiters included. + + A pointer-shaped line in a code fence is documentation, not a pointer; + counting it would manufacture a dangling link out of an example. + """ + mask = [False] * len(lines) + open_marker = None + for i, line in enumerate(lines): + m = FENCE_RE.match(line) + if open_marker is None: + if m: + open_marker = m.group(1) + mask[i] = True + else: + mask[i] = True + if (m and m.group(1)[0] == open_marker[0] + and len(m.group(1)) >= len(open_marker)): + open_marker = None + return mask + + +# --------------------------------------------------------------------------- +# pointer parsing +# --------------------------------------------------------------------------- +def classify_target(raw): + """Normalize a raw inline-link target to ``(kind, value)``. + + kind is ``"page"`` for a same-directory ``.md`` filename, ``"external"`` + for a URL or a cross-directory path (real, but not resolvable against this + memory dir, so never reported dangling), or ``None`` for anything that is + not a page pointer at all (an image, a bare anchor, a non-md asset). + """ + t = raw.strip() + if t.startswith("<") and ">" in t: + t = t[1:t.index(">")].strip() + else: + parts = t.split() + t = parts[0] if parts else "" + t = t.split("#", 1)[0] + if not t.lower().endswith(".md"): + return (None, "") + if "://" in t or t.lower().startswith("mailto:"): + return ("external", t) + # An explicit same-dir link (`./slug.md`, `.\slug.md`) is a page pointer, + # not a cross-directory one -- classifying it external would make the page + # it names read as an orphan. + if t.startswith("./"): + t = t[2:] + elif t.startswith(".\\"): + t = t[2:] + if "/" in t or "\\" in t: + return ("external", t) + return ("page", t) + + +def parse_pointers(lines): + """Every pointer in one index file, in document order. + + Both syntaxes are first-class and a line may carry both (a bare pointer + whose hook happens to contain a markdown link), so the two matchers run + independently rather than as alternatives of one pattern. + """ + out = [] + mask = fence_mask(lines) + for i, line in enumerate(lines): + if mask[i]: + continue + lineno = i + 1 + m = BARE_POINTER_RE.match(line) + if m: + out.append(Pointer(m.group(1), "bare", lineno)) + rest = line[m.end():] + while True: + cm = BARE_CONTINUATION_RE.match(rest) + if not cm: + break + out.append(Pointer(cm.group(1), "bare", lineno)) + rest = rest[cm.end():] + rm = REF_DEF_RE.match(line) + if rm: + kind, val = classify_target(rm.group(1)) + if kind: + out.append(Pointer(val, "inline" if kind == "page" + else "external", lineno)) + for lm in INLINE_LINK_RE.finditer(line): + kind, val = classify_target(lm.group(1)) + if kind == "page": + out.append(Pointer(val, "inline", lineno)) + elif kind == "external": + out.append(Pointer(val, "external", lineno)) + return out + + +def parse_pointers_text(text): + """``parse_pointers`` over a string -- the shape tests and callers want.""" + return parse_pointers(text.split("\n")) + + +def parse_sections(lines, costs, pointers): + """Partition a file into sections at every heading, with **own** bytes. + + Own bytes (heading line through the line before the next heading of *any* + level) rather than subtree bytes, so the rows partition the file exactly: + they sum to the file size and their percentages sum to 100. A subtree + total would double-count a parent and hide which block is actually big -- + the one number this report exists to surface. + """ + mask = fence_mask(lines) + starts = [] # (line_index, heading_text, level) + for i, line in enumerate(lines): + if mask[i]: + continue + m = HEADING_RE.match(line) + if m: + starts.append((i, m.group(2), len(m.group(1)))) + + bounds = [] + if not starts or starts[0][0] > 0: + bounds.append((0, "(preamble)", 0)) + bounds.extend(starts) + + out = [] + for n, (start, heading, level) in enumerate(bounds): + end = bounds[n + 1][0] if n + 1 < len(bounds) else len(lines) + nbytes = sum(costs[start:end]) + npointers = sum(1 for p in pointers + if p.syntax != "external" and start < p.line <= end) + out.append(Section(heading, level, nbytes, npointers)) + return out + + +# --------------------------------------------------------------------------- +# directory scan +# --------------------------------------------------------------------------- +def _md_names(mem_dir): + return sorted( + os.path.basename(p) + for p in glob.glob(os.path.join(glob.escape(mem_dir), "*.md")) + if os.path.isfile(p) + ) + + +def is_index_name(name): + return name.startswith(INDEX_PREFIX) and name.endswith(".md") + + +def index_files(mem_dir): + """Every ``MEMORY*.md`` in the dir, root index first then lanes sorted.""" + names = [n for n in _md_names(mem_dir) if is_index_name(n)] + root = [ROOT_INDEX] if ROOT_INDEX in names else [] + return root + [n for n in names if n != ROOT_INDEX] + + +def is_ck_snapshot(name): + """A retired ck session snapshot -- cruft, not a durable page. Same test + the consolidate and backfill passes use (`startswith("ck_")`).""" + return name.startswith(CK_PREFIX) + + +def page_files(mem_dir, include_ck=False): + """Durable pages: every top-level ``*.md`` that is neither an index nor + state/context nor a retired ``ck_*.md`` snapshot. These are the orphan + candidates. + + ``include_ck=True`` adds the ck snapshots back in, for the one caller that + wants to *see* them: the curate inventory flags them ``ck_snapshot`` so a + curate pass can propose deleting them. Every integrity check wants them + skipped -- they were never indexed and never will be. + """ + return [n for n in _md_names(mem_dir) + if not is_index_name(n) and n not in NON_PAGE_NAMES + and (include_ck or not is_ck_snapshot(n))] + + +class Scan(object): + """Everything both modes need, collected in one pass over the dir.""" + + def __init__(self, mem_dir): + self.dir = mem_dir + self.indexes = index_files(mem_dir) + self.pages = page_files(mem_dir) + self.lines = {} + self.costs = {} + self.pointers = {} # index -> [Pointer] (page targets only) + self.external = {} # index -> [Pointer] (cross-dir / URL targets) + for idx in self.indexes: + lines, costs = read_lines(os.path.join(mem_dir, idx)) + self.lines[idx] = lines + self.costs[idx] = costs + found = parse_pointers(lines) + self.pointers[idx] = [p for p in found if p.syntax != "external"] + self.external[idx] = [p for p in found if p.syntax == "external"] + + present = set(_md_names(mem_dir)) + + # dangling: target missing on disk, attributed to the index it came + # from. Per-index attribution is the whole point -- "something is + # dangling somewhere" is not actionable. + self.dangling = [] + for idx in self.indexes: + seen = set() + for p in self.pointers[idx]: + if p.target in present or (p.target, p.line) in seen: + continue + seen.add((p.target, p.line)) + self.dangling.append(Dangling(idx, p.target, p.line, p.syntax)) + + # linked-from map, for orphans and for spotting a half-finished move. + self.linked_from = {} + for idx in self.indexes: + for p in self.pointers[idx]: + self.linked_from.setdefault(p.target, set()).add(idx) + + self.orphans = [p for p in self.pages if p not in self.linked_from] + + # An unreferenced lane index is an orphan too -- nothing opens it, so + # every pointer it holds is unreachable. The root index is exempt: it + # is the auto-loaded entry point, referenced by nothing by design. + # A self-reference does not count as being referenced. + self.orphan_indexes = [ + idx for idx in self.indexes + if idx != ROOT_INDEX + and not (self.linked_from.get(idx, set()) - {idx}) + ] + + # Pointed at from more than one index: legal, but the usual shape of a + # half-finished move (new pointer added, old one never removed). + self.duplicates = {t: sorted(s) for t, s in self.linked_from.items() + if len(s) > 1 and t in present and not is_index_name(t)} + + @property + def ok(self): + return not (self.dangling or self.orphans or self.orphan_indexes) + + def index_bytes(self, idx): + return sum(self.costs[idx]) + + def pointer_counts(self, idx): + bare = sum(1 for p in self.pointers[idx] if p.syntax == "bare") + inline = sum(1 for p in self.pointers[idx] if p.syntax == "inline") + return bare, inline + + +# --------------------------------------------------------------------------- +# target resolution +# --------------------------------------------------------------------------- +def resolve_target(args): + """The memory dir to inspect. Explicit path wins; else + ``/projects/``, with the project resolved the same way + `okfmem graduate` does (``--project``, else the cwd's git-root name, + registry overrides applied).""" + if args.target: + d = os.path.abspath(os.path.expanduser(args.target)) + if not os.path.isdir(d): + raise ReindexError(f"target directory not found: {d}") + return d + + store = os.path.abspath(os.path.expanduser(args.store)) + project = args.project + if not project: + # Imported lazily: the parsing helpers above are imported by the + # curate inventory script, which has no business loading the init + # module just to read an index file. + sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + from memory_init import _current_git_root, _load_registry + root = _current_git_root() + if not root: + raise ReindexError( + "no TARGET given and cwd is not inside a git repo -- pass a " + "memory dir, or --project NAME") + reg = _load_registry(os.path.join(store, "registry.json")) + project = reg.get("overrides", {}).get(root, os.path.basename(root)) + + d = os.path.join(store, "projects", project) + if not os.path.isdir(d): + raise ReindexError(f"no store project dir for '{project}': {d}") + return d + + +# --------------------------------------------------------------------------- +# report +# --------------------------------------------------------------------------- +def _pct(part, whole): + return (100.0 * part / whole) if whole else 0.0 + + +def _budget_row(name, path, budget): + exists = os.path.isfile(path) + nbytes = os.path.getsize(path) if exists else 0 + return {"file": name, "exists": exists, "bytes": nbytes, + "budget": budget, "over": nbytes > budget, + "pct_of_budget": round(_pct(nbytes, budget), 1)} + + +def report_data(scan, budget=None): + """``--budget`` overrides the ceiling for both auto-loaded files -- they + share one context budget, so splitting the override would let a caller + tighten one and silently leave the other at the default.""" + mem_budget = MEMORY_BUDGET_BYTES if budget is None else budget + st_budget = STATE_BUDGET_BYTES if budget is None else budget + d = scan.dir + + auto = [ + _budget_row(ROOT_INDEX, os.path.join(d, ROOT_INDEX), mem_budget), + _budget_row("STATE.md", os.path.join(d, "STATE.md"), st_budget), + ] + + sections = [] + if ROOT_INDEX in scan.indexes: + total = scan.index_bytes(ROOT_INDEX) + for s in parse_sections(scan.lines[ROOT_INDEX], scan.costs[ROOT_INDEX], + scan.pointers[ROOT_INDEX]): + sections.append({"heading": s.heading, "level": s.level, + "bytes": s.bytes, "pointers": s.pointers, + "pct": round(_pct(s.bytes, total), 1)}) + + indexes = [] + for idx in scan.indexes: + bare, inline = scan.pointer_counts(idx) + indexes.append({"file": idx, "bytes": scan.index_bytes(idx), + "pointers": bare + inline, "bare": bare, + "inline": inline, + "external_refs": len(scan.external[idx]), + "auto_loaded": idx == ROOT_INDEX}) + + page_bytes = sum(os.path.getsize(os.path.join(d, p)) for p in scan.pages) + archived = len(glob.glob(os.path.join(glob.escape(d), "archive", "*.md"))) + return {"mode": "report", "dir": d, "auto_loaded": auto, + "sections": sections, "indexes": indexes, + "on_disk": {"pages": len(scan.pages), "bytes": page_bytes, + "archived": archived, "auto_loaded": False, + "note": "read on demand; costs zero context at " + "session start"}} + + +def render_report(data): + out = ["# Reindex report: " + data["dir"], ""] + out.append("## Auto-loaded bytes (the entire context cost)") + out.append("") + out.append("| File | Bytes | Ceiling | % of ceiling | Status |") + out.append("|---|---|---|---|---|") + for r in data["auto_loaded"]: + if not r["exists"]: + out.append(f"| {r['file']} | (absent) | {r['budget']} | - | - |") + continue + flag = "OVER" if r["over"] else "ok" + out.append(f"| {r['file']} | {r['bytes']} | {r['budget']} | " + f"{r['pct_of_budget']}% | {flag} |") + out.append("") + + out.append("## MEMORY.md, per section") + out.append("") + if not data["sections"]: + out.append("_No MEMORY.md in this directory._") + else: + out.append("Own bytes per section (a heading row excludes its " + "subheadings), so the rows partition the file and the " + "percentages sum to 100.") + out.append("") + out.append("| Section | Bytes | % of file | Pointers |") + out.append("|---|---|---|---|") + for s in data["sections"]: + # plain spaces, not ` `: this is read in a terminal at least + # as often as it is rendered, and the entity is noise in both. + indent = " " * max(0, s["level"] - 1) + head = s["heading"].replace("|", "\\|") + out.append(f"| {indent}{head} | {s['bytes']} | {s['pct']}% | " + f"{s['pointers']} |") + out.append("") + + out.append("## Indexes") + out.append("") + out.append("| Index | Bytes | Pointers | rich | bare | auto-loaded |") + out.append("|---|---|---|---|---|---|") + for i in data["indexes"]: + out.append(f"| {i['file']} | {i['bytes']} | {i['pointers']} | " + f"{i['inline']} | {i['bare']} | " + f"{'yes' if i['auto_loaded'] else 'no'} |") + out.append("") + + od = data["on_disk"] + out.append("## On disk (NOT auto-loaded -- zero context cost)") + out.append("") + out.append(f"- **Pages**: {od['pages']} ({od['bytes']} bytes)") + out.append(f"- **Archived**: {od['archived']}") + out.append(f"- These are {od['note']}. Only the auto-loaded table above " + "spends context.") + return "\n".join(out) + + +# --------------------------------------------------------------------------- +# verify +# --------------------------------------------------------------------------- +def verify_data(scan): + return { + "mode": "verify", + "dir": scan.dir, + "indexes": [{"file": i, "pointers": len(scan.pointers[i])} + for i in scan.indexes], + "pages": len(scan.pages), + "dangling": [{"index": d.index, "target": d.target, "line": d.line, + "syntax": d.syntax} for d in scan.dangling], + "orphans": list(scan.orphans), + "orphan_indexes": list(scan.orphan_indexes), + "duplicates": scan.duplicates, + "external_refs": sum(len(v) for v in scan.external.values()), + "ok": scan.ok, + } + + +def render_verify(data): + out = ["# Reindex verify: " + data["dir"], ""] + out.append(f"- indexes: {len(data['indexes'])} " + f"({', '.join(i['file'] for i in data['indexes']) or 'none'})") + out.append(f"- pointers: {sum(i['pointers'] for i in data['indexes'])} " + f"across {len(data['indexes'])} index file(s)") + out.append(f"- pages: {data['pages']}") + out.append(f"- dangling: {len(data['dangling'])}") + out.append(f"- orphans: {len(data['orphans'])}") + out.append(f"- orphan indexes: {len(data['orphan_indexes'])}") + if data["external_refs"]: + out.append(f"- external/cross-dir refs (not checked): " + f"{data['external_refs']}") + out.append("") + + if data["dangling"]: + out.append("## Dangling pointers (target file does not exist)") + out.append("") + out.append("| Index | Line | Target | Syntax |") + out.append("|---|---|---|---|") + for d in data["dangling"]: + out.append(f"| {d['index']} | {d['line']} | {d['target']} | " + f"{d['syntax']} |") + out.append("") + + if data["orphans"]: + out.append("## Orphan pages (exist on disk, in no index)") + out.append("") + for p in data["orphans"]: + out.append(f"- {p}") + out.append("") + + if data["orphan_indexes"]: + out.append("## Orphan indexes (no other index points at them)") + out.append("") + for p in data["orphan_indexes"]: + out.append(f"- {p}") + out.append("") + + if data["duplicates"]: + out.append("## Pointed at from more than one index (informational)") + out.append("") + for t, idxs in sorted(data["duplicates"].items()): + out.append(f"- {t} -- {', '.join(idxs)}") + out.append("") + + out.append("OK: index is intact." if data["ok"] + else "FAIL: index is not intact (see above).") + return "\n".join(out) + + +# --------------------------------------------------------------------------- +# budget check +# --------------------------------------------------------------------------- +def budget_data(scan, limit=None): + """Pointer lines over the per-line character budget, across every index. + + The unit is the *line*, not the pointer: the budget is a budget on the + index line a session reads, and one list item may legally carry several + slugs under a shared hook. + + Two things this deliberately does differently from the shell one-liner it + replaces (which was blind to both): + + * **Both syntaxes.** A `/^- \\[/` guard sees only the rich form, and the + bare form is exactly what a lane index uses -- i.e. it missed the + pointers most likely to be over budget. + * **Characters, not bytes.** `len` on a decoded line; BSD `awk`'s + `length($0)` counts bytes, so the convention's em-dash alone + over-reports by 2 per occurrence. + + A routing-table row (an index pointing at another index, `- MEMORY-lane.md + -- covers ...`) is counted separately and not held to this budget: its + "covers" clause is deliberately a sentence or two, so measuring it against + the page-pointer budget would flag every correctly-reindexed store. It is + reported, never silently dropped. + """ + limit = POINTER_BUDGET_CHARS if limit is None else limit + over, checked, routing = [], 0, 0 + for idx in scan.indexes: + src = scan.lines[idx] + by_line = {} + for p in scan.pointers[idx]: + by_line.setdefault(p.line, []).append(p.target) + for lineno in sorted(by_line): + targets = by_line[lineno] + if all(is_index_name(t) for t in targets): + routing += 1 + continue + checked += 1 + nchars = len(src[lineno - 1]) + if nchars > limit: + over.append({"index": idx, "line": lineno, "chars": nchars, + "targets": targets}) + return {"mode": "budget", "dir": scan.dir, "limit": limit, + "pointer_lines": checked, "routing_rows": routing, + "over": over, "ok": not over} + + +def render_budget(data): + out = ["# Pointer budget: " + data["dir"], ""] + out.append(f"{len(data['over'])}/{data['pointer_lines']} pointer lines " + f"over the {data['limit']}-char budget " + f"(characters, not bytes).") + if data["routing_rows"]: + out.append(f"({data['routing_rows']} routing-table row(s) point at " + "another index and are not held to this budget.)") + out.append("") + if data["over"]: + out.append("| Index | Line | Chars | Points at |") + out.append("|---|---|---|---|") + for r in data["over"]: + out.append(f"| {r['index']} | {r['line']} | {r['chars']} | " + f"{', '.join(r['targets'])} |") + out.append("") + out.append("Tighten the hook: move detail into the page body, where it " + "costs nothing. Advisory -- this mode never exits non-zero.") + else: + out.append("OK: every pointer line is within budget.") + return "\n".join(out) + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- +def build_parser(): + p = argparse.ArgumentParser( + prog="okfmem reindex", + description="Read-only reindex engine: byte report + link-integrity " + "verification across every MEMORY*.md.") + p.add_argument("target", nargs="?", metavar="TARGET", + help="memory dir (default: /projects/)") + p.add_argument("--report", action="store_true", + help="where the auto-loaded bytes sit (the default)") + p.add_argument("--verify", action="store_true", + help="link integrity; exits 1 on dangling/orphans") + p.add_argument("--budget-check", action="store_true", dest="budget_check", + help=f"pointer lines over the {POINTER_BUDGET_CHARS}-char " + "budget, across every index (advisory; exits 0)") + p.add_argument("--project", help="store project name") + p.add_argument("--store", default=DEFAULT_STORE, help="store path") + p.add_argument("--json", action="store_true", dest="as_json", + help="machine-readable output") + p.add_argument("--budget", type=int, default=None, + help=f"auto-load byte ceiling (default " + f"{MEMORY_BUDGET_BYTES})") + return p + + +def main(argv=None): + argv = list(sys.argv[1:] if argv is None else argv) + # Tolerate the subcommand token when this module is reached other than via + # the dispatcher's runpy call (which already strips it). + if argv and argv[0] == "reindex": + argv = argv[1:] + args = build_parser().parse_args(argv) + + try: + mem_dir = resolve_target(args) + except ReindexError as e: + print(f"okfmem reindex: {e}", file=sys.stderr) + return 2 + + do_verify = args.verify + do_budget = args.budget_check + # no mode flag -> report + do_report = args.report or not (do_verify or do_budget) + + scan = Scan(mem_dir) + chunks = [] + payload = {} + if do_report: + data = report_data(scan, budget=args.budget) + payload["report"] = data + chunks.append(render_report(data)) + if do_verify: + data = verify_data(scan) + payload["verify"] = data + chunks.append(render_verify(data)) + if do_budget: + data = budget_data(scan) + payload["budget"] = data + chunks.append(render_budget(data)) + + if args.as_json: + # One mode -> that mode's object, so a caller asking for --verify gets + # the documented shape rather than a wrapper it has to unpack. + out = next(iter(payload.values())) if len(payload) == 1 else payload + print(json.dumps(out, indent=2, sort_keys=True)) + else: + print("\n\n".join(chunks)) + + if do_verify and not scan.ok: + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/okfmem b/okfmem index 56d0b1b..7d7ea21 100755 --- a/okfmem +++ b/okfmem @@ -11,6 +11,11 @@ Usage: okfmem status # alias for `init --status` okfmem graduate [--to PATH] [--dry-run] # promote a page into CLAUDE.md/ # AGENTS.md + archive the source — #10 + okfmem reindex [--report|--verify|--budget-check] [DIR] + # read-only: where the auto-loaded + # bytes sit, link integrity across + # every MEMORY*.md, and pointer + # lines over the char budget — #54 Core subcommands map to the memory_*.py modules living beside this script. Store is resolved by each module: --store, else $OKFMEM_STORE, else ~/okfmem-store. @@ -31,7 +36,7 @@ HERE = os.path.dirname(os.path.realpath(__file__)) # realpath: resolve symlink MODULES = {"backfill": "memory_backfill.py", "init": "memory_init.py", "consolidate": "memory_consolidate.py", "sync": "memory_sync.py", "pull": "memory_pull.py", "update": "memory_update.py", - "graduate": "memory_graduate.py"} + "graduate": "memory_graduate.py", "reindex": "memory_reindex.py"} # Opt-in plugin commands -> plugins/. Loaded ONLY if the file is present, # so core install stays harness-agnostic. Several commands may share one plugin. PLUGINS = {"index": "memory_search.py", "search": "memory_search.py", diff --git a/skills/okfmem-curate/SKILL.md b/skills/okfmem-curate/SKILL.md index 502914b..db34860 100644 --- a/skills/okfmem-curate/SKILL.md +++ b/skills/okfmem-curate/SKILL.md @@ -24,13 +24,15 @@ origin: user Curate the per-project auto-memory store under `~/.claude/projects//memory/`. Detects stale, superseded, or duplicate-with-CLAUDE.md entries; proposes a deletion/compression plan; on approval, executes and rewrites `MEMORY.md` as tight one-line hooks per the user's auto-memory convention. +**Page deletion and archival are store hygiene and recall precision — not a context optimization (#52).** Only `MEMORY.md` and `STATE.md` are auto-loaded at session start; every other page costs zero context regardless of how many exist. A curate pass that deletes or archives 100 pages and touches no pointer saves **0** tokens. The number that moves the needle is auto-loaded bytes — see Phase 2 and Phase 4. + Applies the "deterministic collection + LLM judgment" principle: a script collects facts, then an LLM cross-reads each candidate and produces verdicts. **Hard rule:** no file is deleted and no index is rewritten until the user has explicitly approved the plan. ## When to use - The user says "clean up memory", "prune memory", "memory hygiene", "tighten MEMORY.md", "context is bloated", or similar. - Periodic curation (monthly, or after a project reaches a milestone where many "X landed" memories accumulate). -- After noticing MEMORY.md exceeds ~200 lines (the auto-load truncation point) or its size has grown well beyond ~10KB. +- **Not** the byte-ceiling trigger — when `okfmem status` or `okfmem reindex --report` flags `MEMORY.md` as over the auto-load byte ceiling (`memory_reindex.MEMORY_BUDGET_BYTES`, 8192 bytes), that is `/okfmem-reindex`'s trigger, not this skill's: the remedy is a lane split (a pointer move), not a deletion. Phase 2 below still surfaces the same numbers, because a curate pass benefits from knowing them too, but this skill does not execute the split. - **Audit-only check-in**: when the user wants the report without committing to deletions yet — invoke with `audit` argument. ## Modes @@ -66,22 +68,78 @@ If the link points into a git-backed location, surface that to the user as the r ### Phase 2: Inventory (deterministic) -Run the inventory script: +**Auto-loaded bytes — the headline numbers.** Run the reindex engine first. +Its `auto_loaded` table is the only thing that costs context at session start +(`MEMORY.md` + `STATE.md` against their ceiling), and its per-section +breakdown of `MEMORY.md` shows **which section holds the bytes** — a single +dominant block is the finding that decides whether the remedy is "tighten a +few hooks" or "split a lane"; total file size alone never shows it: + +```bash +python3 ~/okfmem/okfmem reindex --report "$MEM_DIR" +``` + +**If the `MEMORY.md` row reads `OVER`, the restructure trigger has fired +(#53) — but that trigger is `/okfmem-reindex`'s to act on, not this +skill's.** The recommended remedy is a lane split, never "tighten hooks": +moving a lane's pointers out of `MEMORY.md` into a new or existing +`MEMORY-.md` index recovers far more bytes than shortening a handful +of hooks, and once a pointer moves to a lane index it is never re-flattened +back to the root. `/okfmem-reindex` clusters the split, proposes it for +approval, and executes it — see that skill for the full process. Report +the `OVER` finding here (it belongs in this skill's numbers too, since a +curate pass often runs on the same store), but hand the split itself to +`/okfmem-reindex` rather than doing it inline. + +Also count pointer lines over the per-line budget (#52 — `okfmem-save` +enforces ≤150 chars at write time; this is the check that catches what +slipped through): + +```bash +python3 ~/okfmem/okfmem reindex --budget-check "$MEM_DIR" +``` + +It walks **every** index, not just the root — after a lane split (or once +#53's write-time routing is in effect) most pointers live in lane indexes, +so checking `MEMORY.md` alone would miss almost all of them — and it accepts +**both** pointer syntaxes, so the bare `- slug.md — hook` form a lane index +uses is counted too. It reports each over-budget line with its index file, +line number, and character count, and it is **advisory**: always exit 0, +never a gate (the budget rule itself is advisory — see `okfmem-save` Step 3). + +> Do not hand-roll this as a shell one-liner. The `awk` version this replaced +> was wrong in both directions at once: a `/^- \[/` guard is blind to every +> bare pointer (exactly the ones a lane index holds), and BSD `awk`'s +> `length($0)` counts **bytes**, so the convention's em-dash over-reported +> every line carrying one. Same fail-open-on-macOS class as the old `sed` +> verifier below. The budget constant lives in one place — +> `memory_reindex.POINTER_BUDGET_CHARS`. + +Lead the curation report (Phase 4) with these numbers: `MEMORY.md` / +`STATE.md` bytes vs. ceiling, and pointer overage count. + +**Page-level detail — store hygiene, not context.** Run the inventory script +for per-file age, link status, and heuristic flags: ```bash python3 ~/okfmem/skills/okfmem-curate/scripts/inventory.py "$MEM_DIR" ``` -The script emits a markdown report with three sections: +Page count and on-disk total bytes are **not auto-loaded and cost zero +context at session start** — demote them to context for the plan, not a +finding. Deleting or archiving a page changes none of the headline numbers +above unless it also removes or shortens a `MEMORY.md` pointer. + +The inventory script emits a markdown report with three sections: -1. **Summary**: file count, total bytes, MEMORY.md size + line count, ratio. -2. **Per-file table**: name, size, age (days since mtime), frontmatter `type` if present, frontmatter `name`, MEMORY.md link status (linked / orphan / dangling). +1. **Summary**: page count, total bytes, MEMORY.md size + line count, per-index pointer counts. +2. **Per-file table**: name, size, age (days since mtime), frontmatter `type` if present, frontmatter `name`, link status (linked / orphan). Link status is computed across **every** `MEMORY*.md` in both pointer syntaxes — a page carried by a lane index is linked, not an orphan. 3. **Heuristic flags**: per file, comma-separated flags drawn from filename + content patterns: - - `ck_snapshot` — filename matches `ck_YYYY-MM-DD_*.md` (CK session-end saves; ephemeral by nature) + - `ck_snapshot` — filename starts `ck_` (retired CK session-end saves; ephemeral by nature). These are *never* counted as orphans — they were never indexed by design, and every engine pass (`consolidate`, `backfill`, `reindex --verify`) skips them for the same reason. This skill is the one pass that still surfaces them, so a plan can propose deleting them. - `landed_doc` — filename or frontmatter `name` contains `landed` / `_operational` (project-state, prone to age out) - `superseded_marker` — body contains `SUPERSEDED`, `superseded by`, `replaced by`, or `(Note:` markers indicating self-deprecation - - `orphan` — exists but not linked from MEMORY.md - - `dangling` — linked from MEMORY.md but file missing + - `orphan` — a durable page that exists but is linked from no index (`MEMORY.md` *or* any `MEMORY-*.md` lane index); `ck_*.md` snapshots are excluded + - `dangling` — linked from an index but file missing (reported with the index it came from) - `old_45d` — mtime older than 45 days - `old_90d` — mtime older than 90 days @@ -141,7 +199,7 @@ Recommended action per file: **graduate** a still-valuable page (promote into `C |---|---| | ... | ... | -**Net effect:** -N files, MEMORY.md tightens from XKB to ~YKB, ~Z fewer tokens auto-loaded per session. +**Net effect:** ~Z fewer tokens auto-loaded per session (MEMORY.md XKB → ~YKB; STATE.md unchanged unless also rewritten) — reads **0** if no auto-loaded file changes size. -N files archived/deleted is a store-hygiene count, reported separately; it is not a context saving by itself. → Approve as-is, or call out files to keep, before I touch anything. ``` @@ -165,18 +223,31 @@ Once approved: 6. Run the verification block: ```bash -cd "$MEM_DIR" -echo "=== file count ==="; ls *.md | wc -l -echo "=== MEMORY.md size + lines ==="; wc -c -l MEMORY.md -echo "=== link integrity ===" -grep -oE '\]\([A-Za-z][A-Za-z0-9_-]*\.md\)' MEMORY.md | sed 's/[)(]//g; s/^]//' | while read f; do - [ -f "$f" ] || echo "MISSING: $f" -done -echo "=== orphans (existing files not linked from MEMORY.md) ===" -comm -23 <(ls *.md | sort) <({ echo MEMORY.md; grep -oE '\]\([A-Za-z][A-Za-z0-9_-]*\.md\)' MEMORY.md | sed 's/[)(]//g; s/^]//'; } | sort) +python3 ~/okfmem/okfmem reindex --verify "$MEM_DIR"; echo "verify exit: $?" +python3 ~/okfmem/okfmem reindex --report "$MEM_DIR" # after-numbers for step 8 ``` -If any `MISSING:` lines appear, the new MEMORY.md is broken — fix immediately. If orphans appear, decide per file: add the link back, or delete the orphan (with user confirmation). +`--verify` walks **every** `MEMORY*.md` (not just the root index) and accepts +**both** pointer syntaxes — `[title](slug.md)` and the bare `- slug.md — hook` +a lane index uses. It exits **0** when the index is intact and **1** on any +dangling pointer or orphan, so this line is a real gate: a non-zero exit means +the rewrite broke the index — fix it before reporting success. Each dangling +pointer is named **with the index file it came from**, which is what makes it +fixable. + +> This used to be a `grep | sed | comm` pipeline, and it was wrong in the +> unsafe direction: it read only `MEMORY.md`, only the rich link form, and its +> BRE `\?` is a GNU extension that BSD `sed` (macOS) does not support — so on +> the primary platform it silently failed to strip the prefix and reported +> nothing. A verifier that fails open is worse than no verifier. The logic +> lives in Python now (issue #54) precisely so it cannot diverge per platform. +> If you ever hand-roll this again: anchor on the **link target**, never a +> loose `- \[\?` prefix, or a pointer whose *title* starts with a filename +> (`- [CLAUDE.md subdir lanes](real-slug.md)`) reports a phantom `CLAUDE.md`. + +If any dangling pointers appear, the new MEMORY.md is broken — fix immediately. +If orphans appear, decide per file: add the link back, or delete the orphan +(with user confirmation). 7. Verify each **graduate** with the same rigor as delete/compress — confirm the rule actually landed and the source actually moved: @@ -194,7 +265,7 @@ done A `MISSING:`/`BROKEN:` line here means the graduate did not apply (commonly: `--yes` was omitted so the non-interactive `[y/N]` silently skipped) — re-run `okfmem graduate --yes` and re-verify before rewriting MEMORY.md's counts. -8. Report the final numbers: files before/after, MEMORY.md bytes before/after, estimated tokens saved per session (~bytes/3.5). +8. Report the final numbers from the step-6 `okfmem reindex --report` run, in this order: (1) `MEMORY.md` + `STATE.md` bytes before/after against the ceiling and the resulting tokens saved per session (~bytes delta / 3.5) — the number that matters; (2) pages before/after, labelled non-context — store hygiene, not a context saving. "We deleted N files" must never be presented as a context saving on its own; say so only alongside a `MEMORY.md`/`STATE.md` byte drop. ## Recovery diff --git a/skills/okfmem-curate/scripts/inventory.py b/skills/okfmem-curate/scripts/inventory.py index d16ae02..1d7e156 100644 --- a/skills/okfmem-curate/scripts/inventory.py +++ b/skills/okfmem-curate/scripts/inventory.py @@ -10,6 +10,14 @@ Deterministic only: this script flags candidates; the LLM decides verdicts. +Link status comes from the engine's shared reindex parser (`memory_reindex`), +so it walks **every** `MEMORY*.md` and accepts **both** pointer syntaxes. Parsing +only the root index in only the rich `[title](slug.md)` form -- what this script +used to do -- reported every page carried by a lane index as an orphan, and, far +worse, made a genuinely dangling pointer *inside a lane index* invisible (issue +#54). For the byte-level picture (auto-loaded budget, per-section breakdown), +run `okfmem reindex --report `. + Pure stdlib, cross-platform (no `stat -f`/`stat -c`/GNU `date -d` shell-outs, so this runs the same on macOS, Linux, and Windows). """ @@ -18,7 +26,42 @@ import sys import time -LINK_RE = re.compile(r"\]\(([A-Za-z][A-Za-z0-9_-]*\.md)\)") +# The engine modules normally live at the repo root, four levels up from this +# script (skills/okfmem-curate/scripts/) — realpath first, so that resolves the +# same whether the harness symlinked the file, the skill dir, or neither. +# +# But the tier-3 managed-copy install (`memory_init._make_link`'s copytree +# fallback, used on Windows without symlink privilege or junction support) +# copies the skill dir OUT of the repo, and four-up then lands on the harness +# skills parent, which holds no engine modules at all. So probe candidates in +# order and only import once one actually contains the engine; a miss prints +# one line, never a traceback. +_ENGINE_MODULE = "memory_reindex.py" +_HOME_ENGINE = os.path.expanduser("~/okfmem") + + +def _find_engine_root(): + four_up = os.path.dirname(os.path.dirname(os.path.dirname( + os.path.dirname(os.path.realpath(__file__))))) + for cand in (four_up, os.environ.get("OKFMEM_ENGINE"), _HOME_ENGINE): + if cand and os.path.isfile(os.path.join(cand, _ENGINE_MODULE)): + return cand + return None + + +_ENGINE_ROOT = _find_engine_root() +if _ENGINE_ROOT and _ENGINE_ROOT not in sys.path: + sys.path.insert(0, _ENGINE_ROOT) +try: + import memory_reindex # noqa: E402 +except ImportError: + sys.stderr.write( + "inventory.py: okfmem engine not found (no %s beside this skill, in " + "$OKFMEM_ENGINE, or in ~/okfmem) — run the engine copy instead: " + "python3 ~/okfmem/skills/okfmem-curate/scripts/inventory.py " + "\n" % _ENGINE_MODULE) + sys.exit(2) + FM_KEY_RE_TMPL = r'^{key}:\s*(.*)$' CK_SNAPSHOT_RE = re.compile(r"^ck_\d{4}-\d{2}-\d{2}_.*\.md$") LANDED_RE = re.compile(r"(landed|operational)", re.IGNORECASE) @@ -54,19 +97,16 @@ def fm_field(text, key): return "" -def linked_filenames(memory_dir): - memory_md = os.path.join(memory_dir, "MEMORY.md") - if not os.path.isfile(memory_md): - return set() - text = read_text(memory_md) - return set(LINK_RE.findall(text)) - - def list_md_files(memory_dir): - files = [f for f in os.listdir(memory_dir) - if f.endswith(".md") and f != "MEMORY.md" - and os.path.isfile(os.path.join(memory_dir, f))] - return sorted(files) + """Every file this report has something to say about: durable pages plus + the retired `ck_*.md` snapshots. `MEMORY*.md` indexes and STATE/CONTEXT are + excluded, so they can never be counted as orphans of themselves. + + ck snapshots are included here but NOT in the engine's orphan set: curate + is the one pass that wants to see them (it flags them `ck_snapshot` so a + plan can propose deleting them), while every integrity check skips them. + """ + return memory_reindex.page_files(memory_dir, include_ck=True) def file_age_days(path, today_epoch_days): @@ -85,8 +125,9 @@ def main(): sys.exit(1) today_epoch_days = int(time.time() // 86400) + scan = memory_reindex.Scan(mem_dir) files = list_md_files(mem_dir) - linked = linked_filenames(mem_dir) + linked = set(scan.linked_from) total_bytes = sum(os.path.getsize(os.path.join(mem_dir, f)) for f in files) memory_md_path = os.path.join(mem_dir, "MEMORY.md") @@ -98,20 +139,31 @@ def main(): # len(splitlines()) which would over-count it by one. mem_lines = read_text(memory_md_path).count("\n") - orphan_count = sum(1 for f in files if f not in linked) - dangling = sorted(f for f in linked - if not os.path.isfile(os.path.join(mem_dir, f))) + orphan_count = len(scan.orphans) + dangling = scan.dangling out = [] out.append(f"# Memory inventory: {mem_dir}") out.append("") out.append("## Summary") out.append("") - out.append(f"- **Files** (excl. MEMORY.md): {len(files)}") + ck_files = [f for f in files if memory_reindex.is_ck_snapshot(f)] + out.append(f"- **Pages** (excl. MEMORY*.md / STATE.md): {len(files)}" + + (f" (of which {len(ck_files)} retired ck_*.md snapshots)" + if ck_files else "")) out.append(f"- **Total bytes**: {total_bytes}") out.append(f"- **MEMORY.md**: {mem_bytes} bytes, {mem_lines} lines") - out.append(f"- **Orphans** (file exists, not linked): {orphan_count}") - out.append(f"- **Dangling** (linked, file missing): {len(dangling)}") + out.append(f"- **Indexes** ({len(scan.indexes)}): " + + ", ".join(f"{i} ({len(scan.pointers[i])} pointers)" + for i in scan.indexes)) + out.append(f"- **Orphans** (file exists, in no index): {orphan_count}" + + (" — ck_*.md snapshots excluded; they were never indexed and " + "are flagged `ck_snapshot` below instead" if ck_files + else "")) + out.append(f"- **Dangling** (in an index, file missing): {len(dangling)}") + if scan.orphan_indexes: + out.append("- **Orphan indexes** (no other index points at them): " + + ", ".join(scan.orphan_indexes)) out.append("") out.append("## Files") out.append("") @@ -129,15 +181,20 @@ def main(): name = fm_field(text, "name") or "?" if len(name) > 50: name = name[:47] + "..." - status = "linked" if f in linked else "ORPHAN" + if memory_reindex.is_ck_snapshot(f): + status = "ck_snapshot" # never indexed by design, so not an orphan + else: + status = "linked" if f in linked else "ORPHAN" out.append(f"| {f} | {size} | {age} | {ftype} | {name} | {status} |") if dangling: out.append("") - out.append("### Dangling links (in MEMORY.md but file missing)") + out.append("### Dangling links (in an index, file missing)") out.append("") - for link in dangling: - out.append(f"- {link}") + out.append("| Index | Line | Target |") + out.append("|-------|------|--------|") + for d in dangling: + out.append(f"| {d.index} | {d.line} | {d.target} |") out.append("") out.append("## Heuristic flags") @@ -148,14 +205,18 @@ def main(): for f in files: text = file_texts[f] flags = [] - if CK_SNAPSHOT_RE.match(f): + is_ck = memory_reindex.is_ck_snapshot(f) + if is_ck or CK_SNAPSHOT_RE.match(f): flags.append("ck_snapshot") name = fm_field(text, "name") if LANDED_RE.search(f) or LANDED_RE.search(name): flags.append("landed_doc") if SUPERSEDED_RE.search(text): flags.append("superseded_marker") - if f not in linked: + # `orphan` only for real pages: a ck snapshot is unindexed by design, + # and flagging it here would contradict the Summary's orphan count, + # which (like every integrity check) skips them. + if f not in linked and not is_ck: flags.append("orphan") age = file_age_days(os.path.join(mem_dir, f), today_epoch_days) if age > OLD_90D: diff --git a/skills/okfmem-reindex/SKILL.md b/skills/okfmem-reindex/SKILL.md new file mode 100644 index 0000000..f06cde9 --- /dev/null +++ b/skills/okfmem-reindex/SKILL.md @@ -0,0 +1,257 @@ +--- +name: okfmem-reindex +description: "Topology restructuring for a bloated MEMORY.md — cluster flat pointers into lane indexes, rewrite MEMORY.md as a routing table with 'covers' clauses, verify link integrity. Not deletion (see okfmem-curate); nothing is lost, pointers only move. Hard approval gate before any write. Invoked as /okfmem-reindex." +origin: user +--- + +# okfmem Reindex + +> **Names.** Canonical `/okfmem-reindex`. This skill lives in the `okfmem` +> repo (`~/okfmem/skills/`); `okfmem init` symlinks it into each harness +> alongside `/okfmem`, `/okfmem-save`, and `/okfmem-curate`. + +Restructure a flat `MEMORY.md` that has grown past the auto-load byte ceiling +into a **two-tier index**: a small root `MEMORY.md` that reads as a routing +table, plus one or more `MEMORY-.md` lane indexes holding the actual +pointer bulk. Nothing is deleted, archived, or rewritten in content — every +pointer that moves still points at the same page. This is a topology fix, not +a hygiene pass. + +## Why this is a separate skill from `okfmem-curate` + +`okfmem-curate` answers "is this content still worth keeping" — it deletes, +archives, and compresses. This skill answers "is this content filed in the +right place" — it only moves pointers between index files. The two operations +have opposite risk profiles: + +| | `okfmem-curate` | `/okfmem-reindex` | +|---|---|---| +| Operation | **deletes** pages | **moves** a pointer between index files | +| Reversibility | hard; git-backed store is the only recovery | nothing is lost; no page is touched | +| Gate | hard approval stop, deliberately rare | propose → approve → execute; can be routine | +| Framing | rot | topology | + +A store can come back from a full curate audit completely clean — zero +superseded markers, zero dangling pointers, every page recent — and still +have a `MEMORY.md` several times over budget. That is not a curation problem; +curate has no vocabulary for it. Folding a non-destructive move behind +curate's hard stop would also make it rarer than it deserves to be, since a +reversible operation does not need the friction a destructive one does. + +**Never re-flatten. The remedy for an over-budget `MEMORY.md` is a lane +split, never tighter hooks** — see `skills/okfmem-save/SKILL.md` Step 3 +("Lane routing") and `skills/okfmem-curate/SKILL.md` Phase 2 for the full +statement of this rule and the byte-savings ratio between the two remedies. +This skill exists to execute that split, not to restate the rule. + +**Pages on disk cost nothing.** Only `MEMORY.md` and `STATE.md` are +auto-loaded at session start; every other page — including every lane +index's pointer targets — costs zero context regardless of how many exist. +This skill never proposes deleting or consolidating pages as a context +optimization; for context, that is a no-op by construction. Consolidation is +`okfmem-curate`'s domain, not this skill's, even when a lane happens to +surface duplicate-looking content along the way. + +**`type: feedback` pointers stay at the root.** User preferences are durable +and high-hit-rate — they are not a lane, regardless of how a byte-count +argument might read. + +## When to use + +- `okfmem status` or `okfmem reindex --report` flags a project's `MEMORY.md` + as `OVER` the auto-load byte ceiling (`memory_reindex.MEMORY_BUDGET_BYTES`, + 8192 bytes). +- The user says "reindex memory", "split MEMORY.md", "the index is too big", + or similar. +- **Audit-only check-in**: when the user wants the cluster proposal without + committing to execution yet — invoke with the `audit` argument. + +## Modes + +| Mode | Trigger | What it does | +|------|---------|--------------| +| Default | "reindex memory" / "split the index" / no arg | All five phases; halts at approval gate; executes on approval | +| Audit | `audit` arg, or "memory reindex audit" / "what would you split" | Phases 1–3 only; presents the cluster proposal and stops; no approval question, no execution | + +## How it works + +Five phases, mirroring `okfmem-curate`'s shape so the two feel like +siblings. Phase 4 is the only phase that writes to the store, and it is +gated on explicit user approval in Phase 3 — never create or rewrite an +index file before that approval. + +### Phase 1: Measure + +Resolve the target memory dir (same convention as `okfmem-curate` Phase 1 — +current project by default, or an explicit path argument) and call the +engine: + +```bash +python3 ~/okfmem/okfmem reindex --report "$MEM_DIR" --json +``` + +Read the `auto_loaded` row for `MEMORY.md`: bytes, ceiling, `over`. Read the +`sections` array — the per-section byte breakdown of `MEMORY.md` — to see +**which block holds the bytes**, not just the total. A single dominant +section is the finding that makes a lane split worthwhile; several small, +evenly-sized sections may not be. + +**If `MEMORY.md` is under budget and no section dominates, report a no-op +and stop.** Do not propose a split for its own sake, and do not pad the +report into busywork to justify having run — "this store is fine" is a +complete and correct answer. Re-running this skill on a store it already +reindexed should reach this branch and report no changes. + +### Phase 2: Cluster + +Read every pointer currently in `MEMORY.md` (and any existing lane indexes, +if this is a re-run adding to a prior split) and propose a set of lanes. + +**Explicitly consider a cross-cutting lane, not only subsystem-shaped +ones.** The natural first pass groups pointers by directory/subsystem/slug +prefix — that heuristic finds real lanes, but it can also miss the largest +one. A theme that cuts across every subsystem (recurring workflow +conventions, a class of gotcha that shows up in every area, a review or +verification discipline) has no shared prefix or directory to key on, so a +subsystem-first pass silently shreds it across the lanes it should have +been its own. Read the pointer hooks themselves for a recurring *kind* of +fact, not just a recurring *area* of the codebase, before finalizing the +lane list. + +For each proposed lane, write the **"covers" clause** it will carry in the +root routing table — the one or two sentences that let a session model +decide whether to open that lane index, without opening it. If you cannot +write a tight covers clause for a lane, its boundary is probably wrong; fold +it into a neighboring lane or reconsider the split. + +Decide, per pointer, whether it stays at the root instead of moving to a +lane: + +- Genuinely cross-lane facts (project identity, deploy topology, a product + invariant that every lane's work touches) — root. +- `type: feedback` pages — root, always (see rule above). +- Everything else — its lane. + +### Phase 3: Propose (HARD STOP) + +Present the plan as a single markdown response: + +```markdown +## Reindex plan for review + +### Proposed lanes + +| Lane index | Covers | Pointers moving in | +|---|---|---| +| MEMORY-.md | | N | + +### Staying at root + +| Pointer | Why | +|---|---| +| ... | cross-lane / type: feedback | + +### Projected bytes + +| File | Before | After | +|---|---|---| +| MEMORY.md | X | Y | +| MEMORY-.md (new/extended) | — | Z | + +→ Approve as-is, adjust lane boundaries, or call out pointers to keep at +the root, before anything is written. +``` + +**STOP HERE.** Do not invoke `Write` or `Edit` on any index file until the +user has explicitly approved. This gate is lighter in *stakes* than +`okfmem-curate`'s — nothing is destroyed, and a bad split is recoverable by +re-running this skill — but it is not lighter in *presence*: the carve is +the entire product of this skill, and an unreviewed one is tedious to +unwind by hand across every affected index. If the user proposes +corrections ("merge lane X and Y", "keep pointer N at root"), update the +plan and present again — still no execution. + +In **audit mode** the skill stops here regardless and reports the proposal +as-is — no approval question, no execution. Audit therefore runs Phases 1–3 +(the proposal table above is a Phase 3 artifact) and writes nothing: every +write in this skill lives in Phase 4. + +### Phase 4: Execute + +Once approved: + +1. For each new lane, `Write` `MEMORY-.md` using the pointer-line + format already in use (rich `- [Title](slug.md) — hook` or bare + `- slug.md — hook`, matching what the pointer already used at the root). + Head the file with a one- or two-line note stating **when and why** it + was split out (date + the byte/dominant-block finding from Phase 1) — + this is what lets a future session (or a future run of this skill) + understand the lane's origin without re-deriving it. +2. For each extended (pre-existing) lane, `Edit` it to append the moving + pointers, same format. +3. **Read `MEMORY.md` once** (required before `Write`), then rewrite it as + the routing table: a row per lane index with its "covers" clause, the + cross-lane facts, and the `type: feedback` pointers — nothing else. This + is the file every session auto-loads, so it should read as a map, not a + page list. +4. **Every lane index must be linked from the root routing table.** An + index with **no inbound link from any other index** is an *orphan index* + and Phase 5's `--verify` will fail on it — the mechanism that catches a + lane created and then never referenced. Note the rule here is stricter + than the check: `--verify` only asks whether *some* index links the lane, + so a chain (`MEMORY.md → MEMORY-a.md → MEMORY-b.md`) passes with zero + orphan indexes even though `MEMORY-b.md` has no routing-table row. Route + every lane from the root anyway — the root is the only file a session + auto-loads, and a lane reachable only two hops in is a lane nobody opens. +5. **Never re-flatten.** A pointer that already lives in a lane index is + never moved back to `MEMORY.md` by this skill, including when a lane + turns out smaller than expected after the move — resize the lane + structure (merge two small lanes, rename one), don't undo the split. + +### Phase 5: Verify + +```bash +python3 ~/okfmem/okfmem reindex --verify "$MEM_DIR"; echo "verify exit: $?" +python3 ~/okfmem/okfmem reindex --report "$MEM_DIR" # after-numbers +``` + +`--verify` exits **0** when every `MEMORY*.md` in the dir is intact and +**1** on any dangling pointer, orphan page, or orphan index. Treat non-zero +as failure — the reindex is broken, not merely imperfect — and fix it +before reporting success. A dangling pointer is reported with the index +file it came from. An *orphan index* is one with **no inbound link from any +other index** — a lane created in step 1/2 above and then referenced by +nothing at all; if `--verify` reports one, the fix is adding its +routing-table row to `MEMORY.md`, not deleting the lane. Because the check +accepts an inbound link from *any* index, it will not catch a lane linked +only from another lane — step 4's root-routing rule is what covers that, and +it is on you, not on `--verify`. + +Report, in this order: (1) `MEMORY.md` bytes before → after against the +ceiling, and the lane index(es) created/extended with their byte totals; +(2) the verify result (dangling / orphan / orphan-index counts, ideally all +zero). Do not report a page or archive count here — that is +`okfmem-curate`'s number, not this skill's; a reindex that moves pointers +without deleting anything changes zero pages on disk. + +## Anti-patterns + +- **Executing before Phase 3 approval.** The carve is the whole product; + writing it unreviewed defeats the point of proposing it. +- **Re-flattening a lane pointer back to `MEMORY.md`** on a later run, + including "just to clean up" or "it's a small lane now" — resize lane + structure instead (see Phase 4, step 5). +- **Proposing page deletion or consolidation as part of a reindex plan.** + That is `okfmem-curate`'s domain; for context cost it is a no-op (pages on + disk cost nothing). If a reindex pass surfaces genuinely stale content, + name it as a follow-up for `/okfmem-curate`, don't fold it into this + skill's plan. +- **A subsystem-only clustering pass.** Always check for a cross-cutting + lane before finalizing Phase 2 — see the rationale above. +- **Creating a lane index without a routing-table row in `MEMORY.md`.** + Phase 5's `--verify` catches the blatant case (a lane no index links at + all) as an orphan index, but it does not catch a lane linked only from + another lane — so this one is on the Phase 4 checklist, not on the tool. +- **Padding a no-op into a plan.** If Phase 1 finds `MEMORY.md` under + budget with no dominant section, say so and stop — a proposed split with + no real justification behind it is worse than no action. diff --git a/skills/okfmem-save/SKILL.md b/skills/okfmem-save/SKILL.md index 1e4cd85..f251e88 100644 --- a/skills/okfmem-save/SKILL.md +++ b/skills/okfmem-save/SKILL.md @@ -20,7 +20,7 @@ description: "Session close-out — clean up tool-created worktrees/branches, wr Everything lives under `~/.claude/projects//memory/` (symlinked to `~/okfmem-store/projects//`): - **Active state** lives in `STATE.md` — a bounded, single-session snapshot with a fixed 6-section shape (Summary / Left off / Next steps / Decisions / Blockers / Goal) plus OKF `type: state` frontmatter. It is **overwritten every session**, never appended. The native memory system auto-loads it next session. -- **Durable knowledge** lives in per-topic `.md` pages (OKF v0.1: markdown + YAML frontmatter with a top-level `type:` field ∈ `user` | `feedback` | `project` | `reference`), indexed by one-line pointers in `MEMORY.md`. This is the durable write path: create or update the page, then add/refresh its `MEMORY.md` pointer. +- **Durable knowledge** lives in per-topic `.md` pages (OKF v0.1: markdown + YAML frontmatter with a top-level `type:` field ∈ `user` | `feedback` | `project` | `reference`), indexed by one-line pointers in a `MEMORY*.md` index — the matching lane index by default, the root `MEMORY.md` only for cross-lane or `type: feedback` pages (#53; see Step 3). This is the durable write path: create or update the page, then add/refresh its pointer in that index. - `STATE.md` uses a separate `type: state` (see Step 5) — a different file with a different consumer (active-state snapshot, not the durable-page index), not a fifth value in the `type:` enum above. ## When to use @@ -158,12 +158,24 @@ type: **Existing topic** (slug already a page in `$MEMORY_DIR`) → `Edit` that page to add the new fact, or supersede stale content in place (no manual `SUPERSEDED` markers needed — just rewrite the page to current truth). -**Index pointer** — for every new page, add a one-line pointer to that project's `MEMORY.md`; for an updated page, refresh the existing pointer if its hook changed: +**Index pointer** — for every new page, add a one-line pointer to the matching index (its target — a lane index by default, `MEMORY.md` only when the page qualifies as cross-lane — is decided by the lane-routing rule below, before the budget check); for an updated page, refresh the existing pointer in that same index if its hook changed: ``` - [](<slug>.md) — <one-line hook> ``` +**Budget: the full pointer line is ≤150 chars, enforced here at write time (#52).** Capture runs every session; curate runs rarely — a budget only checked at curate time is a budget that is out of compliance almost always. Count the whole line (`- [<Title>](<slug>.md) — <hook>`) before writing it. Over 150, tighten the hook: move detail into the page body, where it costs nothing, and leave the index line as a bare recall hook. Apply the same check on a *refresh* — updating an existing topic's pointer must not silently re-inflate a line that was already in budget. + +This is advisory, not a hard gate: if a hook genuinely can't be tightened under 150 without losing the recall value, write it anyway and **report it** — see Step 8 — rather than let it pass silently. + +**Lane routing (#53) — decide this before writing the pointer above and before running the budget check.** The pointer's default target is the matching **lane index** (`MEMORY-<lane>.md`), not the root `MEMORY.md`: `MEMORY.md` is a map of content, not a page list, so a captured page routes to `MEMORY.md` only when it is genuinely cross-lane (project identity, deploy topology, a product invariant that cuts across every lane) or its frontmatter `type` is `feedback`. Everything else — most `user`/`project`/`reference` pages — belongs in a lane index. + +If no existing lane index covers this page's topic, **create one** rather than appending to `MEMORY.md`: `Write` `MEMORY-<lane>.md` using the same pointer-line format as the root index, then give the root `MEMORY.md` a one-line routing-table row for it — a short "covers" clause (e.g. `- MEMORY-<lane>.md — covers <what a session model can route on without opening it>`) so recall can route without opening the lane index. A lane is cheap to create; a flat root is not. + +With the target decided, apply the ≤150-char budget check above to the pointer as it will actually appear — in the lane index if it lands there, in `MEMORY.md` if it's cross-lane or `type: feedback`. + +**Once a pointer lives in a lane index, never move it back to the root.** Refreshing an existing page's pointer means editing it in the lane index it already lives in, not re-adding it to `MEMORY.md` — split pointers are never re-flattened (#53). + Do this step before Step 5 so the `STATE.md` summary can mention what was captured. ### Step 4: File Linear issues for pending work @@ -204,6 +216,14 @@ modified: <ISO-8601 UTC timestamp — e.g. 2026-07-23T18:04:00Z> Fill each section from the session. **Stamp `modified:` with the current wall-clock time in ISO-8601 UTC (`YYYY-MM-DDTHH:MM:SSZ`), fresh on every save** — `okfmem sync`/`okfmem pull` read this field to auto-resolve cross-machine `STATE.md` conflicts (newer `modified:` wins, last-write-wins), so a stale or missing value silently defers reconciliation to a hand-merge. Show a draft summary first: `"Session: '<summary>' — save this? (yes / edit)"`. After confirmation, `Write` the file (full overwrite — do not `Edit`/append; the whole file is replaced each session). +**`STATE.md` has a ceiling: 8192 bytes** (~2.2k tokens) — the same auto-load budget `okfmem reindex` checks `MEMORY.md` against (`memory_reindex.STATE_BUDGET_BYTES`; #52). Generous headroom for a bounded, six-section, single-session snapshot, so this is a signal something drifted (a section grew a changelog instead of staying a pointer), not a routine concern. After writing, spot-check it: + +```bash +okfmem reindex --report "$MEMORY_DIR" # or: python3 ~/okfmem/okfmem reindex --report "$MEMORY_DIR" +``` + +Look at the `STATE.md` row's Status column. If it reads `OVER`, report it in Step 8 — advisory, not a rewrite gate. + Keep `## Goal` as the project's standing goal — carry forward the prior value unless the goal shifted this session. Use `(none)` for empty `## Blockers`. ### Step 6: Post impl-complete comment for completed issues @@ -268,6 +288,7 @@ Show the user: - The worktree/branch cleanup result (Step 1c): removed N worktrees / M branches, and the kept list if non-empty - The session summary written to `STATE.md` (one line) - Any insights captured as memory pages (slugs + one-line hooks; note new vs. updated-existing) +- Any `MEMORY.md` pointer over the 150-char budget after this session's writes (slug + length), or `"none"`; likewise if the `okfmem reindex --report` spot-check flagged `STATE.md` as `OVER` - Any issues filed or commented on (Linear or GitHub) - The memory push result — the commit SHA that was pushed (from `okfmem sync`'s status line), or `"no memory changes to push"` if the working tree was clean. @@ -275,7 +296,10 @@ Show the user: - **Active state goes in `STATE.md`**, not in `MEMORY.md` or the memory pages - **`STATE.md` is bounded and overwritten every session** — full replace, never append; keep it to the 6-section template -- **Durable knowledge is captured as `<topic>.md` memory pages + a `MEMORY.md` pointer**, not stored in `STATE.md` +- **Durable knowledge is captured as `<topic>.md` memory pages + an index pointer**, not stored in `STATE.md` +- **A new page's pointer routes to its lane index by default; `MEMORY.md` gets a pointer only for cross-lane or `type: feedback` pages (#53)** — create a lane index (with a routing "covers" row in `MEMORY.md`) when no lane matches; never re-flatten a lane pointer back to the root +- **An index pointer is ≤150 chars, checked at write time (#52)** — new or refreshed, tighten before writing; if it can't be tightened without losing recall value, write it and report it in Step 8 rather than pass silently +- **`STATE.md` has an 8192-byte ceiling** (`memory_reindex.STATE_BUDGET_BYTES`), spot-checked via `okfmem reindex --report` after Step 5's write; report `OVER`, don't gate on it - **Reuse an existing page slug to update a topic** (dedup) — rewrite the page to current truth instead of leaving stale duplicates - **Pending work goes in the issue tracker** (Linear or GitHub, whichever this project uses), not in `STATE.md` `## Blockers` (reserve blockers for "can't progress" not "haven't started") - **Capture memory pages BEFORE writing `STATE.md`** so the session summary can reference what was captured diff --git a/skills/okfmem/SKILL.md b/skills/okfmem/SKILL.md index 11341ca..4bb646a 100644 --- a/skills/okfmem/SKILL.md +++ b/skills/okfmem/SKILL.md @@ -29,7 +29,7 @@ Two git-backed repos + native memory auto-load: Two memory layers, both plain markdown the native memory system auto-loads: - **Active state** — `projects/<name>/STATE.md`, bounded 6-section snapshot, **overwritten** each session by `/okfmem-save`. -- **Durable knowledge** — `projects/<name>/<slug>.md` pages indexed by one-line pointers in `MEMORY.md` (first 200 lines auto-load); pages read on demand. +- **Durable knowledge** — `projects/<name>/<slug>.md` pages indexed by one-line pointers in a `MEMORY*.md` index: the matching lane index by default, the root `MEMORY.md` only for cross-lane or `type: feedback` pages (#53); pages read on demand. Four moving parts: @@ -39,6 +39,7 @@ Four moving parts: | **hygiene** (decay/archive/regen MEMORY.md) | Stop hook → `memory_consolidate.py` | deterministic; archives stale pages (reversible), never deletes | | **sync in** | SessionStart hook → `git pull --rebase` | freshens the store before the session | | **hard curation** | `/okfmem-curate` (rare, gated) | semantic merge / hard purge decay won't do | +| **reindex** | `/okfmem-reindex` (byte ceiling trips, gated) | clusters flat pointers into lane indexes; moves, never deletes | `okfmem sync` (shared git helper) backs **both** `/okfmem-save` and the Stop-hook consolidation, so pull-rebase + concurrency-lock behavior is identical on both. @@ -77,16 +78,21 @@ prints: ``` projects (14): - * okfmem pages:27 MEMORY.md:33 archived:0 STATE:yes - tools pages:158 MEMORY.md:202 archived:0 STATE:yes ! over 200-line auto-load limit + * okfmem pages:27 MEMORY.md:4200 B archived:0 STATE:yes + tools pages:158 MEMORY.md:15600 B archived:0 STATE:yes ! over 8192-byte auto-load ceiling -- split a lane index (/okfmem-reindex) + 11 more (okfmem status --all) decay: epoch 2026-07-16 ``` - The `*` marks the project the current working directory maps to. -- Any project whose `MEMORY.md` exceeds the 200-line auto-load limit is flagged - inline (`! over 200-line auto-load limit`) — a candidate for `/okfmem-curate`. -- The default view collapses to the current project plus any over-limit +- Any project whose `MEMORY.md` exceeds the auto-load byte ceiling + (`memory_reindex.MEMORY_BUDGET_BYTES`, 8192 bytes) is flagged inline + (`! over 8192-byte auto-load ceiling`) — a candidate for `/okfmem-reindex`, + not `/okfmem-curate`. Bytes, not line count, are the trigger (#53): a + store can sit well under the old 200-line mark while over the byte + ceiling once pointers run long. The recommended remedy is a lane split, + not tightening hooks. +- The default view collapses to the current project plus any over-ceiling project. When the user wants the **full** list, re-run `python3 ~/okfmem/okfmem status --all`; for a single project, `python3 ~/okfmem/okfmem status --project <name>`. @@ -112,9 +118,9 @@ Missing canonical skills or an unwired hook → tell the user to run `okfmem ini ### Summarize Close with a 3–5 line health summary: how many projects, anything over the -`MEMORY.md` line cap (from the flagged rows), the store sync state, and any -wiring gap with the one command that fixes it (`okfmem init` for skills/hooks, -`okfmem sync` for a dirty store). +`MEMORY.md` byte ceiling (from the flagged rows), the store sync state, and +any wiring gap with the one command that fixes it (`okfmem init` for +skills/hooks, `okfmem sync` for a dirty store). ## Usage (`/okfmem usage`) @@ -136,6 +142,10 @@ Print this orientation instead of the dashboard: - `/okfmem-save` (`/primer`) — session close-out (capture + STATE + push). - `/okfmem-curate` (`/memory-curate`) — **rare**; judgment-driven purge/merge the automatic decay pass won't do. Routine hygiene is already automatic. +- `/okfmem-reindex` — when `MEMORY.md` trips the byte ceiling: clusters the + flat pointers into lane indexes and rewrites `MEMORY.md` as a routing + table. Nothing is deleted — pointers only move. `audit` mode previews the + cluster proposal without writing. - `okfmem sync -m "…"` — commit+push the store by hand (pull-rebase + lock). - `okfmem init` — run once **in each repo** you want memory for (the link is per-repo; the installer only wired the repo it ran in). Also (re)wires skills diff --git a/tests/test_consolidate_index_drop.py b/tests/test_consolidate_index_drop.py new file mode 100644 index 0000000..52007f3 --- /dev/null +++ b/tests/test_consolidate_index_drop.py @@ -0,0 +1,127 @@ +"""Archiving a page must drop its pointer from EVERY index, not just the root. + +The composition bug this guards: #53's write-time lane routing sends a new +page's pointer straight into a `MEMORY-<lane>.md`, while the consolidation pass +only ever dropped lines from the root `MEMORY.md`. Archiving such a page left a +dangling pointer behind — and consolidation runs unattended from the Stop hook, +so every store that adopted the routing rule would accumulate dangling pointers +until `okfmem reindex --verify` (the gate the reindex/curate skills depend on) +started failing with no user action. + +Driven end-to-end through the module's `main()` in a subprocess, because the +defect was in main's wiring, not in `drop_memory_lines` — a unit test of the +helper passes either way. `OKFMEM_NO_STATUS=1` keeps the badge writer off the +real home dir; `--no-commit` keeps git out of it. Fixtures are synthetic. +""" +import os +import subprocess +import sys + +import memory_reindex as mr + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + +PAGE = ( + "---\n" + "name: old-page\n" + "description: an aged-out page\n" + "type: project\n" + "created: 2026-01-01\n" + "last_accessed: 2026-01-01\n" + "access_count: 0\n" + "---\n\n" + "# Old page\n\nBody.\n" +) + + +def _store(tmp_path, root_index, lane_index=None): + store = tmp_path / "store" + pdir = store / "projects" / "demo" + pdir.mkdir(parents=True) + (pdir / "old-page.md").write_text(PAGE, encoding="utf-8") + (pdir / "MEMORY.md").write_text(root_index, encoding="utf-8") + if lane_index is not None: + (pdir / "MEMORY-lane.md").write_text(lane_index, encoding="utf-8") + (store / "decay_state.json").write_text( + '{"epoch": "2026-01-01"}\n', encoding="utf-8") + return store, pdir + + +def _consolidate(store): + env = dict(os.environ, OKFMEM_NO_STATUS="1") + return subprocess.run( + [sys.executable, os.path.join(ROOT, "memory_consolidate.py"), + "--store", str(store), "--no-commit", "--today", "2026-07-31"], + capture_output=True, text=True, env=env, cwd=ROOT, check=True) + + +def test_pointer_in_a_lane_index_is_dropped_when_the_page_is_archived(tmp_path): + store, pdir = _store( + tmp_path, + root_index="# MEMORY\n\n- MEMORY-lane.md - covers the lane\n", + lane_index="# Lane\n\n- [Old page](old-page.md) - hook\n") + + before = mr.Scan(str(pdir)) + assert before.ok, "fixture must start clean, or the test proves nothing" + + out = _consolidate(store).stdout + # The relative path is built with os.path.join by the code under test, so + # it renders with backslashes on Windows -- a hand-typed POSIX literal here + # passes on macOS/Linux and fails only on the Windows matrix leg. + assert "archive " + os.path.join("projects", "demo", "old-page.md") in out + assert "index lines dropped: 1" in out + + assert (pdir / "archive" / "old-page.md").is_file() + assert "old-page.md" not in (pdir / "MEMORY-lane.md").read_text( + encoding="utf-8") + # the routing-table row is not collateral damage + assert "MEMORY-lane.md" in (pdir / "MEMORY.md").read_text(encoding="utf-8") + assert mr.Scan(str(pdir)).ok + + +def test_root_index_drop_still_works(tmp_path): + """The pre-existing root-only path must keep working unchanged.""" + store, pdir = _store( + tmp_path, + root_index="# MEMORY\n\n- [Old page](old-page.md) - hook\n" + "- [Kept](kept.md) - unrelated\n") + + out = _consolidate(store).stdout + assert "index lines dropped: 1" in out + root = (pdir / "MEMORY.md").read_text(encoding="utf-8") + assert "old-page.md" not in root + assert "kept.md" in root # unrelated pointer survives + + +def test_a_pointer_duplicated_across_two_indexes_is_dropped_from_both(tmp_path): + """The half-finished-move shape: dropping only one copy would still leave a + dangling pointer, so the count must be 2 and both files must be clean.""" + store, pdir = _store( + tmp_path, + root_index="# MEMORY\n\n- MEMORY-lane.md - covers the lane\n" + "- [Old page](old-page.md) - hook\n", + lane_index="# Lane\n\n- old-page.md - bare-syntax copy\n") + + out = _consolidate(store).stdout + assert "index lines dropped: 2" in out + assert "old-page.md" not in (pdir / "MEMORY.md").read_text(encoding="utf-8") + assert "old-page.md" not in (pdir / "MEMORY-lane.md").read_text( + encoding="utf-8") + assert mr.Scan(str(pdir)).ok + + +def test_dry_run_still_writes_no_index(tmp_path): + store, pdir = _store( + tmp_path, + root_index="# MEMORY\n\n- MEMORY-lane.md - covers the lane\n", + lane_index="# Lane\n\n- [Old page](old-page.md) - hook\n") + lane_before = (pdir / "MEMORY-lane.md").read_bytes() + + env = dict(os.environ, OKFMEM_NO_STATUS="1") + subprocess.run( + [sys.executable, os.path.join(ROOT, "memory_consolidate.py"), + "--store", str(store), "--dry-run", "--today", "2026-07-31"], + capture_output=True, text=True, env=env, cwd=ROOT, check=True) + + assert (pdir / "MEMORY-lane.md").read_bytes() == lane_before + assert (pdir / "old-page.md").is_file() diff --git a/tests/test_graduate.py b/tests/test_graduate.py index 62aa162..7fc6374 100644 --- a/tests/test_graduate.py +++ b/tests/test_graduate.py @@ -426,3 +426,86 @@ def flaky_remove(path, *a, **kw): assert memory_md.read_bytes() == memory_before assert not (repo / "CLAUDE.md").read_text( encoding="utf-8").count("does NOT egress") + + +# --------------------------------------------------------------------------- +# (N4) lane routing: the pointer may live in ANY index, so the drop — and the +# rollback snapshot that has to be able to undo it — must cover every one. +# Dropping only the root `MEMORY.md` strands a lane pointer (the same #53 +# composition bug consolidation had); snapshotting only the root while writing +# several would turn a failed graduate into a half-modified store, which is +# worse than the bug being fixed. +# --------------------------------------------------------------------------- +LANE_ROOT_MD = ( + "# MEMORY\n\n" + "- MEMORY-lane.md — covers the lane\n" + "- [Other page](other.md) — unrelated hook\n" +) +LANE_MD = ( + "# MEMORY-lane\n\n" + "- egress-claim.md — bare-syntax pointer in a lane index\n" +) + + +def _lane_store(tmp_path, root_md=LANE_ROOT_MD, lane_md=LANE_MD): + store = _store(tmp_path) + pdir = store / "projects" / "demoproj" + (pdir / "MEMORY.md").write_text(root_md, encoding="utf-8") + (pdir / "MEMORY-lane.md").write_text(lane_md, encoding="utf-8") + return store, pdir + + +def test_pointer_living_in_a_lane_index_is_dropped(tmp_path, monkeypatch): + import datetime + + store, pdir = _lane_store(tmp_path) + repo = _repo(tmp_path) + monkeypatch.setattr(mg, "_current_git_root", lambda: str(repo)) + + plan = mg.build_plan(str(store), "egress-claim", "demoproj", + str(repo / "CLAUDE.md"), None, None) + _dest, dropped = mg.apply_plan(plan, str(store), datetime.date(2026, 7, 23)) + + assert dropped == 1 + assert "egress-claim.md" not in (pdir / "MEMORY-lane.md").read_text( + encoding="utf-8") + # the root routing row and unrelated pointer are not collateral damage + root = (pdir / "MEMORY.md").read_text(encoding="utf-8") + assert "MEMORY-lane.md" in root and "other.md" in root + + +def test_rollback_restores_every_index_it_touched(tmp_path, monkeypatch): + """A failed graduate must leave EVERY index byte-identical, not just the + root one. The pointer is duplicated across both indexes here so both are + genuinely written before the outward step throws — a root-only snapshot + would restore `MEMORY.md` and leave the lane index permanently edited.""" + import datetime + + import pytest + + store, pdir = _lane_store( + tmp_path, + root_md=LANE_ROOT_MD + "- [Egress](egress-claim.md) — root copy\n") + repo = _repo(tmp_path) + monkeypatch.setattr(mg, "_current_git_root", lambda: str(repo)) + + root_md, lane_md = pdir / "MEMORY.md", pdir / "MEMORY-lane.md" + root_before, lane_before = root_md.read_bytes(), lane_md.read_bytes() + src = pdir / "egress-claim.md" + src_before = src.read_bytes() + + # Same failure injection as the CLAUDE.md-write test: a file where the + # target's parent dir should be, so os.makedirs throws AFTER both indexes + # have been rewritten. + blocker = repo / "blocker" + blocker.write_text("i am a file, not a dir\n", encoding="utf-8") + plan = mg.build_plan(str(store), "egress-claim", "demoproj", + str(blocker / "CLAUDE.md"), None, None) + + with pytest.raises(Exception): + mg.apply_plan(plan, str(store), datetime.date(2026, 7, 23)) + + assert root_md.read_bytes() == root_before + assert lane_md.read_bytes() == lane_before + assert src.exists() and src.read_bytes() == src_before + assert not (pdir / "archive" / "egress-claim.md").exists() diff --git a/tests/test_reindex.py b/tests/test_reindex.py new file mode 100644 index 0000000..72d6712 --- /dev/null +++ b/tests/test_reindex.py @@ -0,0 +1,706 @@ +"""Tests for the reindex engine (`okfmem reindex`, issue #54). + +The defect class this file guards is "the checker knows one file and one link +syntax": the old checker parsed only `MEMORY.md`, only the rich +`[title](slug.md)` form, so every page carried by a lane index read as an orphan +and a genuinely dangling pointer *inside* a lane index was invisible. A verifier +that fails open is worse than no verifier, so the exit-code contract is tested +as carefully as the parsing. + +All fixtures are synthetic — no real store, no home paths (leak gate). +""" +import json +import os + +import pytest + +import memory_reindex as mr + + +# --------------------------------------------------------------------------- +# fixtures +# --------------------------------------------------------------------------- +def write(d, name, text): + p = d / name + p.write_text(text, encoding="utf-8") + return p + + +def page(d, slug, body="body\n"): + return write(d, f"{slug}.md", f"---\nname: {slug}\n---\n\n{body}") + + +@pytest.fixture +def mem(tmp_path): + """An empty memory dir. Tests build the index shape they need.""" + d = tmp_path / "memdir" + d.mkdir() + return d + + +def scan(d): + return mr.Scan(str(d)) + + +# --------------------------------------------------------------------------- +# link parsing — both syntaxes are first-class +# --------------------------------------------------------------------------- +def test_rich_and_bare_syntaxes_both_parse(): + text = ( + "# Index\n" + "\n" + "- [Human title](rich-slug.md) - hook\n" + "- bare-slug.md - hook\n" + ) + got = [(p.target, p.syntax) for p in mr.parse_pointers_text(text)] + assert got == [("rich-slug.md", "inline"), ("bare-slug.md", "bare")] + + +def test_title_text_starting_with_a_filename_is_not_a_pointer(): + """The regression the issue names: a greedy `- \\[?` prefix match reads + `CLAUDE.md` out of the *title* and reports a phantom dangling link.""" + text = "- [CLAUDE.md subdir lanes + graduated pages](real-slug.md) - hook\n" + ptrs = mr.parse_pointers_text(text) + assert [p.target for p in ptrs] == ["real-slug.md"] + assert "CLAUDE.md" not in [p.target for p in ptrs] + + +def test_bare_form_still_matches_when_the_slug_itself_looks_titley(): + text = "- claude-md-graduated-pages-and-subdir-lanes.md - hook\n" + assert [p.target for p in mr.parse_pointers_text(text)] == [ + "claude-md-graduated-pages-and-subdir-lanes.md"] + + +def test_bullet_variants_and_indentation(): + text = "* a.md - x\n+ b.md - x\n - c.md - x\n" + assert [p.target for p in mr.parse_pointers_text(text)] == [ + "a.md", "b.md", "c.md"] + + +def test_link_target_variants_normalize(): + text = ( + "See [a](a.md#section) and [b](<b.md>) and [c](c.md 'Title').\n" + ) + assert [p.target for p in mr.parse_pointers_text(text)] == [ + "a.md", "b.md", "c.md"] + + +def test_two_links_on_one_line_both_parse(): + text = "| [One](one.md) | 3 | [Two](two.md) |\n" + assert [p.target for p in mr.parse_pointers_text(text)] == [ + "one.md", "two.md"] + + +def test_multi_slug_bare_pointer_joined_by_plus(): + """One list item may carry several slugs under a shared hook. Catching + only the first is a false-orphan factory for every slug after it.""" + text = "- a-slug.md + b-slug.md + c-slug.md - one shared hook\n" + assert [p.target for p in mr.parse_pointers_text(text)] == [ + "a-slug.md", "b-slug.md", "c-slug.md"] + + +def test_a_filename_named_in_the_hook_is_not_a_pointer(): + """The mirror of the title-prefix bug, at the other end of the line: the + `+` run stops at the hook, so prose naming a file never becomes a pointer. + """ + text = "- real-slug.md - MEMORY.md is the ONLY file loaded, see CLAUDE.md\n" + assert [p.target for p in mr.parse_pointers_text(text)] == ["real-slug.md"] + + +def test_plus_run_only_fires_at_the_head_of_a_list_item(): + text = "Pages load on demand; see CLAUDE.md + docs/ for the rules.\n" + assert mr.parse_pointers_text(text) == [] + + +def test_multi_slug_run_stops_at_a_non_filename_token(): + text = "- a.md + not-a-file + b.md - hook\n" + assert [p.target for p in mr.parse_pointers_text(text)] == ["a.md"] + + +def test_explicit_same_dir_prefix_stays_a_page_pointer(): + """`./slug.md` is the same file as `slug.md`; classifying it cross-dir + would make the page it names read as an orphan.""" + text = "- [A](./a.md) - hook\n" + r"- [B](.\b.md) - hook" + "\n" + assert [(p.target, p.syntax) for p in mr.parse_pointers_text(text)] == [ + ("a.md", "inline"), ("b.md", "inline")] + + +def test_reference_style_link_definition_counts_as_a_pointer(): + text = "See [the page][ref].\n\n[ref]: ref-slug.md\n" + assert [p.target for p in mr.parse_pointers_text(text)] == ["ref-slug.md"] + + +def test_non_md_and_external_targets_are_not_page_pointers(): + text = ( + "![img](pic.png)\n" + "[url](https://example.com/x.md)\n" + "[cross](../other/x.md)\n" + "[anchor](#section)\n" + ) + ptrs = mr.parse_pointers_text(text) + assert [p.syntax for p in ptrs] == ["external", "external"] + assert not [p for p in ptrs if p.syntax in ("inline", "bare")] + + +def test_pointer_shaped_lines_inside_a_code_fence_are_ignored(): + text = ( + "- real.md - hook\n" + "```bash\n" + "- example.md - this is documentation, not a pointer\n" + "[x](example2.md)\n" + "```\n" + "- [After](after.md) - hook\n" + ) + assert [p.target for p in mr.parse_pointers_text(text)] == [ + "real.md", "after.md"] + + +def test_bare_matcher_rejects_a_non_md_first_token(): + assert mr.parse_pointers_text("- notes.txt - hook\n") == [] + assert mr.parse_pointers_text("- foo.md.bak - hook\n") == [] + + +# --------------------------------------------------------------------------- +# dangling — per-index attribution +# --------------------------------------------------------------------------- +def test_dangling_inside_a_lane_index_is_found_and_attributed(mem): + page(mem, "kept") + write(mem, "MEMORY.md", + "# Index\n\n- [Lane](MEMORY-lane-index.md) - lane\n") + write(mem, "MEMORY-lane-index.md", + "# Lane\n\n- [Kept](kept.md) - hook\n- gone.md - hook\n") + + s = scan(mem) + assert len(s.dangling) == 1 + d = s.dangling[0] + assert (d.index, d.target, d.syntax) == ( + "MEMORY-lane-index.md", "gone.md", "bare") + assert d.line == 4 + assert s.orphans == [] + + +def test_dangling_in_the_root_index_is_attributed_to_the_root_index(mem): + write(mem, "MEMORY.md", "# Index\n\n- [Gone](gone.md) - hook\n") + s = scan(mem) + assert [(d.index, d.target) for d in s.dangling] == [ + ("MEMORY.md", "gone.md")] + + +def test_a_missing_lane_index_is_itself_dangling(mem): + write(mem, "MEMORY.md", "# Index\n\n- [Lane](MEMORY-lane-index.md) - x\n") + s = scan(mem) + assert [(d.index, d.target) for d in s.dangling] == [ + ("MEMORY.md", "MEMORY-lane-index.md")] + + +def test_external_refs_are_never_reported_dangling(mem): + write(mem, "MEMORY.md", + "# Index\n\n- see [general](../general/x.md) and " + "[site](https://example.com/y.md)\n") + s = scan(mem) + assert s.dangling == [] + assert len(s.external["MEMORY.md"]) == 2 + + +# --------------------------------------------------------------------------- +# orphans +# --------------------------------------------------------------------------- +def test_page_carried_by_a_lane_index_is_not_an_orphan(mem): + page(mem, "in-lane") + write(mem, "MEMORY.md", "# Index\n\n- [Lane](MEMORY-lane-index.md) - x\n") + write(mem, "MEMORY-lane-index.md", "# Lane\n\n- in-lane.md - hook\n") + s = scan(mem) + assert s.orphans == [] + + +def test_orphan_page_is_reported(mem): + page(mem, "linked") + page(mem, "stray") + write(mem, "MEMORY.md", "# Index\n\n- [Linked](linked.md) - hook\n") + s = scan(mem) + assert s.orphans == ["stray.md"] + + +def test_state_and_context_files_are_never_orphan_candidates(mem): + write(mem, "STATE.md", "# State\n") + write(mem, "CONTEXT.md", "# Context\n") + write(mem, "MEMORY.md", "# Index\n") + s = scan(mem) + assert s.pages == [] + assert s.orphans == [] + + +def test_root_index_is_not_an_orphan_but_an_unreferenced_lane_index_is(mem): + write(mem, "MEMORY.md", "# Index\n") + write(mem, "MEMORY-lane-index.md", "# Lane\n") + s = scan(mem) + assert s.orphan_indexes == ["MEMORY-lane-index.md"] + assert "MEMORY.md" not in s.orphan_indexes + + +def test_a_lane_index_cannot_self_reference_its_way_out_of_orphanhood(mem): + write(mem, "MEMORY.md", "# Index\n") + write(mem, "MEMORY-lane-index.md", + "# Lane\n\n- [me](MEMORY-lane-index.md) - self\n") + s = scan(mem) + assert s.orphan_indexes == ["MEMORY-lane-index.md"] + + +def test_duplicate_pointer_across_two_indexes_is_informational_not_a_failure(mem): + page(mem, "shared") + write(mem, "MEMORY.md", + "# Index\n\n- [Lane](MEMORY-lane-index.md) - x\n" + "- [Shared](shared.md) - hook\n") + write(mem, "MEMORY-lane-index.md", "# Lane\n\n- shared.md - hook\n") + s = scan(mem) + assert s.duplicates == { + "shared.md": ["MEMORY-lane-index.md", "MEMORY.md"]} + assert s.ok + + +# --------------------------------------------------------------------------- +# sections — the per-section byte breakdown +# --------------------------------------------------------------------------- +def test_section_bytes_partition_the_file_exactly(mem): + text = ( + "# Title\n" + "\n" + "preamble prose\n" + "\n" + "## Small\n" + "- [a](a.md) - hook\n" + "\n" + "## Big\n" + + "".join(f"- [p{i}](p{i}.md) - a longer hook line\n" + for i in range(40)) + ) + p = write(mem, "MEMORY.md", text) + lines, costs = mr.read_lines(str(p)) + assert sum(costs) == os.path.getsize(str(p)) + + secs = mr.parse_sections(lines, costs, mr.parse_pointers(lines)) + assert [s.heading for s in secs] == ["Title", "Small", "Big"] + assert sum(s.bytes for s in secs) == os.path.getsize(str(p)) + # own bytes, not subtree: the H1 row excludes the two `##` blocks below it + assert secs[0].bytes < secs[2].bytes + assert [s.pointers for s in secs] == [0, 1, 40] + # the dominant block is the finding the report exists to surface + assert secs[2].bytes > 0.6 * os.path.getsize(str(p)) + + +def test_content_before_the_first_heading_gets_a_preamble_row(mem): + p = write(mem, "MEMORY.md", "---\ntype: index\n---\n\n## One\nx\n") + lines, costs = mr.read_lines(str(p)) + secs = mr.parse_sections(lines, costs, []) + assert secs[0].heading == "(preamble)" + assert sum(s.bytes for s in secs) == os.path.getsize(str(p)) + + +def test_read_lines_byte_costs_survive_a_missing_trailing_newline(mem): + # write_bytes, not write_text: the point of this test is the exact byte + # total, and write_text would translate "\n" to CRLF on Windows. + p = mem / "MEMORY.md" + p.write_bytes(b"a\nb") + lines, costs = mr.read_lines(str(p)) + assert lines == ["a", "b"] + assert sum(costs) == os.path.getsize(str(p)) == 3 + + +# --------------------------------------------------------------------------- +# report +# --------------------------------------------------------------------------- +def test_report_leads_with_auto_loaded_bytes_and_labels_pages_non_context(mem): + page(mem, "one") + page(mem, "two") + write(mem, "STATE.md", "# State\n") + write(mem, "MEMORY.md", + "# Index\n\n## Pages\n- [One](one.md) - h\n- [Two](two.md) - h\n") + + data = mr.report_data(scan(mem)) + files = {r["file"]: r for r in data["auto_loaded"]} + assert set(files) == {"MEMORY.md", "STATE.md"} + assert files["MEMORY.md"]["budget"] == mr.MEMORY_BUDGET_BYTES + assert files["MEMORY.md"]["over"] is False + assert data["on_disk"]["pages"] == 2 + assert data["on_disk"]["auto_loaded"] is False + + text = mr.render_report(data) + assert "NOT auto-loaded" in text + assert "per section" in text + + +def test_report_flags_a_file_over_the_ceiling(mem): + write(mem, "MEMORY.md", "# Index\n" + "x" * 500 + "\n") + data = mr.report_data(scan(mem), budget=100) + row = [r for r in data["auto_loaded"] if r["file"] == "MEMORY.md"][0] + assert row["over"] is True + assert "OVER" in mr.render_report(data) + + +def test_report_lists_per_index_pointer_counts_for_every_index(mem): + page(mem, "a") + page(mem, "b") + write(mem, "MEMORY.md", "# Index\n\n- [Lane](MEMORY-lane-index.md) - x\n") + write(mem, "MEMORY-lane-index.md", "# Lane\n\n- a.md - h\n- [B](b.md) - h\n") + + data = mr.report_data(scan(mem)) + by_file = {i["file"]: i for i in data["indexes"]} + assert set(by_file) == {"MEMORY.md", "MEMORY-lane-index.md"} + assert by_file["MEMORY.md"]["auto_loaded"] is True + assert by_file["MEMORY-lane-index.md"]["auto_loaded"] is False + assert by_file["MEMORY-lane-index.md"]["pointers"] == 2 + assert by_file["MEMORY-lane-index.md"]["bare"] == 1 + assert by_file["MEMORY-lane-index.md"]["inline"] == 1 + + +def test_report_tolerates_a_dir_with_no_memory_md(mem): + page(mem, "loose") + data = mr.report_data(scan(mem)) + assert data["sections"] == [] + assert "No MEMORY.md" in mr.render_report(data) + + +# --------------------------------------------------------------------------- +# exit-code contract — a verifier that fails open is worse than none +# --------------------------------------------------------------------------- +def test_verify_exits_zero_on_a_clean_dir(mem, capsys): + page(mem, "one") + write(mem, "MEMORY.md", "# Index\n\n- [One](one.md) - hook\n") + assert mr.main(["--verify", str(mem)]) == 0 + assert "OK: index is intact." in capsys.readouterr().out + + +def test_verify_exits_one_on_a_dangling_pointer_in_a_lane_index(mem, capsys): + write(mem, "MEMORY.md", "# Index\n\n- [Lane](MEMORY-lane-index.md) - x\n") + write(mem, "MEMORY-lane-index.md", "# Lane\n\n- gone.md - hook\n") + assert mr.main(["--verify", str(mem)]) == 1 + out = capsys.readouterr().out + assert "FAIL" in out + assert "MEMORY-lane-index.md" in out and "gone.md" in out + + +def test_verify_exits_one_on_an_orphan(mem, capsys): + page(mem, "stray") + write(mem, "MEMORY.md", "# Index\n") + assert mr.main(["--verify", str(mem)]) == 1 + assert "stray.md" in capsys.readouterr().out + + +def test_verify_exits_one_on_an_unreferenced_lane_index(mem): + write(mem, "MEMORY.md", "# Index\n") + write(mem, "MEMORY-lane-index.md", "# Lane\n") + assert mr.main(["--verify", str(mem)]) == 1 + + +def test_report_exits_zero_even_when_the_index_is_broken(mem): + """--report measures; it never gates. Only --verify carries the contract.""" + page(mem, "stray") + write(mem, "MEMORY.md", "# Index\n\n- [Gone](gone.md) - hook\n") + assert mr.main(["--report", str(mem)]) == 0 + + +def test_missing_target_exits_two(mem, tmp_path, capsys): + assert mr.main(["--verify", str(tmp_path / "nope")]) == 2 + assert "not found" in capsys.readouterr().err + + +def test_no_mode_flag_defaults_to_report(mem, capsys): + write(mem, "MEMORY.md", "# Index\n\n- [Gone](gone.md) - hook\n") + assert mr.main([str(mem)]) == 0 + assert "Reindex report" in capsys.readouterr().out + + +def test_both_flags_run_both_and_the_exit_code_comes_from_verify(mem, capsys): + write(mem, "MEMORY.md", "# Index\n\n- [Gone](gone.md) - hook\n") + assert mr.main(["--report", "--verify", str(mem)]) == 1 + out = capsys.readouterr().out + assert "Reindex report" in out and "Reindex verify" in out + + +def test_dispatcher_subcommand_token_is_tolerated(mem): + write(mem, "MEMORY.md", "# Index\n") + assert mr.main(["reindex", "--verify", str(mem)]) == 0 + + +# --------------------------------------------------------------------------- +# --json (the machine contract downstream skills read) +# --------------------------------------------------------------------------- +def test_json_verify_shape(mem, capsys): + write(mem, "MEMORY.md", "# Index\n\n- [Lane](MEMORY-lane-index.md) - x\n") + write(mem, "MEMORY-lane-index.md", "# Lane\n\n- gone.md - hook\n") + assert mr.main(["--verify", "--json", str(mem)]) == 1 + data = json.loads(capsys.readouterr().out) + assert data["mode"] == "verify" + assert data["ok"] is False + assert data["dangling"] == [{"index": "MEMORY-lane-index.md", + "target": "gone.md", "line": 3, + "syntax": "bare"}] + # path assertions derive from the code's own normalizer, never a literal + assert data["dir"] == os.path.abspath(str(mem)) + + +def test_json_report_shape(mem, capsys): + page(mem, "one") + write(mem, "MEMORY.md", "# Index\n\n## Pages\n- [One](one.md) - h\n") + assert mr.main(["--report", "--json", str(mem)]) == 0 + data = json.loads(capsys.readouterr().out) + assert data["mode"] == "report" + assert [s["heading"] for s in data["sections"]] == ["Index", "Pages"] + assert data["dir"] == os.path.abspath(str(mem)) + + +# --------------------------------------------------------------------------- +# the curate inventory script shares this parser (no second implementation) +# --------------------------------------------------------------------------- +def test_curate_inventory_uses_the_shared_multi_index_parser(mem): + import importlib.util + + root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + path = os.path.join(root, "skills", "okfmem-curate", "scripts", + "inventory.py") + spec = importlib.util.spec_from_file_location("curate_inventory", path) + inv = importlib.util.module_from_spec(spec) + spec.loader.exec_module(inv) + + page(mem, "in-lane") + write(mem, "MEMORY.md", "# Index\n\n- [Lane](MEMORY-lane-index.md) - x\n") + write(mem, "MEMORY-lane-index.md", "# Lane\n\n- in-lane.md - hook\n") + + # the page a lane index carries is a page, and it is not an orphan + assert inv.list_md_files(str(mem)) == ["in-lane.md"] + assert mr.Scan(str(mem)).orphans == [] + + +# --------------------------------------------------------------------------- +# retired ck_*.md snapshots are not orphan candidates +# +# Every sibling pass already skips them by this exact prefix test +# (`memory_consolidate.scan_project`, `memory_backfill`), and curate flags them +# `ck_snapshot` cruft. Counting them here made `--verify` fail on most real +# projects for a known-benign reason — and a gate nobody can turn on is not a +# gate, so the mode could not serve as the consolidation-hook gate #54 +# describes. +# --------------------------------------------------------------------------- +def test_ck_snapshots_are_not_orphan_candidates(mem): + page(mem, "real") + write(mem, "ck_2026-01-01_session.md", "# snapshot\n") + write(mem, "ck_notes.md", "# snapshot, no date in the name\n") + write(mem, "MEMORY.md", "# Index\n\n- [Real](real.md) - hook\n") + + s = scan(mem) + assert s.pages == ["real.md"] + assert s.orphans == [] + assert s.ok + + +def test_verify_exits_zero_with_unindexed_ck_snapshots(mem): + page(mem, "real") + for i in range(3): + write(mem, f"ck_2026-01-0{i + 1}_session.md", "# snapshot\n") + write(mem, "MEMORY.md", "# Index\n\n- [Real](real.md) - hook\n") + assert mr.main(["--verify", str(mem)]) == 0 + + +def test_ck_skip_uses_the_same_prefix_test_as_the_sibling_passes(mem): + assert mr.is_ck_snapshot("ck_anything.md") + assert not mr.is_ck_snapshot("check-this.md") # prefix, not substring + assert not mr.is_ck_snapshot("page-ck_x.md") + + +def test_curate_inventory_still_sees_ck_snapshots_via_include_ck(mem): + """Curate is the one pass that must still *see* them, so it can propose + deleting them — so the skip is opt-out, not unconditional.""" + page(mem, "real") + write(mem, "ck_2026-01-01_session.md", "# snapshot\n") + assert mr.page_files(str(mem)) == ["real.md"] + assert mr.page_files(str(mem), include_ck=True) == [ + "ck_2026-01-01_session.md", "real.md"] + + +def test_an_indexed_ck_file_is_still_not_dangling(mem): + """Skipping them as orphan *candidates* must not make them invisible as + link *targets* — the file is on disk, so a pointer at it resolves.""" + write(mem, "ck_2026-01-01_session.md", "# snapshot\n") + write(mem, "MEMORY.md", "# Index\n\n- ck_2026-01-01_session.md - hook\n") + assert scan(mem).dangling == [] + + +# --------------------------------------------------------------------------- +# what `--verify` actually checks for an orphan index (the skill text says so +# too): NO inbound link from ANY other index. A chain passes. +# --------------------------------------------------------------------------- +def test_a_lane_linked_only_from_another_lane_is_not_an_orphan_index(mem): + write(mem, "MEMORY.md", "# Root\n\n- MEMORY-a.md - covers a\n") + write(mem, "MEMORY-a.md", "# A\n\n- MEMORY-b.md - covers b\n") + write(mem, "MEMORY-b.md", "# B\n") + assert scan(mem).orphan_indexes == [] + + +# --------------------------------------------------------------------------- +# --budget-check: pointer lines over the per-line CHARACTER budget (#52) +# +# The `awk` one-liner this replaced was wrong in both directions at once: a +# `/^- \[/` guard is blind to every bare pointer (exactly what a lane index +# holds), and BSD `awk`'s `length($0)` counts BYTES, so the pointer +# convention's em-dash over-reported every line carrying one. +# --------------------------------------------------------------------------- +def test_budget_check_counts_bare_pointers_the_awk_guard_missed(mem): + """The reviewer's reproduction fixture, verbatim in shape: two rich + pointers at the root (one short, one under budget) and two bare pointers in + a lane (one far over). awk reported 0/2; the truth is 1/4.""" + for slug in ("a", "b", "c", "d"): + page(mem, slug) + write(mem, "MEMORY.md", + "# Index\n\n" + "- [A](a.md) - hook\n" + "- [B](b.md) - " + "z" * 130 + "\n") + write(mem, "MEMORY-lane.md", + "# Lane\n\n" + "- c.md - " + "y" * 200 + "\n" + "- d.md - short\n") + + data = mr.budget_data(scan(mem)) + assert data["pointer_lines"] == 4 + assert len(data["over"]) == 1 + assert data["over"][0]["index"] == "MEMORY-lane.md" + assert data["over"][0]["targets"] == ["c.md"] + assert data["over"][0]["chars"] == 209 + assert data["limit"] == mr.POINTER_BUDGET_CHARS + + +def test_budget_check_counts_characters_not_bytes(mem): + """A line of multibyte text: `len` on the decoded line is characters, so a + line under the character budget must NOT be reported, even though its byte + length is over it. BSD awk's byte count is what made this fail open.""" + page(mem, "accented") + hook = "é" * 128 + " — café" + line = "- accented.md " + hook + assert len(line) <= mr.POINTER_BUDGET_CHARS + assert len(line.encode("utf-8")) > mr.POINTER_BUDGET_CHARS + write(mem, "MEMORY.md", "# Index\n\n" + line + "\n") + + data = mr.budget_data(scan(mem)) + assert data["pointer_lines"] == 1 + assert data["over"] == [] + assert data["ok"] + + +def test_budget_check_reports_a_multibyte_line_that_is_genuinely_over(mem): + """The other half of the same guard: characters must still be counted, not + ignored — an em-dash line really over the budget is reported.""" + page(mem, "accented") + line = "- accented.md — " + "é" * 200 + write(mem, "MEMORY.md", "# Index\n\n" + line + "\n") + + data = mr.budget_data(scan(mem)) + assert [r["chars"] for r in data["over"]] == [len(line)] + + +def test_budget_check_excludes_routing_rows_but_reports_their_count(mem): + """A routing-table row's `covers` clause is deliberately a sentence or two, + so holding it to the page-pointer budget would flag every correctly + reindexed store. Excluded from the budget, never silently dropped.""" + page(mem, "one") + write(mem, "MEMORY.md", + "# Index\n\n" + "- MEMORY-lane.md - " + "c" * 200 + "\n" + "- [One](one.md) - hook\n") + write(mem, "MEMORY-lane.md", "# Lane\n") + + data = mr.budget_data(scan(mem)) + assert data["routing_rows"] == 1 + assert data["pointer_lines"] == 1 + assert data["over"] == [] + + +def test_budget_check_is_advisory_and_always_exits_zero(mem, capsys): + page(mem, "one") + write(mem, "MEMORY.md", "# Index\n\n- [One](one.md) - " + "z" * 200 + "\n") + assert mr.main(["--budget-check", str(mem)]) == 0 + out = capsys.readouterr().out + assert "1/1 pointer lines over the 150-char budget" in out + # the mode flag selects ONLY that mode -- no report is emitted alongside it + assert "Auto-loaded bytes" not in out + + +def test_budget_check_json_shape(mem, capsys): + page(mem, "one") + write(mem, "MEMORY.md", "# Index\n\n- [One](one.md) - " + "z" * 200 + "\n") + assert mr.main(["--budget-check", "--json", str(mem)]) == 0 + data = json.loads(capsys.readouterr().out) + assert data["mode"] == "budget" + assert data["limit"] == 150 + assert data["ok"] is False + assert data["over"][0]["line"] == 3 + assert data["dir"] == os.path.abspath(str(mem)) + + +def test_budget_check_composes_with_the_other_modes(mem, capsys): + """Two or more modes -> the wrapper object, each under its own key, and the + exit code still comes from --verify alone.""" + page(mem, "one") + write(mem, "MEMORY.md", "# Index\n\n- [One](one.md) - hook\n") + assert mr.main(["--verify", "--budget-check", "--json", str(mem)]) == 0 + data = json.loads(capsys.readouterr().out) + assert sorted(data) == ["budget", "verify"] + + +def test_budget_check_exit_stays_zero_even_when_verify_would_fail(mem): + """Advisory means advisory: a broken index does not make this mode fail, + and an over-budget line does not make it fail either.""" + write(mem, "MEMORY.md", "# Index\n\n- [Gone](gone.md) - " + "z" * 200 + "\n") + assert mr.main(["--budget-check", str(mem)]) == 0 + assert mr.main(["--verify", str(mem)]) == 1 + + +# --------------------------------------------------------------------------- +# the curate inventory script survives the tier-3 managed-copy install +# --------------------------------------------------------------------------- +def test_inventory_script_runs_from_a_copied_install(tmp_path): + """`memory_init._make_link`'s copytree fallback (Windows without symlink + privilege or junction) copies the skill dir OUT of the repo, so resolving + the engine as four dirnames up lands on the harness skills parent, which + holds no engine modules. That used to be a bare ModuleNotFoundError + traceback with no output at all — not even a degraded report.""" + import shutil + import subprocess + import sys + + root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + copy = tmp_path / "copy" + (copy / "skills").mkdir(parents=True) + shutil.copytree(os.path.join(root, "skills", "okfmem-curate"), + str(copy / "skills" / "okfmem-curate")) + script = copy / "skills" / "okfmem-curate" / "scripts" / "inventory.py" + + memdir = tmp_path / "memdir" + memdir.mkdir() + page(memdir, "one") + write(memdir, "MEMORY.md", "# Index\n\n- [One](one.md) - hook\n") + + # A home with no ~/okfmem in it, on both platform conventions (ntpath's + # expanduser reads USERPROFILE, posixpath's reads HOME), so the last-resort + # candidate genuinely misses. + empty_home = tmp_path / "home" + empty_home.mkdir() + env = {k: v for k, v in os.environ.items() + if k not in ("OKFMEM_ENGINE", "PYTHONPATH")} + env["HOME"] = env["USERPROFILE"] = str(empty_home) + + miss = subprocess.run([sys.executable, str(script), str(memdir)], + capture_output=True, text=True, env=env, + cwd=str(tmp_path)) + assert miss.returncode == 2 + assert "Traceback" not in miss.stderr + assert len(miss.stderr.strip().splitlines()) == 1 + assert "okfmem engine not found" in miss.stderr + assert "inventory.py" in miss.stderr # the command to run instead + + # ... and $OKFMEM_ENGINE rescues it, producing a real report. + env["OKFMEM_ENGINE"] = root + hit = subprocess.run([sys.executable, str(script), str(memdir)], + capture_output=True, text=True, env=env, + cwd=str(tmp_path)) + assert hit.returncode == 0, hit.stderr + assert "# Memory inventory:" in hit.stdout + assert "one.md" in hit.stdout diff --git a/tests/test_status_inventory.py b/tests/test_status_inventory.py index 6e13e6d..4d7e910 100644 --- a/tests/test_status_inventory.py +++ b/tests/test_status_inventory.py @@ -4,6 +4,7 @@ import pytest import memory_init as mi +from memory_reindex import page_files # --------------------------------------------------------------------------- @@ -18,21 +19,28 @@ # --------------------------------------------------------------------------- def make_project(store, name, *, pages=0, archived=None, memory_lines=None, - state=False, extra_files=()): + memory_bytes=None, state=False, extra_files=()): """Create <store>/projects/<name>/ with the requested contents. ``archived`` = None -> no archive/ dir; an int -> archive/ dir with that many .md files. ``memory_lines`` = None -> no MEMORY.md; an int -> MEMORY.md - with exactly that many lines. + with exactly that many lines. ``memory_bytes`` (mutually exclusive with + ``memory_lines``) writes a MEMORY.md padded to exactly that many bytes — + used for the byte-ceiling tests, where the exact line count doesn't matter. """ d = store / "projects" / name d.mkdir(parents=True) for i in range(pages): (d / f"page{i}.md").write_text(f"# page {i}\n", encoding="utf-8") if memory_lines is not None: + # newline="\n" pins the on-disk bytes to LF on every platform: without + # it Path.write_text translates to CRLF on Windows and any byte-count + # expectation computed from this same string is off by one per line. (d / "MEMORY.md").write_text("\n".join(f"line {i}" for i in range(memory_lines)) + "\n", - encoding="utf-8") + encoding="utf-8", newline="\n") + elif memory_bytes is not None: + (d / "MEMORY.md").write_bytes(b"x" * memory_bytes) if state: (d / "STATE.md").write_text("state\n", encoding="utf-8") if archived is not None: @@ -64,12 +72,27 @@ def row_for(inv, name): def test_memory_and_state_excluded_from_pages(store): make_project(store, "proj", pages=3, memory_lines=10, state=True) - name, pages, archived, mem_lines, has_state, has_arch = row_for( + name, pages, archived, mem_bytes, has_state, has_arch = row_for( mi.project_inventory(str(store)), "proj") # 3 real pages; MEMORY.md and STATE.md are NOT pages. assert pages == 3 assert has_state is True - assert mem_lines == 10 + assert mem_bytes == len("\n".join(f"line {i}" for i in range(10)) + "\n") + + +def test_lane_indexes_are_not_counted_as_pages(store): + """#53's lane routing creates MEMORY-<lane>.md index files. A root-only + "not MEMORY.md/STATE.md" filter counts every one of them as a durable page, + so the reported page total inflates by one per lane on exactly the stores + the lane-routing rule tells users to build. Pages come from the engine's + page_files(), which knows an index from a page.""" + make_project(store, "proj", pages=2, memory_lines=3, state=True, + extra_files=("MEMORY-parser.md", "MEMORY-tooling.md", + "CONTEXT.md", "ck_2026-01-01_snap.md")) + _, pages, _, _, _, _ = row_for(mi.project_inventory(str(store)), "proj") + assert pages == 2 + # The engine's own enumerator is the single source of truth for this rule. + assert pages == len(page_files(str(store / "projects" / "proj"))) def test_archive_counted(store): @@ -97,36 +120,60 @@ def test_empty_archive_dir_distinguishable_from_missing(store): # --------------------------------------------------------------------------- -# MEMORY.md line count + the 200-line auto-load boundary +# MEMORY.md byte count + the auto-load byte ceiling (#53 — supersedes the old +# 200-line trigger: a store can sit well under 200 lines while over the byte +# ceiling once pointers run long, since bytes and lines diverge whenever +# pointer length isn't uniform). # --------------------------------------------------------------------------- -def test_memory_line_count_exact(store): - make_project(store, "proj", memory_lines=137) - _, _, _, mem_lines, _, _ = row_for( +def test_memory_byte_count_exact(store): + make_project(store, "proj", memory_bytes=500) + _, _, _, mem_bytes, _, _ = row_for( mi.project_inventory(str(store)), "proj") - assert mem_lines == 137 + assert mem_bytes == 500 -def test_no_memory_file_is_zero_lines(store): +def test_no_memory_file_is_zero_bytes(store): make_project(store, "proj", pages=1, memory_lines=None) - _, _, _, mem_lines, _, _ = row_for( + _, _, _, mem_bytes, _, _ = row_for( mi.project_inventory(str(store)), "proj") - assert mem_lines == 0 + assert mem_bytes == 0 + +def test_byte_ceiling_shared_with_reindex_engine(): + # The ceiling has exactly one home (memory_reindex.MEMORY_BUDGET_BYTES); + # memory_init imports it rather than restating the number (#53). + assert mi.MEMORY_BUDGET_BYTES == 8192 -def test_autoload_boundary_200_no_warn_201_warn(store): - make_project(store, "at_limit", memory_lines=200) - make_project(store, "over_limit", memory_lines=201) + +def test_autoload_boundary_at_ceiling_no_warn_over_warns(store): + make_project(store, "at_limit", memory_bytes=mi.MEMORY_BUDGET_BYTES) + make_project(store, "over_limit", memory_bytes=mi.MEMORY_BUDGET_BYTES + 1) inv = mi.project_inventory(str(store)) at = row_for(inv, "at_limit")[3] over = row_for(inv, "over_limit")[3] - assert at == 200 - assert over == 201 - # The warning fires on strictly-greater-than the constant: 200 is clean, - # 201 trips. This is the exact predicate cmd_status renders. - assert mi.MEMORY_AUTOLOAD_LINES == 200 - assert (at > mi.MEMORY_AUTOLOAD_LINES) is False - assert (over > mi.MEMORY_AUTOLOAD_LINES) is True + assert at == mi.MEMORY_BUDGET_BYTES + assert over == mi.MEMORY_BUDGET_BYTES + 1 + # The warning fires on strictly-greater-than the ceiling: at-budget is + # clean, one byte over trips. This is the exact predicate cmd_status + # renders. + assert (at > mi.MEMORY_BUDGET_BYTES) is False + assert (over > mi.MEMORY_BUDGET_BYTES) is True + + +def test_lines_can_stay_low_while_bytes_trip_the_ceiling(store): + # The scenario #53 exists to fix: few lines, long pointers -> bytes blow + # the ceiling while the old line-count trigger would never fire. + long_pointer = "- [" + ("x" * 200) + "](slug.md) -- hook\n" + n_lines = 40 + text = long_pointer * n_lines + (store / "projects" / "proj").mkdir(parents=True) + (store / "projects" / "proj" / "MEMORY.md").write_text( + text, encoding="utf-8") + _, _, _, mem_bytes, _, _ = row_for( + mi.project_inventory(str(store)), "proj") + assert n_lines < 200 # old trigger: would not fire + assert mem_bytes > mi.MEMORY_BUDGET_BYTES # byte trigger: fires # ---------------------------------------------------------------------------