feat(reindex): two-tier index — engine, write-time budgets, lane routing, skill

[s-annam]

Summary

The always-loaded index bloats where it is written, not where it is cleaned. MEMORY.md is one of only two files that cost context at session start, yet the pointer budget lived in a monthly curate pass while capture runs every session. The restructure trigger compounded it by counting lines when the budget that actually blows is bytes — a store can sit at 60% of the line trigger and 110% of the byte budget.

This lands the measurement engine, moves both budgets to the write path, makes the index two-tier by default, and adds the skill that performs the carve.

• Reindex engine: per-section byte report + --verify across all MEMORY*.md, both link syntaxes #54 — new memory_reindex.py engine behind an okfmem reindex dispatcher route. --report gives auto-loaded bytes vs ceiling, a per-section byte breakdown (so a single dominant block is visible — total file size alone can't tell you whether the remedy is tighter hooks or a lane split), per-index pointer counts, and on-disk pages explicitly labelled non-context. --verify walks every MEMORY*.md in both link syntaxes, attributes each dangling pointer to the index it came from, and exits non-zero. --budget-check counts per-pointer length in characters, not bytes. Verified against a real store: 0 dangling / 0 orphans where the old checker reported 614 false positives.
• Budget the index at write time; make curate report auto-loaded bytes, not file count #52 — the ≤150-char pointer budget moves from "documented in curate, checked rarely" to enforced at write time in okfmem-save; STATE.md gains a stated 8192-byte ceiling; okfmem-curate reports auto-loaded bytes and tokens-per-session instead of file counts, and states plainly that deleting pages is not a context optimisation (pages on disk cost zero context).
• Two-tier index: route pointers to lane indexes, make MEMORY.md a map of content #53 — okfmem-save routes new pointers to a lane index by default, with the root reserved for genuinely cross-lane and type: feedback pages. The restructure trigger becomes MEMORY.md bytes over a ceiling rather than a line count, surfaced by both okfmem status and curate Phase 2. The remedy is a lane split, never tighter hooks; split pointers are never re-flattened.
• /okfmem-reindex skill: cluster lanes, move pointers, rewrite MEMORY.md as a routing table #55 — new /okfmem-reindex skill: five phases (measure via the engine, cluster lanes including cross-cutting ones, propose behind a hard approval stop, execute the carve, verify via the engine). Auto-wired by okfmem init's existing skill glob, inside the existing apply_config consent gate.

Closes #52
Closes #53
Closes #54
Closes #55

Test plan

• python3 scripts/check-leaks.py exits 0 (69 tracked files)
• python3 -m pytest tests/ — 314 passed, 1 skipped (baseline before this branch: 292/1)
• Suite re-run under simulated Windows CRLF newline translation — 314 passed (the simulation was proven to bite: it reproduces the 80-vs-70 byte mismatch before the fix)
• ruff check . reviewed — only the 2 pre-existing E702s in plugins/memory_search.py
• Smoke: okfmem status, okfmem reindex --report, okfmem reindex --verify, okfmem init --dry-run all run clean
• Full OS-matrix verify — CI gate

Adversarial review

Two bounded review rounds ran against the accumulated diff before this PR existed, each reproducing findings end-to-end rather than reasoning about the code. Six blocking defects were found and fixed; every fix has a fail-before proof.

Round 1 — 3 blocking, all fixed:

1. Archive/graduate stranded lane pointers. memory_consolidate.py and memory_graduate.py dropped pointers only from the root MEMORY.md. Composed with Two-tier index: route pointers to lane indexes, make MEMORY.md a map of content #53's lane routing, the unattended consolidation Stop hook left a dangling pointer behind on every archived page — silently breaking the very --verify gate both new skills depend on, with no user action. Reproduced: --verify clean before, dangling: 1 and exit 1 after a single consolidate run. Both now enumerate every index; graduate's rollback snapshots each index it writes (verified by injecting a mid-loop failure and confirming full restoration — a partial rollback would be worse than the original bug).
2. The new curate budget check was fail-open BSD shell — an awk one-liner blind to bare-form pointers (the form lane indexes actually use) and counting bytes on macOS, so em-dash pointers over-reported. It printed 0/2 where the truth was 1/4. Replaced by the engine's --budget-check; no shell survives in that path.
3. inventory.py crashed under the managed-copy install. A new module-level import resolved four dirnames up, which does not hold under the copytree fallback (Windows without symlink privilege). Produced a bare ModuleNotFoundError traceback and no output. Now probes 4-up → $OKFMEM_ENGINE → ~/okfmem and fails with one actionable line, never a traceback.

Plus one finding promoted from nit to blocking: --verify counted retired ck_*.md snapshots as orphans, failing on 10 of 14 real projects for a known-benign reason. A gate that fails on most real stores is a gate nobody turns on. Now skipped via the siblings' exact convention (10 → 6, the remainder genuine).

Round 2 — 3 blocking, all fixed:

1. Two Windows-only test failures of the hand-typed-literal class this repo has been bitten by repeatedly: byte-count expectations written as LF literals against files created through Path.write_text, which translates to CRLF on Windows. Both would have failed only on the windows-latest matrix leg. Fixed by pinning the fixture newline and using write_bytes where the byte total is the point.
2. A live sibling instance of round 1's defect class: project_inventory still filtered pages root-only, so every MEMORY-<lane>.md counted as a durable page — okfmem status over-reporting on precisely the stores Two-tier index: route pointers to lane indexes, make MEMORY.md a map of content #53 tells users to create (635 vs 628 on a real store). Now delegates to the engine's page_files(), with a regression test confirmed to fail 6 == 2 against the old filter.

Round 3 — caught by CI, not by the loop. The first push failed the windows-latest leg on a third instance of the same hand-typed-POSIX-literal class: tests/test_consolidate_index_drop.py asserted "archive projects/demo/old-page.md" against output the code builds with os.path.relpath, which renders backslashes on Windows. Fixed by deriving the expected value from os.path.join, and the whole diff was then swept for the rest of the class (the only other slashed literals are a 1/1 ratio in prose and the engine's markdown-link parsing, where / is correct on every OS). Worth recording: two review rounds plus a targeted audit still missed one, and the OS matrix is what found it.

Surviving nits, deliberately not iterated on (recorded for the reviewer's eye): INLINE_LINK_RE does not skip inline code spans, so a .md link inside backticks parses as a pointer; classify_target is case-sensitive on .MD; a bare okfmem invocation survives in okfmem-save/SKILL.md matching that file's pre-existing house style; --budget (byte ceiling) and --budget-check (char budget) read as related but are not. Two pre-existing items were found and left for separate issues: curate Phase 1's SLUG=$(pwd -P | sed 's|/|-|g'), which encode_root supersedes, and plugins/memory_distill.py reading only the root index.

The round-2 fixes were applied and verified mechanically (fail-before proofs, the CRLF simulation, and a read-only check against a real store) but were not put through a third independent review round — the loop's cap is two.

Provenance

Stage	Model	Effort
Implementation — #54 (reindex engine)	Claude Opus 5	ultra
Implementation — #52 (pointer budget)	Claude Sonnet 5	high
Implementation — #53 (lane routing)	Claude Sonnet 5	high
Implementation — #55 (reindex skill)	Claude Sonnet 5	high
Adversarial review — round 1	Claude Opus 5	high
Review fixes — round 1	Claude Opus 5	ultra
Adversarial review — round 2	Claude Opus 5	high
Review fixes — round 2 + orchestration + PR	Claude Opus 5	session default
Verification	CI verify	—

Every model string above is self-reported by the agent that ran that stage; the orchestrator requested an alias and cannot infer what it resolved to.