Generic file-store migration system (JSONL + markdown) with CLI commands + app-wide wiring

[juliarvalenti]

Generic store-migration system (file-based, JSONL + markdown) with CLI + app-wide wiring

Mycelium has no database — state is markdown-with-frontmatter, JSONL, JSON, and TOML files under .mycelium/. Today a format/schema change (e.g. #497/#499 moving the transcript from fenced transcript.md → plain transcript.jsonl) has no migration path: we hand-migrate the local dev service and move on. That's fine pre-release (no users), but we'll need a real system before anything hosted/multi-user, and before the next breaking format change lands on a store someone cares about.

This issue tracks a generic, versioned migration framework for the file store — Alembic/Django-migrations in spirit, but for files, spanning both JSONL and markdown surfaces — with CLI commands and interactions wired through the whole app.

Storage surfaces a migration system must cover

• Room transcripts — .mycelium/rooms/{room}/log/transcript.jsonl (JSONL; just changed format in fix(persister): append-only JSONL transcript + transcript-backed read path (#497) #499). TRANSCRIPT_FILENAME, fastapi-backend/app/services/persister.py.
• Memory files — .mycelium/rooms/{room}/{namespace}/{key}.md, YAML frontmatter carries a version field already. serialize_memory/parse_memory, fastapi-backend/app/services/filesystem.py.
• Search index — .mycelium/rooms/{room}/.search-index.jsonl (JSONL). Already has a rebuild path (mycelium reindex → services/reindex, services/indexer, search_index.py); a migration system should reuse/subsume it, not duplicate.
• Delivery cursors — .mycelium/rooms/{room}/log/.delivery-cursors.json (JSON). _cursors_path, persister.py.
• Episode records — .mycelium/rooms/{room}/log/episodes/{id}.md (markdown). l9_episode.py.
• Plan namespace — .mycelium/rooms/{room}/plan/*.md.
• L9 envelope schema — records embed an SSTP version (currently 0.0.6); a bump may need a record rewrite.
• Project config — .mycelium/config.toml. (mycelium config apply regenerates .env; related but distinct.)

What "generic" should mean

• A store schema version persisted somewhere durable (e.g. .mycelium/.store-version or a [store] version in config.toml), separate from per-file version fields.
• An ordered set of migration steps, each declaring from → to, idempotent, resumable, and format-aware: helpers for "rewrite every JSONL line under glob", "transform frontmatter across every *.md under glob", "rename/move a file", "drop a fenced wrapper", etc.
• Dry-run + diff before apply; backup/rollback (or at least a copy-aside) since these mutate user files in place.
• Per-room and whole-store scoping (rooms are independent folders; a migration may run room-by-room).
• Fail-soft + logged: a partial/failed migration must never orphan data; report exactly what was and wasn't migrated (no silent truncation).

CLI commands (mycelium-cli)

• mycelium migrate status — current store version, pending migrations, per-surface counts.
• mycelium migrate up [--to VERSION] [--dry-run] [--room NAME] — apply pending (all or scoped), with a diff/preview in dry-run.
• mycelium migrate plan — show what each pending step would touch (globs, file counts) without running.
• mycelium migrate rollback [--to VERSION] — revert to a prior version where a step defines a reverse.
• Fold the existing mycelium reindex in as a migration primitive (rebuild-from-source is the canonical "reindex" step).

Interactions through the app

• Backend on startup — detect a store older than the code's expected version; refuse to serve (or auto-migrate behind a flag) rather than read a stale format. Wire into the provision/room-load path (room_channels.py, persister.py init) and /health (surface store_version + pending_migrations).
• mycelium doctor — warn on a pending migration, point at mycelium migrate status.
• Frontend — surface a "store needs migration" banner when /health reports pending; don't silently render an empty/half-read room.
• CLI guardrails — commands that read/write the store check version compatibility and prompt to migrate on mismatch.
• Sharing (git) — a pulled room from a peer on a newer store version must be detected and migrated (or rejected) on first local touch.

Non-goals / notes

• Not needed for the current release (no users). This is groundwork for hosted/multi-user, which per CLAUDE.md is gated behind real identity (JWT/SPIRE) anyway.
• Prompted by fix(persister): append-only JSONL transcript + transcript-backed read path (#497) #499 (transcript .md → .jsonl), where we deliberately ripped out one-off migration code rather than carry it — a generic system is the right home for that logic.