Room transcript: unbounded whole-file rewrites, markdown-wrapped JSONL, and a dual message store the read path ignores

[juliarvalenti]

Surfaced while poking at a room whose live view went empty after a backend restart. Three distinct problems, all in the transcript / message-store layer.

1. Unbounded growth + O(n) writes (the scaling bug)

write_transcript(room, records) re-renders the entire record list and rewrites the whole log/transcript.md on every recorded message (app/services/persister.py:326 → render_transcript at :282). And load_transcript parses the whole block back on every read (:315).

So per-message cost is O(n) (rewrite all N records), total file size is unbounded, and every load re-parses the full history into memory. A busy or long-lived room degrades quadratically in write cost and grows without bound — no rotation, truncation, or segmenting.

Directions:

• Append-only writes instead of whole-file re-render.
• Segment/rotate (e.g. log/transcript/NNNN.jsonl) with a bounded tail kept hot.
• Cap/compact old history (episodes already get their own records under log/episodes/).

2. It's markdown-wrapped JSONL (the format smell)

The transcript is semantically JSONL — one JSON record per line — but it's wrapped in a fenced ```jsonl block inside a .md file with a header (render_transcript/parse_transcript, persister.py:282/:295). The wrapper is cosmetic (it rides the generic write_memory_file "everything is a markdown memory" path) but it:

• forces the whole-file re-render in (1) — you can't cleanly append a line to a fenced block,
• adds a fence-parsing step on every load,
• isn't actually human-friendlier than plain JSONL.

Direction: make it a real append-only log/transcript.jsonl (or segmented), drop the markdown fence.

3. Two stores, and the read path only sees one (the coherence bug)

There are two message stores:

• Durable transcript (transcript.md, disk) — written by the persister/ingest_local; survives restarts; what await/the aligner read via the delivery cursor.
• Live list (local_state, in-memory) — what GET /messages reads (app/routes/messages.py:145); wiped on every backend restart.

Consequences observed:

• Restart flushes the room view. GET /messages returns empty after a restart even though transcript.md has the full history — it never reads the transcript.
• respond vs room send land in different stores. mycelium respond (/reply) writes the transcript only; mycelium room send (broadcast) writes local_state. So whether a message shows in GET /messages depends on the path taken (and whether a SLIM channel is live). Same room, two lenses, different contents.

Proposed direction (from discussion): backload the read path from the transcript — GET /messages serves from the durable transcript, newest-first, paginated (the route already takes limit/offset), so the room view survives restarts and both post paths converge.

Gotchas to design around:

• Shape mismatch. Transcript records are L9 envelopes (sender, content, kind/subkind, message_id, recorded_at); local_state StoredMessage carries message_type, recipient_handle, and event semantics (status, ttl, kind, episode). Mapping transcript → MessageRead is lossy.
• Dedup across stores. StoredMessage.id and the envelope message_id are different id spaces, so merging the two stores needs a real correlation key, not just id equality.
• Event-ledger messages (ttl/status/PATCH-by-id) only exist in local_state today; a transcript-backed read path has to decide where those live.

Suggested framing

1 and 2 are the same fix (append-only JSONL, optionally segmented) and are low-risk. 3 is the bigger design call (unify the read path / reconcile the two stores) and is worth its own pass.