Metadata-missing dies inside persisted cluster entity handlers still escape as defects

[lucas-barake]

Split out of #14, which typed the storage-format-version mismatch and ReplicaBootstrap's own metadata-missing read. Five metadata-singleton-missing Effect.die sites were deliberately left as defects there, because the obvious conversion is worse than the bug:

• packages/local-sql/src/DocumentStore.ts:95 (nextSequence) — reachable: build the production ReplicaBootstrap → ReplicaGate → Recovery → DocumentStore stack, DELETE FROM effect_local_metadata WHERE singleton = 1, then store.create(...).
• packages/local-sql/src/ReplicaGate.ts:125 (readState) — reachable via validate / claim. Note refresh has zero production callers, so a test against refresh proves nothing.
• packages/local-sql/src/DocumentStore.ts:109, packages/local-sql/src/PeerSync.ts:742, :748 — unreachable, each dominated by an earlier read of the same singleton row in the same transaction.

Why converting them naively is wrong

All five are reachable inside DocumentEntity's Create/Mutate/Delete/ApplySync handlers, which are annotated ClusterSchema.Persisted, true with a primaryKey (packages/local-sql/src/DocumentEntity.ts:54-119), over the real SqlMessageStorage (packages/local-sql/src/internal/clusterStorage.ts:14). In effect@4.0.0-beta.99:

• a defect goes RpcServer.ts:298-303 → entityManager.ts:280-282 case "Defect" → onDefect, which restarts the entity and redelivers the in-flight requests, persisting no reply;
• a typed failure is responded as Reply.WithExit and persisted, and a later request with the same primary key is answered straight from the stored exit by Runners.ts:204-212 — the entity is never re-invoked.

So converting these turns "retry until the storage is repaired" into "durably record a terminal StorageCorrupt and replay it for that primary key forever". For ApplySync that is permanent, silent peer divergence; for the command RPCs it poisons the caller's commandId idempotency key.

Independently, packages/local-sql/src/ReplicaEvolution.ts:123-136 is the repo's only Effect.catchReason("ReplicaError", "StorageCorrupt", ...) handler and treats StorageCorrupt as a per-document recoverable condition — it logs a warning and marks that one document projection_status = 'Blocked'. A replica-wide metadata failure routed through it would quarantine a healthy document. Today only the incidental fact that gate.validate fails in the same transaction stops that write from committing, which is an accident rather than a design.

What this issue needs

A decision, not a mechanical conversion:

1. whether metadata-missing should be typed at all inside a persisted entity handler, or stay a defect so the entity restarts;
2. if typed, whether it needs a reason distinct from StorageCorrupt so ReplicaEvolution.ts:125 cannot absorb a replica-wide condition as a per-document one;
3. either way, a test that drives DocumentEntity.Mutate / ApplySync with the metadata singleton deleted and asserts what does or does not land in the cluster reply table.

Separately noticed

packages/local-sql/src/ReplicaBootstrap.ts's pre-migration populated probe covers only the ten migration-1 tables; the peer tables are excluded because migration 2 creates them. A replica populated only with peer rows and no metadata singleton is therefore migrated before being rejected. Reachability requires foreign keys to be off, which is the case for @effect/sql-sqlite-wasm in the browser but not for @effect/sql-sqlite-node. This predates #14 (before it, migrations always ran first regardless) and is worth folding into whatever fix this issue lands.