feat(server): central-mode durable objects — leases, embedded replicas, sync-before-serve (scale-out phase 3)

[davidwkeith]

Phase 3 of the horizontal scale-out plan (spec/scale-out.md §15): bring the Tier 2 (actor) cohort — solid-pod, activitypub, remotestorage, webauthn, atproto-pds — up in central storage mode, with the Durable Object guarantees enforced across replicas. Builds on #428/#430 (LibsqlKv) and #431 (central mode + Tier 1); fleet pollers/cron/drain are phase 4.

What

Per spec/scale-out.md §6, adopting @dwk/deno-host's implemented per-request-lease model unchanged (v1):

1. DO namespace over the coordination store: compose @dwk/deno-host's createDurableObjectNamespace with LibsqlKv — per-event lease acquire/release (["dwk_lease", className, idHex], CAS + expireIn crash safety net), process-local per-id chain within a replica, KV lease across replicas. LeaseContendedError maps to 503 + Retry-After at the host.
2. getStorageClient(idHex) → per-object libSQL database via embedded replica: the libsql npm package's sync client against storage.libsql.replicaDir scratch files (a rebuildable cache; ephemeral disk suffices). This is the first hard dependency on the libsql native module loading on Node in this repo — verify early.
3. The sync-before-serve rule (§6.2 — load-bearing): after acquiring an id's lease and before the event runs, the embedded replica MUST sync from the primary — a different replica may have written since this one last held the id; skipping the sync serves stale reads and is a correctness bug. Note: createDurableObjectNamespace currently calls getStorageClient once per id and caches the instance, with no per-dispatch hook — this likely needs a small, contract-reviewed extension to @dwk/deno-host (e.g. an onLeaseAcquired/sync callback in DurableObjectNamespaceOptions), amended in the same PR per the host-contract §8 growth discipline's spirit.
4. Alarms: the KV-indexed due/by-id schedule works as-is over LibsqlKv (proven in feat(server): add LibsqlKv, the DenoKvLike coordination store over libSQL #430's tests); wire pollAlarms to a per-replica jittered interval timer (~1 s default — real timers, better than platform-cron granularity), claim-based so concurrent pollers are safe. Sweep LibsqlKv.sweepExpired from the same tick.
5. WebSockets (§6.4, v1 stance): sockets stay pinned to the terminating replica via the existing web-socket-upgrade.ts bridging; document the lease/socket window and the LB-affinity recommendation rather than architecting around it (v2 residency is the eventual fix — see the phase 6 issue).

Testing (spec §14 items 2–3)

Two DwkServer instances sharing one LibsqlKv + SQL backing + S3 fake:

• Racing same-pod writes from both replicas serialize — one wins the lease, the other queues or 503s; never interleaved state.
• Sync-before-serve regression: write a pod resource via replica A, immediately read via replica B, assert B sees it (the test that fails if the sync rule is skipped).
• An alarm scheduled via A fires exactly once though both replicas poll (activitypub Accept-delivery retry as the lifecycle).
• Kill replica A mid-request (simulate: never release) → B recovers the id after leaseTtlMs.
• Tier 2 lifecycles across replicas, modeled on the existing phase5-*.integration.test.ts suite.

Out of scope

• Queue pollers, cron tick lease, drain, readiness (phase 4).
• Residency leases / internal forwarding (phase 6, demand-gated).
• Live Turso/sqld verification (phase 5's checklist; the embedded-replica native-module smoke test above is the one early exception).

References

• spec/scale-out.md §6 (the whole design), §10 (consistency table), §14 (testing)
• spec/packages/deno-host.md — the feat(deno-host): single-writer actor + alarm emulation via a Deno KV lease #398 actor/alarm design being composed, including its documented WebSocket lease limitation
• feat(server): LibsqlKv — DenoKvLike coordination adapter over libSQL (scale-out phase 1) #428 / feat(server): add LibsqlKv, the DenoKvLike coordination store over libSQL #430 (phase 1, merged) · feat(server): central storage mode — Tier 1 bindings over centralized stores (scale-out phase 2) #431 (phase 2)

[davidwkeith]

This was implemented and merged in #441 (feat(deno-host,server): central-mode durable objects across replicas (scale-out phase 3)), which covers everything in the description above:

• @dwk/deno-host's createDurableObjectNamespace gained an opt-in onLeaseAcquired?(idHex, client) hook, called once per dispatch after the lease is acquired and before the event runs.
• @dwk/server's createCentralDurableObjectNamespace (central-durable-object.ts) uses that hook to call client.sync() unconditionally, making sync-before-serve structural.
• DurableObjectAlarmPoller (central-do-poller.ts) is the per-replica jittered interval timer central-mode alarms need.
• server.ts's dispatch path maps LeaseContendedError to 503 + Retry-After: 1.
• libsql-native.smoke.test.ts verifies the libsql native module loads under this repo's Node version.
• The Tier 2 endpoint packages (solid-pod, activitypub, remotestorage, webauthn, atproto-pds) needed no changes, since their DurableObject base class only wires ctx/env in its constructor.

PR #441's body used "closing #432" rather than a GitHub-recognized closing keyword, so this issue was left open despite the merge. Closing now — the remaining four Tier 2 packages' own cross-replica lifecycle suites and the fleet lifecycle items (queue poller, cron tick lease, drain, readiness) are tracked as phase 4 follow-ups per the PR description.

No code changes are needed on this branch (claude/issue-432-dbqx7n) — it is already even with main, which contains #441's merge.

---

Generated by Claude Code

---

Generated by Claude Code