feat(server): central-mode fleet lifecycle — queue pollers, cron tick lease, drain, readiness (scale-out phase 4)

[davidwkeith]

Phase 4 of the horizontal scale-out plan (spec/scale-out.md §15): make the replica fleet operable — queues consumed by every replica, cron firing once fleet-wide, orderly drain, and health surfaces. Builds on #431 (central mode) and #432 (DOs). This is the phase that completes Tier 1 in central mode: until it lands, the queue-dependent consumers (webmention/microsub/websub) can't run there.

What

Per spec/scale-out.md §7 and §12:

1. Queue pollers on every replica: @dwk/deno-host's createQueueBroker over LibsqlKv (already exercised in feat(server): add LibsqlKv, the DenoKvLike coordination store over libSQL #430's tests), with each replica running pollQueues on the jittered pollIntervalMs timer (shared with feat(server): central-mode durable objects — leases, embedded replicas, sync-before-serve (scale-out phase 3) #432's alarm tick). Claim-CAS makes concurrent pollers safe; adding replicas adds consumer throughput. Note the conformance point already recorded in spec §7.1: central mode uses the deno-host broker's conforming redeliver-by-default semantics, not @dwk/cf-shims' auto-acking broker — consumer registration in the host must bind to the right broker per mode.
2. Cron tick lease (§7.2): each replica ticks on the configured cadence and first attempts ["dwk_cron", handlerName, tickBucket] (atomic set with expireIn covering the cadence; tickBucket = scheduled time rounded to cadence). One winner runs the scheduled handler; losers skip silently. A crashed winner is covered by handler idempotency + the next tick (missed-tick coalescing is permitted by host-contract §3.7).
3. Drain-aware shutdown (§12, in order): stop accepting connections → stop alarm/queue/cron pollers (in-flight polls finish; claimed-but-unfinished messages redeliver by contract) → drain the WaitUntilTracker → close WebSockets with a going-away code → release held leases via normal finally paths → exit. Rolling deploys then need no choreography — old/new replicas coexisting is the normal, lease-safe state.
4. Health surfaces (§12): a liveness endpoint (process up) and a readiness endpoint re-running the §9.2 store probes cheaply (cached ~seconds), so orchestrators pull an unhealthy replica without killing in-flight work.
5. Observability (§12): @dwk/log events for lease acquire/contend/expire, poll-tick lag and claimed-batch sizes, replica sync duration, sweep counts, probe failures — per the observability.md taxonomy.

Testing (spec §14)

• Two replicas + one store set: a message sent on replica A is delivered exactly once on either (both pollers running); redelivery-until-acked across a replica "crash" mid-batch (claimed entries requeued by contract when the handler never decides).
• Cron: both replicas tick the same bucket; the handler runs once; the next bucket admits a (possibly different) single winner.
• Drain: shutdown mid-poll finishes in-flight work, unacked messages redeliver on the surviving replica, and no lease is left dangling past its TTL.
• Readiness flips on an induced store outage and recovers.

Out of scope

• docker-compose/k8s reference, dwk migrate, live verification, conformance runs (phase 5).
• Concrete thresholds for coordination-KV growth alarms — spec §16.5 leaves these until real traffic numbers exist; land the events here, tune later.

References

• spec/scale-out.md §7 (queues/cron), §12 (operations), §14 (testing)
• host-contract.md §3.6–3.7 — the queue/cron semantics being satisfied fleet-wide
• feat(server): LibsqlKv — DenoKvLike coordination adapter over libSQL (scale-out phase 1) #428 / feat(server): add LibsqlKv, the DenoKvLike coordination store over libSQL #430 (phase 1, merged) · feat(server): central storage mode — Tier 1 bindings over centralized stores (scale-out phase 2) #431 (phase 2) · feat(server): central-mode durable objects — leases, embedded replicas, sync-before-serve (scale-out phase 3) #432 (phase 3)

[davidwkeith]

This was already implemented and merged in #442, but the PR description used "closing #433" instead of a GitHub auto-close keyword (Closes/Fixes/Resolves), so the issue never auto-closed.

Verified on main:

• packages/server/src/central-fleet-poller.ts — queue pollers on every replica (§7.1)
• packages/server/src/central-cron.ts — cron tick lease (§7.2)
• packages/server/src/central-health.ts — liveness/readiness (§12)
• DwkServer.closeCentral in server.ts — drain-aware shutdown in spec order (§12)
• spec/scale-out.md §7/§12 already carry the "Update (issue feat(server): central-mode fleet lifecycle — queue pollers, cron tick lease, drain, readiness (scale-out phase 4) #433): implemented" notes

No new work needed here; closing as a duplicate of the already-merged #442.

---

Generated by Claude Code

---

Generated by Claude Code