Skip to content

feat(server): add LibsqlKv, the DenoKvLike coordination store over libSQL - #430

Merged
davidwkeith merged 2 commits into
mainfrom
claude/horizontal-worker-scaling-y00cc6
Jul 24, 2026
Merged

feat(server): add LibsqlKv, the DenoKvLike coordination store over libSQL#430
davidwkeith merged 2 commits into
mainfrom
claude/horizontal-worker-scaling-y00cc6

Conversation

@davidwkeith

Copy link
Copy Markdown
Owner

Summary

Implements scale-out phase 1 (#428, spec/scale-out.md §8): LibsqlKv, a standalone @dwk/server module presenting @dwk/deno-host's DenoKvLike seam over one dedicated libSQL database — the coordination store the proposed central storage mode's lease/alarm/queue machinery will run on. Nothing composes it into the host yet; no behavior change for existing users.

Design highlights (per the spec, with two refinements now noted in an update callout in §8):

  • Order-preserving tuple key encoding (encodeKvKey/decodeKvKey, exported): element-tagged FDB-style encoding with escaped terminators and sign-flipped IEEE-754 numerics, so BLOB-memcmp primary-key order equals key-array order — numeric due-index parts sort numerically, prefixes match per element (["a"] never matches ["ab"]), and list's prefix/start/end (exclusive) ranges are plain indexed SQL range scans.
  • CAS with a store-wide monotonic versionstamp (kv_meta.seq) rather than the spec sketch's per-key counter — a per-key counter could reissue a stamp after a sweep deletes and a later write recreates the key, letting a stale lease release wrongly succeed; store-wide can't. All sets in one atomic commit share one stamp (equality-only CAS makes both refinements invisible to the seam's consumers; documented in the module doc).
  • Checks evaluated before mutations: each atomic().check(...).set/delete(...).commit() becomes one batch(..., "write") transaction that stores the check verdict in a scratch column first and guards every mutation on it — pre-mutation-state semantics exactly matching the seam's reference implementation.
  • Lazy TTL: expireInexpires_at; expired rows are immediately invisible to get/list/checks and physically removed by sweepExpired(), which the future host poll ticks will call.
  • Injected LibsqlClientLike — the module never constructs a connection or reads the environment, per the composition contract.

Testing (spec §14 item 1, plus the first slice of item 2's multi-replica posture): codec round-trips and rejection cases; an ordering property test comparing list scan order against a model comparator across all five key-part types; CAS interleavings (absence checks, stale stamps, racing claims, all-or-nothing multi-op); expiry (invisible-before-sweep, stamp-never-reissued, physical sweep); and the real @dwk/deno-host acquireLease/releaseLease, setAlarm/getAlarm/deleteAlarm, and QueueBroker code driven against LibsqlKv — including release-after-expiry keeping the new holder, single-slot due-index replacement, ack/backoff/maxAttempts semantics, and two brokers sharing the store delivering a message exactly once. Unit tests run against a node:sqlite-backed LibsqlClientLike fake (same posture as @dwk/deno-host's unpublished harness); no live service required.

Also: @dwk/deno-host added to @dwk/server's dependencies (type-only in production code today; the runtime machinery arrives with the central mode), exports added to src/index.ts, the §8 implementation-update callout in spec/scale-out.md, and the server CLAUDE.md file-layout/deps notes.

Closes #428.

Packages affected

@dwk/server (private, never published; @dwk/deno-host added as a dependency, its code unchanged)

Checklist

  • Read the relevant spec(s) under spec/packages/ and updated them if
    behaviour changed
  • Added/updated colocated tests (src/*.test.ts)
  • Ran the local CI gate: pnpm lint && pnpm format:check && pnpm typecheck && pnpm build && pnpm test
  • Added a changeset (pnpm changeset) — not applicable, @dwk/server is private and never published
  • Updated catalog.json / conformance/status.json — not applicable, no new worker

Generated by Claude Code

…bSQL

Implements scale-out phase 1 (#428, spec/scale-out.md §8): a standalone
module presenting @dwk/deno-host's DenoKvLike seam over one dedicated
libSQL database, so the lease/alarm/queue machinery can later span
replicas in the proposed central storage mode. Order-preserving tuple
key encoding (numeric parts sort numerically under BLOB memcmp), CAS
via a store-wide monotonic versionstamp (never reissued after a sweep,
unlike per-key counters), checks evaluated into a scratch row before
any mutation inside one batch("write") transaction, lazy TTL with an
explicit sweepExpired for poll ticks.

Colocated tests cover codec round-trips and ordering properties, CAS
interleavings, expiry semantics, and drive the real @dwk/deno-host
acquireLease/setAlarm/QueueBroker code against LibsqlKv, including the
two-replica claim race. Nothing composes it into the host yet — no
behavior change for existing users.

Closes #428

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgeyeFnkNceEwUxBKgCTrC

@davidwkeith davidwkeith left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

LibsqlKv: @dwk/deno-host's DenoKvLike seam implemented over one libSQL table, for the future scale-out central coordination store (#428). Order-preserving tuple key codec (encodeKvKey/decodeKvKey), store-wide monotonic versionstamp, and pre-mutation-state CAS via a scratch-column guard evaluated before any mutation in one batch(..., "write") transaction. Standalone — nothing composes it into the host yet.

Blocking: CI is red (build-test job fails)

This is the one that matters most — the PR's own checklist claims the local CI gate ("pnpm lint && pnpm format:check && pnpm typecheck && pnpm build && pnpm test") ran clean, but the actual build-test GitHub Actions job fails at the typecheck step:

packages/server typecheck: src/libsql-kv.ts(49,8): error TS2307: Cannot find module '@dwk/deno-host' or its corresponding type declarations.
packages/server typecheck: src/libsql-kv.test.ts(18,8): error TS2307: Cannot find module '@dwk/deno-host' or its corresponding type declarations.
...5 more implicit-any errors cascading from that unresolved import...
packages/server typecheck: src/libsql-kv.ts(258,5): error TS2322: Type 'Promise<void> | undefined' is not assignable to type 'Promise<void>'.

Two distinct, real bugs:

  1. packages/server/tsconfig.json's paths map was never updated for the new @dwk/deno-host dependency. That file (not touched by this PR at all, per the diff) exists specifically so pnpm typecheck works before pnpm build runs (see its own header comment) — every other workspace dependency @dwk/server has is listed there, this one is the sole omission:
    "@dwk/deno-host": ["../deno-host/src/index.ts"],
    (alphabetically between @dwk/cf-shims and @dwk/dpop). This is almost certainly why the local run looked clean: if @dwk/deno-host's dist/ already existed on disk locally (e.g. from a prior pnpm build), module resolution would fall through to the built output and mask the missing source-path mapping — but CI's typecheck step runs before build, so it hits the gap. Couldn't leave this inline since the file isn't part of this diff.
  2. libsql-kv.ts:258 — a real, separate type error in #ready(). Left inline; it's the classic TS limitation where this.#schema ??= someCall() doesn't narrow the field for the very next return this.#schema, because the RHS contains a call.

Both need fixing before this can merge — CONTRIBUTING.md's CI gate (lint → format → typecheck → build → test) is a hard requirement, not just a local nice-to-have.

Code quality / correctness (reviewed independent of the CI failure)

The design itself looks sound once it compiles:

  • Key codec (encodeKvKey/decodeKvKey): FDB-style element-tagged encoding with proper 0x000x00 0xFF escaping, correct IEEE-754 sign-flip transform for total numeric order (verified the forward/reverse bit logic by hand), and the prefix ‖ 0xFF upper bound for list() range scans is a valid strict bound since every tag byte is ≤ 0x05. NaN and out-of-range bigints correctly rejected.
  • CAS commit path (_commit): checks are evaluated into the kv_meta.ok scratch column before any mutation statement runs in the same batch(..., "write") transaction, so a failed check genuinely makes every mutation a no-op — matches the documented pre-mutation-state semantics. The store-wide (not per-key) versionstamp is the right call for the reason given in the doc comment (a per-key counter really could let a stale CAS wrongly succeed after a delete+recreate).
  • All SQL is parameterized; the only string-built SQL fragments (guard, conditions.join(...)) are static clause shapes, not user data — no injection surface.
  • No issues found in sweepExpired, get, set, delete, or the LibsqlKvAtomic builder.

CONTRIBUTING.md conformance

  • ✅ PR title feat(server): add LibsqlKv, the DenoKvLike coordination store over libSQL — correct Conventional Commits scope/format.
  • ✅ PR body keeps Summary / Packages affected / Checklist headings verbatim.
  • ✅ Both unchecked checklist items (changeset, catalog/conformance) carry valid one-line reasons rather than being deleted — @dwk/server is genuinely private/unpublished and this adds no new worker.
  • ✅ Spec updated in the same PR: spec/scale-out.md §8 gets an "Update (issue #428): implemented" callout describing the two documented divergences from the sketch.
  • ✅ Colocated tests present and substantive (libsql-kv.test.ts): codec round-trip/rejection, an ordering property test across all five key-part types, CAS interleavings, expiry, and the real @dwk/deno-host lease/alarm/queue-broker code driven against LibsqlKv including a two-replica exactly-once delivery case.
  • CI gate is not actually green — see above. Checklist item is checked but the claim doesn't hold.

No other conformance issues found. Once the two typecheck errors are fixed, this looks like a solid, well-tested increment.


Generated by Claude Code

Comment thread packages/server/src/libsql-kv.ts Outdated
Adds the missing @dwk/deno-host paths entry to packages/server/
tsconfig.json — CI runs typecheck before build, so without the
source mapping the import only resolved when a stale local dist/
happened to exist. Also restructures LibsqlKv's #ready() to return
the ??= expression directly, sidestepping the field-narrowing
limitation the unresolved import surfaced as TS2322. Verified by
deleting deno-host's dist and re-running typecheck.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DgeyeFnkNceEwUxBKgCTrC

Copy link
Copy Markdown
Owner Author

Both typecheck failures are fixed in aa76d84: the missing @dwk/deno-host entry is added to packages/server/tsconfig.json's paths map (alphabetical, as suggested), and #ready() now returns the ??= expression directly (details in the resolved inline thread — it was a cascade of the unresolved import, but the restructured form is better regardless).

Your diagnosis of why the local gate looked clean was exactly right: @dwk/deno-host's dist/ existed locally from a prior build, so module resolution fell through to built output and masked the missing source mapping. To make sure the fix closes CI's actual failure mode rather than the masked one, I deleted packages/deno-host/dist and re-ran pnpm --filter @dwk/server typecheck clean before re-running the full gate (lint → format:check → typecheck → build → test, 2827 tests passing). CI is re-running against aa76d84 now.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(server): LibsqlKv — DenoKvLike coordination adapter over libSQL (scale-out phase 1)

2 participants