#23 Add durable peer store and forward#85
Draft
lucas-barake wants to merge 26 commits into
Draft
Conversation
Working-tree snapshot pushed on behalf of the issue-23 session so the work is not stranded locally. Adds relay store error typing and observability coverage (`peerRelayStoreErrors.ts`, `PeerRelayStoreErrors.test.ts`, `PeerRpcObservability.test.ts`), and broadens the relay, outbox, sync-envelope and documentation surface across `local-rpc` and `local-sql`. Known incomplete: `packages/local-rpc/test/PeerRelayStore.test.ts` has two `Metric.MetricRegistry` type errors (value used as a type) that still need fixing.
lucas-barake
force-pushed
the
ship/issue-23-store-and-forward
branch
from
July 26, 2026 17:07
4713d67 to
6aefe95
Compare
lucas-barake
force-pushed
the
ship/issue-23-store-and-forward
branch
from
July 26, 2026 17:16
627b674 to
203881d
Compare
The repository rules duplicated the global agent rules by naming AGENTS.local.md as a second task ledger, so the two could drift and an agent reading one would miss the other. Task tracking now lives only in the worktree LEDGER.md the global rules already mandate. Also close a loophole in the service rules. "Capture every service used later by its methods or expose that service as a Layer requirement" let an implementation push its dependencies into every method's requirement channel, where they spread to every transitive caller and let a method run under a context the Layer never validated. Dependencies are now resolved once at construction, and leaking a requirement needs a stated per-call reason.
The existing relay keeps endpoint ownership, the claim ledger, delivery wakeups and admission counters in process local state, so only one server process can own a custody shard. Modelling each recipient device as a cluster entity moves all of that to sharding, which elects exactly one owner per key across every runner. Being the sole writer for its device is what makes the store contract small. Claim tokens, lease deadlines, session generations and work ownership exist only to arbitrate between competing processes, and there are none. There is also no in-flight state: a message stays Pending until terminal and what is being delivered lives only in memory, so losing a runner loses that memory and nothing else and the next owner redelivers. At-least-once falls out of the schema instead of a recovery protocol. Delivery is stop-and-wait per sender channel and concurrent across channels, so a stalled sender cannot hold up other senders to the same device, and the connection epoch is part of the channel identity because the sender's sequence restarts at zero on every reconnect. Two details are load bearing and easy to lose. Subscribe is forked so it does not hold the entity's single handler permit for the life of a session, which would leave the settlement that ends each delivery unable to run and stall the inbox with no error anywhere. And a session's liveness is bounded by a server side deadline rather than by a disconnect notification, because the cross node interrupt is best effort and is never sent at all when the front door's own node fails.
Backs the inbox contract with a single table over a generic SqlClient, so the operator chooses the database. Supports SQLite, PostgreSQL and MySQL, matching the coverage of the store this replaces. Every guard is a predicate inside the write it protects. The owning entity is the sole writer for an inbox, but its rpc lane and its forked dispatcher fibers issue statements concurrently, so a read followed by a separate write is still a race within one process. Ordering is indexed on a channel digest rather than the six raw channel columns, which would exceed MySQL's key length limit once the subject is sized for real input. The index is unique because head selection picks the lowest sequence in a channel, and two rows sharing one would make that choice arbitrary and could starve either indefinitely. Replaying an identity the inbox already abandoned revives it rather than reporting a duplicate. A duplicate is a success, and the sender deletes its outbox entry on success, so reporting one for a dead lettered or expired row would make the sender's own recovery destroy the last durable copy of the message. Deduplication horizons only ever grow, since shrinking one reopens a window a sender may still be replaying into. Integers are decoded through a schema that accepts strings because PostgreSQL returns BIGINT, COUNT and SUM that way and installs no default parser. The MySQL index path checks information_schema instead of ignoring errors: that dialect commits DDL implicitly and the migrator writes its completion marker before running the body, so a swallowed failure would leave a half built schema that later runs treat as done.
Terminates the client socket, authenticates and authorizes each request, and forwards it to the cluster entity that owns the addressed device. The wire contract is unchanged, so this is a swap of the server's internals rather than a new protocol. Which inbox a request reaches is derived per operation and never taken from the request body. Open, Acknowledge and Reject address the caller's own inbox, computed from the authenticated principal, so settling another device's message is not expressible. Push addresses the counterparty the handshake authorized, so a session can only ever send where it was allowed to. Handshake state is per connection. A socket lives on one node and dies with it, so a client that loses this node reconnects and opens again. What could not be process local is the recipient's delivery session, because a message accepted on any node has to reach whichever node holds that recipient's socket, and that is exactly what the entity provides. The sender's message hash is recomputed rather than believed, since it is the identity the recipient deduplicates on. Every entity call is bounded by a timeout: sharding retries an unassigned entity or an unavailable runner indefinitely, so an unbounded call would hang through a rebalance and leak a fiber per in-flight request. A live session is kept alive by a heartbeat driven from here, because what must be proven alive is this node and its socket rather than the peer at the other end.
Two guards from the previous server were missing, and both let an unauthorized peer keep receiving. Receive authorization only ran at the handshake and its result was then discarded, so the delivery stream was piped unfiltered. Anything holding a Send grant to a device can write into that device's inbox, and the entity has no concept of grants, so a recipient received documents it never requested and was never authorized to receive. Every delivery is now re-authorized, and a message that fails is withheld rather than emitted, staying durable for a later session instead of erroring the stream. A session also outlived the credential and grants that created it. All three expose an invalidated signal and a deadline, none of which the client observes, so the relay is the only place that can act on them. Nothing did, and the relay's own heartbeat kept the session alive for as long as the socket stayed open. A session now ends as soon as any of its authority is withdrawn or lapses. Also: an Open beyond the per subject session cap is refused, since sessions were the front door's only unbounded resource and an authenticated client can open them in a loop; an envelope that decodes but cannot be canonicalized is reported as permanently invalid rather than as unavailability, which had the sender's outbox retrying it forever at the cost of a canonical encode each time; and a heartbeat interval that cannot outrun the entity's session deadline is refused at construction, because that combination silently reaps every session.
Nineteen cases over the production store and its real migrations, so the properties the design depends on are asserted rather than argued. Most of these exist because a review found the corresponding defect by running the store rather than reading it. The cross inbox sweep is the clearest: the key is (inbox, id), the same relay message id can legally address two devices, and a sweep filtered on the id alone destroyed a message in an inbox that had most of its lifetime left. Writing them found one more. The delivery budget off by one had been reported and I believed I had fixed it, but the edit never applied and nothing failed; a budget of one dead lettered the message on its first delivery, wasting the only attempt the recipient could have settled. The test is what noticed, which is the argument for having it. Also covered: reviving a message the inbox gave up on rather than reporting a duplicate the sender would treat as permission to drop its last copy; charging that revive against the quota; keeping its identity deduplicated across the sender's replay window; refusing a message whose channel disagrees with its envelope; and reaching every channel regardless of its own sequence numbering.
`pnpm check` has been failing on the branch since the suite landed, so the `check` CI job was red while the test job passed. `channel` declared its options as `readonly epoch?: string`, but every call site forwards `options.epoch`, which is `string | undefined`. Under `exactOptionalPropertyTypes` an optional property and an explicitly-undefined one are different types, so passing the latter is an error. Widen the parameter to accept `undefined` rather than making the call sites strip it, because forwarding an absent field is exactly what these helpers are for.
Adds test/RelayInbox.test.ts against the explicit in-memory cluster and the real SQL store over SQLite. Two harness notes worth keeping: TestRunner.layer bakes ShardingConfig defaults that stall every test under virtual time, and shard acquisition runs on scheduled fibers, so the clock has to be advanced once before any entity is reachable. The suite exposed three defects, each with a regression test that fails for its own reason before the corresponding change. deliverHead built the finalizer that fails a pending settlement as an eagerly evaluated argument to Effect.ensuring. It ran when deliverHead was called, before the body registered the settlement, so the lookup always missed and the failure went to a throwaway Deferred. A recipient's Settle then waited forever whenever the durable write did not land, and because Settle runs on the entity's serialized lane that hung request blocked every later Deliver and Settle for the device. Suspended so the map is read at finalization time. channelId omitted senderConnectionEpoch while ChannelKey treats it as identity. The store partitions heads by the full key, so two epochs of one sender are two heads at once, but the coarser in-memory key marked both busy from a single delivery. One unsettled head in a stale epoch held the live epoch's entire stream for the life of the session, which is the reconnect path. closeSession interrupted the Deferred that Settle awaits. Settle declares a closed error union, and an interrupt is not in it: the front door cannot catch it, so releasing a session took down the peer's acknowledgement fiber instead of returning a retryable failure. It now fails with ServerUnavailable, while the deferred only the delivering fiber awaits is still interrupted.
Both RpcTest.makeClient and Entity.makeTestClient hand the handler an already decoded value, so every other relay suite exercises the schemas without ever serializing them. That is the one thing this PR changed most: the bespoke length-prefixed framing is gone and the relay now speaks standard Effect RPC. This drives a full Open/Push/Acknowledge round trip between two clients and a real RpcServer over a TCP socket with RpcSerialization.layerJson, and asserts the payload survives byte identically in both directions. The payload carries high bytes on purpose, since a Uint8Array is the part of the contract a JSON codec is most likely to mangle. The no-redelivery check pushes a second message and requires it to arrive first, rather than sleeping and asserting nothing showed up: a redelivery of the acknowledged message would be the older head and would necessarily arrive ahead of it.
expire and collect are global sweeps that take no inbox key, and the entity owning an inbox is passivated as soon as it goes idle, so no entity can host them: the inboxes that most need expiring are exactly the ones with nobody connected. Nothing called either operation, which made messageTtlMillis and terminalRetentionMillis inert - undelivered messages were never expired, terminal rows were never collected, and the retained row count would climb until admission was refused for good. A cluster singleton gives it one owner at a time that moves with the shard rather than being pinned to a process. Interval, batch limit, retention window and enabled are all required: enabling retention silently would have every deployment delete durable rows on a schedule it never chose, and defaulting it off would leave TTL quietly dead. A sweep failure is caught and logged rather than allowed to escape. Sharding converts a failed singleton into a defect, which would stop retention for the whole deployment until a rebalance happened to move ownership. index.ts is regenerated by pnpm codegen; it had not been run since the relay modules landed, so five of them were missing from the package entry point.
RULES.md requires spans at meaningful workflow, I/O and remote boundaries, and the new relay modules had none, so a stalled inbox or a slow sweep was invisible. Every store operation and every entity handler now carries a stable span name. The store is spanned where the service is constructed rather than inside each body, so the names sit together and the database round trips are all visible at one boundary. Attributes are limited to identifiers: inbox_key is a digest, and relay message ids and outcomes are opaque. The envelope and payload are never recorded. Subscribe spans the installation of the session rather than its lifetime, since the handler returns as soon as the session is reachable and the stream long outlives it.
Two guarantees the old server made were missing from the rebuild. Neither was untested by accident; both were simply gone. Settling was not authorized at all. The handler resolved the session and forwarded to the entity, so a peer whose Receive authority had been withdrawn could still perform a durable terminal mutation. The session monitor does end a session whose grants lapse, but asynchronously, which leaves a window and reports SessionUnavailable where the contract calls for AccessDenied. Receive is now re-checked on every settle. The unbounded-decode risk acknowledgement was never requested. The old server required it before admitting a Push and before a delivery, on its own port that denies by default, because relaying a document commits its recipient to a decode that is not allocation bounded on the Automerge version this protocol targets. The new front door inherited that risk from the ordinary Send grant instead, which is not the same decision. It is now required on Push, on each delivery, and on settle. Also spans the four front-door RPCs, and makes retention part of RelayServer.layer rather than a separate layer a deployment can forget: TTL and collection are inert without it, and the failure mode is a table that grows until admission stops succeeding. The digest assertion now recomputes the outer envelope digest independently and compares it, rather than checking it looks like a hex string - a digest over the wrong inputs matches that pattern just as well as a correct one.
A broken authorization backend must not be reported as AccessDenied: that tells a legitimate client it is unauthorized and makes it stop retrying, when the truth is the relay cannot answer. The wire contract encodes a defect as InternalError to keep the two distinct, so the test asserts the cause carries a defect and no typed failure at all. Credential invalidation ends the session by closing the stream rather than erroring it. Nothing but the relay observes the credential, so it is the only thing that can end a session when authority is withdrawn, and an orderly close lets the client reconnect on its own terms. The delivered-but-unsettled message stays durable across the revocation.
Removes PeerRpcServer, PeerRelayIngress, PeerRelayStore, SqlPeerRelayStore and their migration and transaction helpers, along with the tests and bench that exercised them. The packages are unpublished, so the cluster relay replaces them outright rather than living alongside. All 25 client-visible contract guarantees the deleted tests encoded were audited first. Twenty are re-expressed against the new stack; five are waived because their subject no longer exists - process-local runtime shutdown is now cluster rebalance and passivation, there is no claim or payload-load split to race, and the front door has no admission queue to drain across a revocation, so exactly-once there is structural rather than a drain guarantee. The store contract moves into its own module and now runs against SQLite, PostgreSQL and MySQL. Multi-dialect parity was a decision, and a single dialect cannot establish this contract: PostgreSQL returns BIGINT, COUNT and SUM as strings, which made the store non-functional there while SQLite stayed green, and MySQL compares identity columns case-insensitively without a binary collation. Both were real defects in this store. Each check owns a distinct inbox key and none asserts a sweep's returned count, because expire and collect are deployment-wide and the dialect containers are shared across the block. Asserting the effect on the check's own inbox is the real guarantee anyway.
Re-authorizing every delivery introduced a stall. The front door takes a message off the entity's rendezvous queue and only then decides the recipient may not have it. By that point the delivering fiber has returned from the offer, charged the delivery, and is parked waiting for a settlement that can never arrive, because the client never saw the message. It holds its channel the whole time. The damage is not limited to that channel. Enough withheld heads occupy every one of the session's delivery slots, at which point channels the recipient is fully entitled to are never delivered at all - a permanent stall bounded only by the session ending. A reviewer reproduced both halves against the real composition. The seam was the problem: Subscribe and Settle gave the front door no way to say "not deliverable to this session, release it". Release does that. It ends the attempt without settling, so the row stays Pending for a later session, frees the channel, and records the channel as withheld for this session only - skipping the whole channel rather than the message, because passing over a head would break the ordering the channel exists to preserve. A reconnect starts clean, so a grant that comes back is picked up. The dispatcher also had to widen its poll by the number of withheld channels. Heads come back oldest first, so once enough withheld ones accumulate the query returned nothing else and the channels behind them were never looked at. Both halves are load bearing: removing either fails the regression test. The previous withhold test could not have caught any of this. It asserted an absence one scheduler pass too early, so it passed whether the message was withheld, delivered, or failed the stream. It now waits on a real rendezvous. The risk-grant knob was also tied to the ordinary authorization knob, which hid whether either port was enforced on its own; they are now independent.
Mutation testing found three guards that could be deleted outright with the whole suite still green. All three are the kind that only ever matter when someone is doing something they should not. sessionFor's ownership check was one of them. A session id is unguessable, but unguessable is not authenticated, and nothing else in the handler re-derives the caller from the session - so a peer holding a perfectly valid credential of its own could push and settle on another peer's session. No test had ever presented one principal's session id under another principal's credential. The tenant check was another: every principal in the harness lived in the same tenant, so refusing a peer authenticated into a tenant this relay does not serve was never exercised. The third was the claimed-local-identity comparison. The existing case differed in two fields at once, so the clauses masked each other and any one of them could be removed unnoticed - including the peerId clause, which is what stops a credential being reused for a sibling device of the same subject. Each field is now checked on its own. Each test was verified by neutralizing the guard it covers and confirming it fails.
A test-reviewer ran 69 mutations against this suite and found that the handshake's two grants, the recheck that runs after authorization returns, the per-message re-authorization, the unbounded-decode risk port, the heartbeat, the entity-call timeout and the per-subject cap could all be deleted or weakened without turning it red. Authorization that nothing pins is authorization that will be refactored away. Each gap is now closed by a test that denies exactly one thing at exactly one call site, and every one was verified by applying the mutation to a copy of the repo and watching it fail. Reaching the post-authorization rechecks needed a new seam: the authorization port already refuses a grant that was expired when it answered, so the only way the relay's own recheck can matter is if time passes while authorization is in flight, which is what the `onAuthorize` knob simulates. The risk-port delivery test survived its own mutation on the first attempt because "assert the queue is empty" was racing the dispatcher rather than observing a withheld message. It now requires a younger, authorized message on another channel to arrive first, which is the only shape in this suite that can distinguish withheld from not-yet-dispatched. Four tests are deleted. Three were decided entirely by the authentication middleware or by store-level deduplication, and one asserted a rejection outcome that no store operation can read back. The protocol-version pin moves to PeerRpc.test.ts, where the schema it exercises lives.
The per-channel dispatcher runs on a forked fiber that nothing observes, so a defect in it made the fiber quietly disappear. Everything else went on looking healthy: the session stayed installed, the front door kept heartbeating it, Deliver kept reporting success to senders, and the device received nothing at all until it happened to reconnect. A forked fiber's defect reaches neither RpcServer's defect path nor the entity's own defect restart, so nothing in the framework noticed either. The dispatcher now hands its cause to the one thing the recipient is watching. Failing the queue with the raw defect does not work - Subscribe declares a closed error union, so a Die cause has nowhere to go across the entity RPC boundary and the recipient still waited forever. The defect goes to the log at fatal with its whole cause, and the recipient gets the one thing it can act on: a retryable ServerUnavailable, after which the front door's session finalizer ends the session and the client opens a fresh one. Three entity calls that ran without a deadline are now bounded. The session finalizer's EndSession is the serious one: finalizers run uninterruptibly and Sharding retries an unassigned entity forever, so during a rebalance that call never returned, the request fiber never exited, and the node hung on shutdown over a connection whose client was already gone. The entity finalizer's comment claimed it runs before the framework's termination handshake. It does not, and it cannot: the build effect only ever sees the ResourceRef's inner scope, while the handshake is registered on the outer entity scope afterwards, and scope finalizers are last-registered-first. The comment now states the real ordering and what it costs on a rebalance.
PeerRelayLimits carried the tuning surface of a single-process relay that owned its own socket framing, claim leases, SQL lanes and cross-scope quota counters. None of that machinery survived the move to cluster entities, so 87 of its 97 values had no reader at all - 79 of them referenced only by cross-field validations that validated each other. Dead configuration is worse than absent. An operator who sets maxActiveMessagesPerTenant believes their deployment is quota bounded; nothing reads it, so it is not. The ten values that are actually read stay, along with the two relations that still mean something: the receipt retention nesting rule, which the front door's Open validation mirrors and therefore has to agree with, and the authentication burst floor. The test now pins the exact surviving key set rather than a count, so a value cannot come back without a reader. Two families are gone with no replacement, and both are accepted regressions rather than oversights. Per-scope quotas span many inboxes, so no entity is their sole writer and enforcing them would reintroduce the cross-process arbitration this design removes; admission is now bounded per inbox, inside the transaction that performs the write. Connection and frame accounting belonged to the length-prefixed framing that standard RPC over a socket replaces; a single payload is still bounded by PeerRpc's schema check, but concurrent connections are now the deployment's socket server to bound. The observability module loses nineteen exports for the same reason. Beyond the session and queue metrics, the whole setRelay family and the six gauges behind it had no production writer either - only their own test drove them, which is a test keeping dead code alive. A gauge that is always zero reads as "no backlog" rather than as "nobody is measuring". internal/peerRpcProtocol.ts stays. Its two constants look inlinable, but PeerRelayLimits reading them from PeerRpc would close a cycle through PeerAuthentication. The leaf module is what breaks it.
A configured duration named somethingMillis puts the unit in the identifier instead of the type, and makes every deployment do its own arithmetic to say "seven days". Every configuration surface this branch owns now takes Duration.Input, so a deployment can write Duration.days(7), "7 days", or a raw number, and each is converted once where the Layer is built rather than on every request. Two things deliberately keep their unit in the name, and RULES.md now says why: values that cross the wire contract, and values that feed a millisecond column. Duration has no stable serialized encoding, so a protocol or a column has to name its unit. The first attempt at this renamed the store's operation arguments along with the configuration and the typechecker caught it - when the boundary between configuration and protocol is a field name, a regex is the wrong tool. The two duration schemas are Schema.declare guards over Duration.fromInput, which is safe on arbitrary input, so InvalidPeerRelayLimits still names the offending field.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #23
This PR makes the durable store and forward relay the only RPC topology. Since the packages are not published yet, the durable protocol becomes version 1. The previous direct protocol variants and their version history are removed.
Architecture
Communication uses the typed Effect RPC protocol over Effect's generic
SocketandSocketServerabstractions. The package defines one streamingOpenRPC plusPush,Acknowledge, andReject. It does not own an HTTP server, WebSocket upgrade route, or TLS configuration. Applications supply the socket transport and authentication boundary.The backend relay depends on the semantic
PeerRelayStoreservice.SqlPeerRelayStoreimplements that service against Effect's genericSqlClient, following the structure and naming of Effect Workflow's SQL storage. The implementation supports SQLite, PostgreSQL, and MySQL through the matching Effect SQL driver. Applications may provide anotherPeerRelayStoreimplementation when they need a different persistence system.Database portability is separate from relay server clustering. PostgreSQL and MySQL allow operators to choose a backend with different durability and operational properties. They do not make
PeerRpcServerhorizontally scalable.The frontend remains SQLite backed. It owns the sender outbox, recipient receipts, replica state, and the transactional boundary that applies a delivered message before acknowledging it.
Custody And Replay
The sender writes every outgoing envelope to its local SQLite outbox before attempting network delivery. Each entry has a stable relay message identity and digest. On connection,
RpcPeerTransportpushes all due outbox entries before accepting new sends. The frontend removes an entry only after the relay confirms durable custody and the returned identity matches.Relay admission is transactional. It reserves the applicable quota scopes, stores the payload, assigns relay owned channel ordering, and records a stable deduplication identity. Repeating the same identity and digest returns duplicate success. Reusing the identity with a conflicting digest is rejected.
Opencreates a streaming relay session. When an endpoint connects, the server replaces any previous session for that endpoint and schedules pending work. Delivery workers select the oldest eligible channel head, claim it with a random token, session generation, and lease deadline, then stream the stored message through a bounded queue.The recipient applies or deduplicates the envelope and records a relay receipt inside the same SQLite transaction. Only after that transaction commits does it call
Acknowledge. Acknowledgement or permanent rejection releases active payload custody while retaining terminal identity until the deduplication horizon expires.Concurrency And Recovery
Live concurrency uses Effect scopes, fibers, bounded queues, semaphores, deferred values, interruption, and finalizers. Database concurrency uses transactions, channel sequences, claim tokens, session generations, and claim deadlines.
A stale session in the same server process cannot acknowledge a claim owned by its replacement. Disconnect cleanup releases known claims with retry backoff. If the process crashes first, maintenance recovers expired leases and makes those messages eligible again.
SQLite relay writes use the required durability settings and explicit write transactions. PostgreSQL and MySQL serialize competing relay mutations through locked database state. MySQL migrations use a pinned advisory lock and schema verification because its DDL commits implicitly.
Deployment Boundary
The current
PeerRpcServersupports one active server process per custody shard. Its session registry, endpoint replacement, subject admission counters, work ownership, delivery queues, and wakeups are process local.Running multiple
PeerRpcServerinstances against the same SQL database is not a supported active active deployment. SQL transactions can prevent some conflicting row transitions, but there is no distributed endpoint ownership, cross node session revocation, cross node work notification, global admission accounting, coordinated maintenance ownership, or cluster membership protocol.Current scaling is explicit sharding. Each shard has one active
PeerRpcServerowner and its own custody boundary. Applications route a tenant or peer deterministically to that shard. The database behind a shard may use its own replication and failover, but server failover and single owner enforcement remain deployment responsibilities.True horizontal relay scaling requires a separate cluster design. That design must add globally fenced endpoint ownership, a distributed session directory, cross node wakeups, coordinated maintenance and limits, and routing or forwarding to the node that owns the live recipient connection.
Delivery Contract
The guarantee is durable custody with at least once delivery and durable recipient deduplication inside one custody shard.
A failed or ambiguous
Pushleaves the sender outbox entry available for retry. A recipient crash after its local commit but beforeAcknowledgecan cause redelivery. The persisted receipt prevents the application message from being applied twice, after which the repeated delivery can be acknowledged safely.Exactly once external effects, automatic server failover, active active relay operation, quorum replication, and distributed consensus are not claimed.
The shared relay storage contract is exercised against real SQLite, PostgreSQL, and MySQL databases. The production Effect RPC composition is covered through custody transfer, acknowledgement, interruption, reconnect replacement, replay, and legacy protocol rejection.