[Feature]: MeshClient delivery-on-reconnect (inbox replay)

[imran-siddique]

When a MeshClient reconnects after being offline, it sends a \connect\ frame but never requests pending messages. A sub-agent that was offline during a send won't receive the message on reconnect.

Fix: Add \etch_pending\ / \inbox_replay\ call in \MeshClient.connect()\ to retrieve queued messages from the relay's offline inbox.

Ref: Azure/kars#27 review (blocker #8)

[lawcontinue]

We hit something similar in distributed inference — a node reconnects after a GPU fault and has no idea what it missed. We added a heartbeat with a sequence counter, which tells you what got lost but not how to replay it.

One question on the fetch_pending design: is the replay order guaranteed to match the original send order? If a relay queues 50 messages while the client is offline, FIFO replay could diverge from causal order if any of those messages were responses to earlier ones in the same batch. Curious whether the intent is strict FIFO or something ordering-aware.

The sequence counter approach (detect gaps) and inbox replay (fill gaps) feel complementary rather than competing — you would want both for full reliability.

[imran-siddique]

Thanks for the question @lawcontinue. The current implementation (PR #1451) uses strict FIFO replay: the relay's offline inbox is iterated in queue order, and the client dispatches each message sequentially via handleMessage(). The test suite explicitly verifies ordering preservation (sequences 0-4 arrive in order).

You're right that FIFO and causal ordering can diverge when messages in the same batch are causally dependent. For now, strict FIFO matches the relay's store-and-forward semantics (messages are queued in arrival order). If we need causal ordering later, that would require adding vector clocks or causal dependency metadata to the wire protocol, which is a larger change.

And agreed: sequence counters (gap detection) and inbox replay (gap filling) are complementary. The sequence counter approach you described in #1368 would be a natural addition for detecting missed messages in real-time, while inbox replay handles the reconnect case.