fix(paddock): reload stale tabs, honour 429s, and resume streams instead of replaying

[jhgaylor]

Why

After #67 deployed (04:56Z) the account-wide list calls fell from 14.5/s to 1.1/s, but one browser tab still running the old bundle went into a hotter loop. In five minutes on Fountain: 3,277 stream opens on one conversation, 4,319 receipt reads on one sandbox, events?limit=1000 at ~7/s, and Fountain answering 429 rate_limited with Retry-After up to 35 s. Rates: stream 9/s, file 20/s, events 6.6/s (from 2/s, 4/s, 1.5/s before the deploy).

Mechanism (verified against the old bundle's code)

The old App.tsx keyed the stream effect on the Tab object and kept lastEventId and backoff as locals of that effect. Every 4 s poll produced a new array, a new Tab, a re-run: the stream reopened with no Last-Event-ID, Fountain replayed the 40-turn history, every replayed turn end called refreshConversations() and readReceipt(), and the backoff went back to 1 s so the 429s slowed nothing. It ran at ~2/s before only because each refresh paid a 200-500 ms unfiltered list on the server; the 5 s memo from #67 made that instant, so the same loop spun 5-10x faster.

The new bundle's callbacks (refreshConversations on [client, paddockId], readReceipt on [client, boxId] where boxId is a string, loadEvents on [client]) are already stable; this PR removes the dependency on that being true.

What changed

1. Stale-bundle self-heal. The server's build id is the hash of the built index.html, so a server-only change reloads no tab. It is stamped on every response as x-paddock-build, written into the served HTML as <meta name="paddock-build">, and exposed on /api/config. The bundle reads its own id from the meta tag (not from the first API answer, which a deploy between the HTML and that call would poison) and reloads once on the first mismatch (src/lib/build.ts). The strip poll refuses a request whose header does not name the current build (409 stale_client); every other route still serves it. A current tab reloads on the mismatch anyway. An old tab, which knows nothing of builds, logs a failed poll and keeps the strip it has, and stops, because its loop only turned on a successful poll. That is the lever the server has over the tab looping right now.
2. Honour 429. src/lib/hold.ts is one shared pause: any 429 sets it to Retry-After (default 15 s) plus jitter, and the poll, the receipt read, the scrollback fetch and the stream reconnect all wait on it. The stream open is now made by the client itself so a non-200 reaches the caller as an ApiError carrying status and Retry-After (the suite's readSse reported it as "the stream closed"); the read loop is the suite's, verbatim.
3. The tail is a state machine (src/lib/tail.ts), not an effect body. Last-Event-ID is kept per conversation across reconnects; the backoff resets only when a stream actually opens (onOpen fires only after a 200); a 429 waits what the server asked. The App.tsx effect depends on [client, activeId, hold] and reads its callbacks through a ref.
4. Bounded replay. The catch-up read on open passes after=<newest id held>, so a reconnect fetches the gap, not limit=1000 pages of history.

Tests

• src/lib/tail.test.ts: a turn-ended stage event does not reopen the stream and is reported once; the second open carries the id the first saw and catches up from it; backoff 1/2/4 s and reset only on a real open; a 429 waits 35 s not the backoff; stop means stop.
• src/lib/hold.test.ts, src/lib/build.test.ts: the pause and the reload-once rules.
• server/app.test.ts: every answer names the build (refusals included) and /api/config exposes it; a stale or missing header is turned away from the strip and nowhere else; an unstamped server turns nobody away; the served HTML carries the meta tag.

221 pass, tsc --noEmit and bun run build green locally.

After deploy

Watch sum by (route) (rate(phoenix_router_dispatch_stop_count{namespace="fountain"}[1m])) for the stream, file and events routes to fall, and paddock's logs for the 429s to stop. The looping tab stops on its next strip poll (it will show a stale strip until somebody reloads it); every tab thereafter reloads itself on the next deploy.

🤖 Generated with Claude Code

https://claude.ai/code/session_018R1XyyWd9MDMYXqFL71owk