fix(paddock): stop hammering the Fountain API from the proxy and the client by jhgaylor · Pull Request #67 · managoat/demos

jhgaylor · 2026-09-07T04:49:55Z

Why

Measured in production on 2026-09-07, the three demo apps on the operator's server key were behind a four-day OOM / DB-pool-exhaustion incident on managoat.com. Paddock was the largest contributor:

Call	Rate	Note
`GET /api/conversations`	~50,000/hour (14/s)	every call returns all ~1,000 conversations on the account and runs an aggregate server-side
`GET /api/sandboxes/:id/file` (the receipt)	~13,000/hour	>99% answered 409 `sandbox_not_ready`; paddock logs show 40 consecutive 409 lines for `.paddock/applied.json`
`GET /api/conversations/:id/stream`	~7,000 opens/hour (2/s)	reopened constantly instead of held

Mechanism

One client bug drove most of it. src/App.tsx keyed the active tab's stream effect on the Tab object, and tabsOf() (shared/tabs.ts) builds fresh Tab objects from every 4 s poll of the strip. So on every poll the effect tore the stream down and reopened it, with lastEventId (a local of the effect) reset to null, so Fountain replayed the tab's entire history each time. Every replayed stage: turn boundary then re-read the receipt and the strip. That is the 40-in-a-row 409 signature: one reconnect on a 40-turn tab on a parked box.

On the server, every one of those proxied requests re-derived "which machine is this and which tabs are on it" from an unfiltered list of the owner's whole account: machineOf() in server/proxy.ts was called from the strip branch, the per-tab branch, the sandbox branch and the write gate, with no memo.

What changed

Server (server/proxy.ts, new server/machine-cache.ts):

machineOf() is memoised for 5 s per paddock and per credential; concurrent misses share one in-flight promise, so a burst of proxied requests costs one list call. Opening a tab, ending one (/terminate) and retiring/rebuilding the machine (lifecycle.ts) call forget() so the next read is fresh.
The cache key includes a fingerprint of the API key (FountainClient.credentialId), so a claim's credential rotation is never served an answer read on the key it just revoked (the existing "compute credential rotates" test covers this).
Once the machine's agent is known, the list is narrowed with ?agent_id=; the whole account is read only when the narrowed list shows no live machine (what a rebuild onto a new agent looks like).
A 409 on the receipt read is no longer logged as a refusal.

Client (src/App.tsx):

The stream effect is keyed by the active conversation id; Last-Event-ID lives in a ref across reconnects, so a reconnect resumes rather than replays.
The strip poll keeps the previous array when the payload is unchanged, so nothing derived from it (tabs, active tab, stream) looks new; and it pauses while the tab is hidden, catching up on visibilitychange.
The receipt is not re-read while the box is parked (a 409 sets a flag); it is read again on the turn-ended event, the one moment a parked box is known to be awake.

Tests

Four new tests in server/app.test.ts pin the cache: one list call per burst of proxied requests; a fresh read after opening or ending a tab; the agent_id narrowing; and the fallback when the machine moved to another agent. bun test (206 pass), tsc --noEmit and bun run build are green locally.

🤖 Generated with Claude Code

https://claude.ai/code/session_018R1XyyWd9MDMYXqFL71owk

…client Measured in production on 2026-09-07, paddock was the largest source of a four-day OOM / DB-pool incident on managoat.com: ~50,000 GET /api/conversations an hour (each returning every conversation on the account and running an aggregate), ~13,000 receipt reads an hour of which >99% were 409 sandbox_not_ready on a parked box, and ~7,000 conversation stream opens an hour. One client bug drove most of it. App.tsx keyed the active tab's stream effect on the Tab object, and tabsOf() builds fresh Tab objects from every 4 s poll, so the effect tore the stream down and reopened it on every poll — with Last-Event-ID reset to null, so Fountain replayed the tab's whole history each time, and every replayed turn boundary re-read the receipt and the strip. Each of those proxied requests re-derived the machine from an unfiltered list of the owner's account. Server (proxy.ts, machine-cache.ts): - machineOf() is memoised for 5 s per paddock and credential, with concurrent misses sharing one in-flight promise, so a burst of proxied requests costs one list call. Opening a tab, ending one and retiring the machine call forget() so the next read is fresh. The key includes a fingerprint of the API key so a claim's rotation is never served the old key's answer. - Once the machine's agent is known the list is narrowed with ?agent_id=, and the whole account is read only when the narrowed list shows no live machine. - A 409 on the receipt read is no longer logged as a refusal; it is the ordinary answer of a parked box. Client (App.tsx): - The stream effect is keyed by the active conversation *id*; Last-Event-ID is kept in a ref across reconnects so a reconnect resumes instead of replaying. - The strip poll keeps the previous array when the payload is unchanged, so nothing derived from it looks new; and it pauses while the tab is hidden. - The receipt is not re-read while the box is parked (a 409 sets the flag); it is read again on the turn-ended event, the one moment the box is awake. Tests pin the cache: one list call per burst, a fresh read after open/end, the agent_id narrowing, and the fallback when the machine moved agents. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018R1XyyWd9MDMYXqFL71owk

…ead of replaying (#69) After #67 deployed (04:56Z) the account-wide list calls fell 14.5/s -> 1.1/s, but one browser tab still running the OLD bundle went into a hotter loop: 3,277 stream opens on one conversation and 4,319 receipt reads on one sandbox in five minutes, events?limit=1000 at ~7/s, and Fountain answering 429 rate_limited with Retry-After up to 35 s. The old bundle's loop was: poll -> new array -> new Tab object -> stream effect re-runs -> reopen with no Last-Event-ID -> replay the 40-turn history -> each replayed turn end re-reads the receipt and the strip. It ran at ~2/s before only because every refresh paid a 200-500 ms unfiltered list on the server; the 5 s memo made that instant, so the same loop spun 5-10x faster. And `backoff` was a local of the effect, reset to 1 s on every re-run, so the 429s slowed nothing. Four changes, each of which stands alone: 1. Stale-bundle self-heal. The server's build id is the hash of the built index.html (so a server-only change reloads no tab). It is stamped on every response as x-paddock-build, written into the served HTML as <meta name="paddock-build">, and exposed on /api/config. The bundle reads its own from the meta tag (not from the first API answer, which a deploy between the HTML and that call would poison) and reloads once on the first mismatch (src/lib/build.ts). The strip poll refuses a request whose header does not name the current build (409 stale_client): a current tab reloads on the mismatch anyway; an old tab, which knows nothing of builds, logs a failed poll and keeps the strip it has -- and stops, because its loop only turned on a *successful* poll. That is the lever over the tab looping now. 2. Honour 429. src/lib/hold.ts is one shared pause: any 429 sets it to Retry-After (default 15 s) plus jitter, and the poll, the receipt read, the scrollback fetch and the stream reconnect all wait on it. The stream open is now made by the client itself so a non-200 reaches the caller as an ApiError with status and Retry-After (the suite's readSse reported it as "the stream closed"); the read loop is the suite's, verbatim. 3. The tail is a state machine (src/lib/tail.ts), not an effect. Last-Event-ID is kept per conversation across reconnects; the backoff resets only when a stream actually opens (onOpen fires only after a 200); a 429 waits what the server asked. App.tsx's effect depends on [client, activeId, hold] and reads its three callbacks through a ref, so no state change can re-run it. Tests pin: a turn-ended stage event does not reopen the stream and reports once; the second open carries the id the first saw; backoff 1/2/4 s and reset on open only; a 429 waits 35 s; stop means stop. 4. Bounded replay. The catch-up read on open passes after=<newest id held>, so a reconnect fetches the gap, not limit=1000 pages of history. Claude-Session: https://claude.ai/code/session_018R1XyyWd9MDMYXqFL71owk Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

jhgaylor merged commit 8d20188 into main Sep 7, 2026
2 checks passed

jhgaylor mentioned this pull request Sep 7, 2026

fix(switchyard): memoise the machine lookup instead of listing per request #68

Merged

jhgaylor deleted the fix/paddock-api-load branch September 7, 2026 04:59

jhgaylor mentioned this pull request Sep 7, 2026

fix(paddock): reload stale tabs, honour 429s, and resume streams instead of replaying #69

Merged

Navigation Menu

Uh oh!

fix(paddock): stop hammering the Fountain API from the proxy and the client - #67

fix(paddock): stop hammering the Fountain API from the proxy and the client#67
jhgaylor merged 1 commit into
mainfrom
fix/paddock-api-load

jhgaylor commented Sep 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jhgaylor commented Sep 7, 2026

Why

Mechanism

What changed

Tests

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant