feat(sandbox): add bounded capacity queue

[jhgaylor]

Summary

• add a bounded per-tenant FIFO queue for fresh sandbox starts at the tenant or fleet capacity ceiling
• let API callers opt in with queue: true and automatically preserve scheduled teammate runs
• drain on capacity-release events with an Oban backstop, atomic claims, stale-claim recovery, audit events, and telemetry
• expose queue list, outcome, and cancellation endpoints and update the OpenAPI contract, generated TypeScript types, operator docs, and ADR 0042

This is a clean implementation on current main. It uses the useful product shape from #1054 as reference without rebasing or reviving that branch, and incorporates the review feedback around current credit semantics, fleet-full handling, claim-before-start, channel-aware replay, terminal prompt removal, and SDK contract coverage.

Verification

• mix compile --warnings-as-errors
• changed-area suite: 179 tests, 0 failures
• migration rollback and reapply
• Credo, Dialyzer, and Sobelow
• SDK wire contract: 161 operations, 182 schemas
• TypeScript typecheck and 101 SDK tests
• documentation style, links, Vale, and ADR validation
• full ExUnit suite: 4,194 tests; five timing/integration failures all passed on targeted retry after branch-specific fixes

Closes #1033

[BinaryBourbon]

Requesting changes — same two reasons as #1489: the branch can't be reviewed in this state, and the test story needs resolving.

207 commits behind main, conflicting in 11 files: conversations.ex, fountain_web/telemetry.ex, decisions/index.md, docs/api.md, docs/configuration.md, sdk/contract/contract.json, sdk/typescript/src/generated/openapi.ts, both SDK version files and sdk/typescript/src/http.ts. Last run September 3rd. decisions/index.md conflicting is a reminder that the index is generated — it wants scripts/decisions-index.sh on the rebased tree, not a manual merge.

~2,171 lines across 44 files, which is over the line. A bounded queue splits along its own architecture almost too neatly: the table + migration, the claim/drain state machine, the capacity-release event wiring and Oban backstop, queue: true on the API, then the list/outcome/cancel endpoints with their OpenAPI and SDK surface. Five PRs, each with its own tests, each reviewable in an evening.

And this:

full ExUnit suite: 4,194 tests; five timing/integration failures all passed on targeted retry after branch-specific fixes

Five timing failures on a PR whose entire subject is queueing under capacity pressure is not a footnote — that's the feature's own failure mode showing up in its test suite. "Passed on targeted retry" is the specific thing CLAUDE.md tells us not to accept: keep the failed run's evidence, work out what the test asserts on versus what it waits on, and file it if it's real. If those five were genuinely ambient, they should have flake issues. If they weren't, they're the bug.

Worth pinning down before a rebase, because the answer probably changes the design rather than the test.

Recommendation: close and re-cut on current main as a stack, carrying the queue semantics forward. The product shape described here reads well — bounded FIFO, atomic claims, stale-claim recovery, :fleet_full staying a 503 rather than a 402 — and none of that is in question. It's the delivery that needs changing.

[jhgaylor]

Closing in favour of a chained stack of seven small PRs on current main, under the no-big-PRs rule. This PR was 2,171 lines across 44 files and 207 commits behind; conversations.ex alone had since absorbed the apply-kinds (#1636) and labels (#1637) stacks and the execution-limits work, so the stack is re-derived against today's code rather than rebased. This PR was read as the spec.

PR	Branch	What it covers
#1869	stack/1033-1-requests-table	The sandbox_requests table, SandboxQueue.Request, and the tenant-scoped half of the context: enqueue under the per-tenant depth bound (advisory lock, namespace 4317), list, get, position, cancel. Audit rows in audit_guardrail_test.exs.
#1870	stack/1033-2-claim-drain	The claim/drain state machine: FIFO claim, capacity/transient/terminal classification, expiry, and the any_active_requests? / user_ids_with_active_requests reads.
#1871	stack/1033-3-drain-on-release	Workers.SandboxQueueDrainer, the fan-out poke at Conversations.update_sandbox/2, and the five-minute cron backstop.
#1872	stack/1033-4-queue-opt-in	queue: true on POST /api/conversations (202 + position), and a teammate schedule's cron firing queueing itself. SandboxRequest schema and JSON view.
#1873	stack/1033-5-queue-endpoints	GET/GET :id/DELETE :id on /api/sandbox-queue, with the OpenAPI operations and the SDK omission claim.
#1874	stack/1033-6-telemetry	The three telemetry series, the OpsGauges live-row gauge, and the RetentionPruner window.
#1876	stack/1033-7-docs	ADR 0042, the 0005 addendum, decisions/index.md, docs/api.md, docs/configuration.md, docs/architecture.md, the prices guide, the marketing copy, .env.example, CHANGELOG.md, SDK 1.28.0.

Every decision in this PR is carried. Four things are deliberately different:

1. The claim is fenced on the version it observed. This PR recovered stale claims in a separate update_all pass and then wrote outcomes with a blind Repo.update/1. That leaves a window where a replay slow enough to lose its claim overwrites the row a recovering drain has already replayed — one request reads as one conversation while two are running. feat(sandbox): claim before replay, fenced on the version claimed (#1033) #1870 folds recovery into the claim and fences every later write on (status, updated_at), with two regression tests.
2. The queued-attributes allow list carries labels, execution_limits and sandbox_api_access. This PR predates all three, so a queued start on this branch would silently have dropped a conversation's labels and its execution limits.
3. docs/api.md is prose on today's main (the Diataxis rebuild, Docs IA redesign: the remaining steps #903) rather than the route listing and status table this PR edited, so the same content lands as a "Wait for capacity" section.
4. The persistent-mode "refused, not queued" sentence on the marketing home page stays. That ceiling is sandbox_at_capacity on one machine, which this queue does not cover; only the two sandbox-cap sentences change.

On the five timing failures

The verification notes here read "full ExUnit suite: 4,194 tests; five timing/integration failures all passed on targeted retry after branch-specific fixes". That is not a footnote in a PR whose subject is queueing under capacity pressure, and a targeted retry is exactly what CLAUDE.md says not to accept, so it was treated as a finding rather than noise.

They never happened in CI. Both CI runs on fix/1033-bounded-sandbox-queue were green on the suite: run 33735429555 and run 33742346612, attempt 1 each, no reruns. The only red job on that branch was "A change to the SDK says whether it releases" on the earlier push, which is the version-bump gate and not a test. So the five were failures of a local umbrella mix test, and no artefact of them survives — no assertion text, no file, no seed.

That rules out the first hypothesis, a race between the event-driven drain and the Oban backstop. config/test.exs sets testing: :manual, so no drain job ever executes in the suite; the only thing a poke does in a test is insert a row. There is no "the test waits for the drain but asserts on the claim" in this design, because nothing waits.

What is left is the local-only contention this feature is unusually exposed to. Quotas.with_sandbox_reservation/3 takes pg_advisory_xact_lock(4315, 0) — one database-wide lock, on a fixed key, before the per-user one (quotas.ex:232). Under the SQL Sandbox a test's transaction is never committed until the test ends, so every test that reserves a sandbox holds that single global lock for its whole body, and pg_advisory_xact_lock has no timeout. Two things make that bite locally and not in CI:

• worktrees share fountain_test, and an advisory lock is scoped to the database, not the connection pool — so a second agent's suite running at the same time contends on the same key;
• this feature's drain calls start_or_resume_conversation/2 once per queued request, so a single drain test takes that global lock several times over.

The repo has already written this failure signature down twice, from both ends. apps/fountain/test/fountain/quotas_fleet_ceiling_test.exs opens with a header explaining that capacity assertions failed "a different one each time, on some seeds and not others, and never locally", and apps/fountain/test/fountain/conversations/admission_lock_isolation_test.exs exists solely to assert that a launch does not hold the fleet lock while it waits. This PR's sandbox_queue_test.exs is async: true, and its fill_fleet/0 and fill_tenant_cap/1 helpers put it squarely in that family.

One failure was reproducible, and it was the feature's own bug. This PR's test "a claim recovered mid-attempt is released without failing the job" stubs Billing.check_spend/1 to move the claimed row back to queued on every call. Writing the equivalent test against the re-derived code, it returned started: 2 where it asserted started: 1: the drain loop re-claimed the row the stub kept resetting, and started a second conversation for one request. That is not a test artefact. It is the blind Repo.update/1 in finish/2 — a replay that loses its claim still writes "started" and its own conversation id over the row a recovering drain has already replayed, so one request reads as one conversation while two are running. #1870 fixes it by folding stale-claim recovery into the claim itself and fencing every write on the (status, updated_at) the drain observed; the two regression tests are in sandbox_queue_drain_test.exs.

No flake issue filed. gh issue list --label flake --state all has no match for these, but CLAUDE.md is explicit that such an issue needs the failing assertion with its left:/right:, the file and line, and how often out of what — and none of that was recorded for the five. "Five unnamed local failures on a branch that is now closed" is an issue nobody could close. The one mechanism that was identifiable is fixed in the stack, and the contention above is a pre-existing property of with_sandbox_reservation/3 under the SQL Sandbox rather than something this feature introduces.

The stack's own verification, rebased onto main at ea71d0ed and run against a private database so nothing else contended: core + ee 4,934 tests, 0 failures (seed 424242), fountain_buzz 144, 0, fountain_support 33, 0. Nothing was retried.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S9jevFQT5MkF3rJUieeiHW

[jhgaylor]

Re-cut as #1869 through #1876; see the comment above for the mapping and the finding on the five timing failures.

[jhgaylor]

Correction to the verification line above: the 4,934 / 0 failures figure was the run against the stack at its original base. The stack has since been rebased onto main at ea71d0ed, and the run against that tree is core + ee 4,977 tests, 0 failures (seed 424242), fountain_buzz 144, 0, fountain_support 33, 0, exit 0. Nothing was retried in either run.

The rebase also produced the SDK version collision the comment anticipated: 1.27.0 had been published to npm by the #1637 stack, the release gate refused it (@managoat/fountain-sdk@1.27.0 is already on npm), and #1876 now carries 1.28.0.