Sandbox-binding campaign (#1767): the lifecycle writes it leaves unfenced

[jhgaylor]

Found while reviewing the nine fix: PRs that bind conversation lifecycle
writes to the actor's sandbox (#1996, #1998, #1999, #2000, #2001, #2002, #2003,
#2006, #2007 — all approved 2026-09-12). Each of those fences the write it
names, and the mechanism is sound: state.sandbox_id is set once from init args
(conversation_server.ex:463) and never reassigned, so a mismatch is genuinely
a stale actor.

This issue collects what the campaign does not cover. Every item below was
verified at the stack tip (codex/p2-1767-machine-gone-senders), and none is
claimed by any of the 25 codex/p2-1767-* branches.

1. mark_orphan/3 reaches the write unbound

apps/fountain/lib/fountain/conversations/conversation_server.ex:1443

defp mark_orphan(_state, running_turn, why),
  do: Conversations._unsafe_orphan_turn(running_turn, why)

It is handed state and discards it, and its turn comes from
find_running_turn(conv_id), which is scoped by conversation and never by
sandbox. It is reached from the reattach path — :1330 :1339 :1390
:1438 — which runs inside a live actor.

Failure: old actor A on sandbox S1 restarts and reattaches; the conversation has
since been rebound to S2 with actor B running a turn; A's list_sessions retry
expires under a partition; mark_orphan flips B's live turn to
interrupted and the conversation to idle. That is the exact bug class this
campaign exists to close, in a function the campaign edits.

Do not confuse this call site with AutonomousTurnReaper's unbound one
(apps/fountain/lib/fountain/workers/autonomous_turn_reaper.ex:69) — that is
safe because it is unbound, since it only fires when
ConversationServer.whereis/1 is nil and no live actor exists.

2. handle_call(:release_conv, ...) is a third unfenced conversation-ending path

conversation_server.ex:1609

def handle_call(:release_conv, _from, state) do
  state = drop_connection(state, "released")
  conv = Conversations._unsafe_get_conversation!(state.conversation_id)
  {:ok, _} = Conversations.update_conversation(conv, %{status: "terminated"})

The clause above it refuses on current_turn ({:error, :busy}), but that is
the stale actor's own turn — an old actor holding no local turn releases
happily while a new actor runs one on the new sandbox.

This one compounds. _unsafe_complete_turn/3 guards on
conv.status not in ["terminated", "failed"], so a stale release does not just
flip the row to a terminal state, it then makes the new actor's legitimate turn
completion refuse as well. _unsafe_finish_conversation_termination/2 drops
straight in.

3. The remaining unbound update_conversation/2 status writes

The campaign fenced the ending writes it enumerated and left these going
through plain update_conversation(conv, ...):

• %{status: "running"} at conversation_server.ex:1419 and :2317 (the
  pre-spawn write in the fresh-turn path, from a possibly stale struct)
• %{status: "failed"} at :627, :790, :830, :869, :1027, :1039

Reachability by a stale actor is unproven per site — several are provisioning
paths where the writer is necessarily the current actor. This wants an audit
pass deciding which need the fence, not a blanket change.

4. follow_cotenants breaks the guard's premise

apps/fountain/lib/fountain/conversations.ex:5282 rebinds a co-tenant's row from
a sibling conversation's provisioning and starts no server for it. So the
guard's assumption — "a rebind means a newer actor owns the status" — does not
hold: it casts machine_gone with the old id, update_alls the rows to S2, and
the receiver's finish call sees S2 vs S1 and :noops. The conversation is left
status: "running" with no server, where the old code idled it immediately.
Healed only by AutonomousTurnReaper after 30+15 min, or not until the next
prompt. The fix belongs with the sender, which already owns both the
update_all and the stage publish.

5. Nothing reconciles a :noop

Every PR in the chain defers to a "durable forced-teardown recovery" follow-up,
and none adds a reconciler. While an abandoned turn row stands
(running, started_at set, ended_at: nil, and finish/4 nils row so
terminate/2's orphan_on_normal_stop net cannot see it):

• _unsafe_sandbox_busy_elsewhere? (conversations.ex:2036) counts
  t.status == "running", so a live co-tenant never idle-parks and bills to max
  lifetime;
• SandboxUsage clips an unended turn to the sandbox end
  (sandbox_usage.ex:445-467), inflating busy_seconds/turn_seconds on the
  replacement sandbox, while CreditPricer charges nothing because it needs
  ended_at;
• _unsafe_running_turns_elsewhere/2 joins through the conversation's current
  sandbox, so the phantom consumes a capacity slot on the replacement (ADR
  0023). With gemini or opencode (concurrency: 1 in the adapter table), a
  co-tenant's next turn is refused :sandbox_at_capacity until a backstop
  fires.

Two backstops do eventually close it — the next ConversationServer start via
reattach_running_turn → mark_orphan (which works only because of item 1's
unbound read), and AutonomousTurnReaper once no server holds the conversation.
A sweep for abandoned running rows belongs here.

6. One-line hardening worth taking now

apps/fountain/lib/fountain/conversations/turn_machine.ex:111

sandbox_id: Map.get(state, :sandbox_id),

Every sibling field in from_state/1 uses direct access; only the fence input
is read leniently, and @type admits String.t() | nil so dialyzer is blind to
a nil. That value is the fence input at :624 (_unsafe_complete_turn), :673
(_unsafe_interrupt_turn) and :698 (_unsafe_idle_interrupted_turn), each
comparing against a NOT NULL column — so a nil matches nothing and silently
:noops every completion, interrupt and idle, with no crash and no warning.
A renamed key or a partial state map turns the whole campaign's guard off
quietly.

No fixture needs the lenient read: the round-trip test at
apps/fountain/test/fountain/conversations/turn_machine_test.exs:62 already
supplies sandbox_id: "s", and the live state gets it from Keyword.fetch! at
init. state.sandbox_id is strictly better, and four of the nine PRs reuse this
helper.

Also observed, not defects

• No branch in the programme touches CHANGELOG.md, for a fix whose
  user-visible symptom is a lost turn result. One ### Fixed line on the last
  PR of the chain covers Make sandbox reset and retirement atomic with turn admission #1767.
• The size pin has slack. conversation_server_size_test.exs:18 is
  @pin 2759 (the only pin) against a 2743-line file. main is exact at
  2646/2646; the +116 raise happened below the reviewed batch, in the
  test(sandboxes): prove reset and admission commit order #1968-fix(conversations): persist forced teardown intent separately #1983 range. Those 16 lines quietly let a later PR grow the file for
  free — tighten to the real length after the final rebase.
• Deploy ordering is only safe in one direction. fix: scope machine-gone handling to the actor sandbox #2006 is safe standalone
  (its 4-tuple clause at :1687 self-qualifies with state.sandbox_id, so
  un-migrated senders still work). The unsafe direction is a mixed-version
  cluster: a new node's 5-tuple cast reaching an actor still on main, whose
  receiver takes a 4-tuple (main:1740) and whose catch-all (main:1758) logs
  "unexpected cast" and :noreplys, keeping a dead handle. Bounded by the
  rolling restart, but "deploy the receiver first" is unachievable if fix: scope machine-gone handling to the actor sandbox #2006 and
  fix: identify the lost sandbox in machine-gone notifications #2007 ship in one release.

[jhgaylor]

Two corrections to this issue, from reviewing the lower half of the same
programme (#1943-#1983) after filing it.

The size-pin note was wrong, and the real problem is merge-blocking

I wrote that the pin "was raised to 2762 somewhere below the reviewed batch,"
creating slack. There was no raise. @pin in
apps/fountain/test/fountain/conversations/conversation_server_size_test.exs
has only ever shrunk on main:

commit	date	@pin
59c5ebc8	2026-09-07	2775
317b00d5 (#1764)	2026-09-09	2774
a372ce44 (#1891)	2026-09-12	2762
94f879c4 (#1749)	2026-09-12	2652
4f0c040c (#1751)	2026-09-12	2646, zero headroom

Every branch in this programme carries @pin 2762 because that is what its
merge base (3f59a76a, 2026-09-11) had. main then lowered it twice, today.

So the actual consequence is the opposite of slack: conversation_server.ex is
2691-2757 lines across these branches against main's 2646/2646. Every
green CI run in the programme predates the lowering, so on rebase onto current
main they all fail conversation_server_size_test.exs — the nine PRs at the
top (#1996-#2007, 2737-2757 lines) included.

This needs one decision for the whole programme before anything merges: raise the
pin once, or extract the new clauses into a Fountain.Conversations.* module.
Nineteen PRs cannot each answer it separately.

Item 5 overstated the reconciliation gap

"Nothing reconciles a :noop" is too broad. AutonomousTurnReaper sweeps any
running turn with no live server (30 min stuck + 15 min quiet) and
_unsafe_orphan_turn/3 flips the conversation back to idle, so every
stop-normal path — which is what the retirement, wake and park exits in
#1943-#1946 produce — is covered.

The gap as it actually stands is narrower: a :noop fence where the server
stays alive. The reaper skips those, because reap_if_unowned/1 will not act
while ConversationServer.whereis/1 finds a server. That is still worth a
sweep for abandoned running rows, and the billing and capacity consequences
listed in item 5 hold while such a row stands — but the reaper is not absent, it
is merely gated on liveness.

Items 1, 2, 3, 4 and 6 stand as filed.

[jhgaylor]

Item 7: no reconciler for a fence whose teardown died

Three independent reviews of the teardown-fence half of this programme
(#1973/#1975/#1977, and #1982 by a second door) converged on one mechanism, so
adding it here as its own item. Verified on origin/main.

Both SandboxReaper sweeps filter the fence column out:

• apps/fountain/lib/fountain/workers/sandbox_reaper.ex:125 —
  s.status in ^@active_statuses and is_nil(s.reset_requested_at) and s.updated_at < ^cutoff
• :228 (sweep_abandoned_sandboxes/0) —
  s.status == "ready" and is_nil(s.reset_requested_at) and s.updated_at < ^grace_cutoff

while Quotas.active_sandboxes/0 (quotas.ex:145) keeps counting the row.

So a fence that commits and whose terminal write then does not land
leaves a row that is invisible to both reclaim sweeps and still consuming one
tenant slot and one fleet slot, with the provider sprite leaked. It is permanent:
nothing clears reset_requested_at for a teardown that died.

Ways to reach it, all widened by this programme:

• the retirement write after the fence fails (fix(conversations): fence termination without a running actor #1982's own test asserts this
  outcome as acceptable);
• Managoat.Sandbox.destroy/1 or Egress.release/2 raises before the row is
  terminated (lifecycle.ex:472-486) — on main
sweep_abandoned_sandboxes reclaimed exactly this;
• the pod dies between the fence commit and the terminal write;
• an account deletion halts mid-fence (see fix(accounts): fence admission before tenant compute teardown #1977), leaving rows ready +
  fenced.

This is the same shape as item 5 but a different cause: item 5 is an abandoned
turn row that the turn reaper skips while a server is alive; this is an
abandoned sandbox row that the sandbox reaper skips because the fence column
is set. Both need a sweep that looks for the fenced-and-stale case rather than
excluding it.

teardown_requested_at (added by #1983) gives such a sweep the predicate it
needs — teardown_requested_at is not null and status not in ('terminated','failed')
— and #1894 is the open issue describing the dead end. Today that column has one
reader: the fence's own idempotency check.