Journal bounded turn deadlines and remote stop intent

[jhgaylor]

A late ACP response must not turn an expired review into success, and a lost stop acknowledgment must not authorize another execution. This draft adds the durable journal and database fences needed for those guarantees.

It persists immutable deadlines and remote bindings, arbitrates completion versus expiry, preserves failed/interrupted turns until remote termination is confirmed, prevents reset while execution remains unresolved, and recovers abandoned stop attempts without replay.

Timeout activation remains pending API/account policy, trusted session-identity transport, scheduling and the remaining lifecycle integration. Sprites session metadata awaits superfly/sprites-ex#33. The proposed ADR records the integration checklist and validation limits. This supplies the journal foundation for #1732.

Validation: full mix precommit passed (4,632 tests + six doctests; zero failures, two skips, seven exclusions). The standalone PostgreSQL proof ran 20 completion/expiry races and competing termination claims on separate connections, plus confirmed parent/turn lock-delay cases. It made no provider calls. Source hashes and results are in decisions/evidence/turn-deadline-races.json.

No production deadline has been activated.

Part of #1732 — held. Superseded on main by #1773–#1793; the residual hunks are being re-cut as focused PRs tracked by #1864, which keeps these frozen until each has a replacement or an explicitly linked deferral. Do not close: they are the reference for that mapping.

[BinaryBourbon]

Review: the journal itself is good; two things need answering before it becomes the base

I reviewed this in a worktree off pr-1744 against its own isolated database (mix deps.get, ecto.create, ecto.migrate, targeted probes). CI on 6c885c7b is green across all six partitions, static analysis, coverage and the release/contract job; only CI required is still pending. The state machine is careful work — the lock ordering (machine → parent → journal → turn), reading the clock after the locks, Audit.record outside the transaction, and the immutability checks in TurnExecution.changeset/2 are all right, and the 30 regressions genuinely cover them.

Requesting changes on two points, both about what this schema commits the next eight PRs to rather than about the code as written.

---

1. awaiting_identity and uncertain are absorbing states, and one of them takes the owner's escape hatch with it

Nothing in this PR — or anywhere in the rest of the stack, I checked all nine — ever revisits a row in awaiting_identity or uncertain:

• _unsafe_due/2 selects state == "active" and deadline_at <= now, or state == "ready".
• _unsafe_recover_submissions/2 selects state == "submitted" only.
• _unsafe_claim_termination/2 refuses anything that is not "ready".
• _unsafe_record_termination/4 needs the attempt_id, and refuses uncertain unless last_error == "termination_unconfirmed".

Both states are in @fenced, and both are outside ["completed", "stopped"], so while a row sits in one of them _unsafe_fenced?/1 is true, open_execution?/1 refuses registration, and _unsafe_sandbox_open?/1 makes reset_sandbox/2 return :sandbox_mid_turn — permanently, with no timeout, no retry and no operator override.

Two probes against this branch, both pass, i.e. both dead-ends are real:

# spawn intent recorded, provider never reports a session id
{:ok, _} = ExecutionGuard._unsafe_claim_spawn(ex.id, now: now)
{:ok, _} = ExecutionGuard._unsafe_expire(ex.id, now: deadline)
assert Repo.get!(TurnExecution, ex.id).state == "awaiting_identity"

assert ExecutionGuard._unsafe_due(DateTime.add(deadline, 86_400)) == []
assert ExecutionGuard._unsafe_recover_submissions(DateTime.add(deadline, 86_400)) == []
assert {:error, :execution_fenced} =
         ExecutionGuard._unsafe_register(successor.id, Ecto.UUID.generate(), later)
assert {:error, :sandbox_mid_turn} = Conversations.reset_sandbox(ctx.home)   # forever

# two identities on one connection
{:ok, _} = ExecutionGuard._unsafe_bind_identity(ex.id, conn, "sess1")
{:ok, _} = ExecutionGuard._unsafe_bind_identity(ex.id, conn, "sess2")
# => state "uncertain", last_error "conflicting_identity"
assert {:error, :not_ready}      = ExecutionGuard._unsafe_claim_termination(ex.id)
assert {:error, :stale_attempt}  = ExecutionGuard._unsafe_record_termination(ex.id, nil, :ok)
assert {:error, :sandbox_mid_turn} = Conversations.reset_sandbox(ctx.home)   # forever

This is not hypothetical once the stack lands. ExecutionDeadlineWorker in #1746 rescues any failed provider call to {:error, :termination_unconfirmed} and records it, and its tasks carry a 10 s :timer.kill_after — so every provider termination that fails or is slow produces an uncertain row that nothing ever retries, and the conversation plus its shared home are fenced for good. On a persistent home under ADR 0023 that is every conversation on the machine, not just the bounded one.

reset_sandbox/2 is the documented owner-facing recovery for exactly this class of problem (#1071). Making it refusable-forever from a journal the owner cannot see or clear is the part I would not want to discover in production.

What I think this needs before the base lands, in rough order of preference:

• a bounded obligation: awaiting_identity and uncertain carry an age, and past it the row retires to stopped with the uncertainty recorded on the turn — the fence exists to stop a replay, and after the sandbox is gone there is nothing left to replay onto;
• failing that, an explicit operator lever (an admin release, or reset_sandbox(force: true) that retires the journal alongside the machine) and an admin surface that shows why a home will not reset;
• at minimum, name this in ADR 0046. The "Required integration and acceptance" list covers retention and account deletion but says nothing about a row that never resolves, and the state table describes uncertain as "retain the fence" without saying for how long.

2. reset_sandbox/2 changes behaviour today, and the duration it bills no longer matches the row

Terminating inside the transaction is the right call for the fence, but _unsafe_retire_home/1 still runs afterwards and still writes the row. Since update_sandbox/2 re-reads current under FOR UPDATE (#1761), the second write now finds status == "terminated", so record_sandbox_usage/2 no-ops — no double metering, good — but it does still bump terminated_at to the post-destroy clock, after the usage row was computed from the pre-destroy one. Measured on this branch with a 1.2 s stubbed destroy:

inserted_at=~U[2026-09-11 04:35:05Z]  terminated_at=~U[2026-09-11 04:35:06Z]
usage duration_ms=0
actual span ms=1000

On main both come from the same post-destroy timestamp and agree. It is a small absolute error (one provider call per reset), but sandbox_terminated.duration_ms is what a provider bill is reconciled against, and having it disagree with the row's own terminated_at is the kind of drift that is painful to explain later. Either pass the post-destroy stamp through, or stop the retire helper writing the row on this path — right now that half of _unsafe_retire_home/1 is dead on the reset path while its docstring still claims to do it.

Also worth a line in the docstring: record_sandbox_usage/2 (a usage_events insert plus the PostHog mirror) now runs inside the transaction that holds pg_advisory_xact_lock(4316, …). It is cheap and Sink.enqueue does not block, but the rule elsewhere in this codebase is that nothing under a process-wide advisory lock waits on anything, and this is a new write under it.

---

Smaller things, none blocking

• _unsafe_write_turn/3 costs a turn_executions lookup on every turn write. Repo.get_by(TurnExecution, turn_id: turn.id) runs before every _unsafe_update_turn/2 — including Pending.ex:91, which fires on every permission request of every turn. Indexed and cheap, but it is unconditional on the hottest write path in the system and worth a sentence saying so.
• maybe_put_reply_text/2 moved from get_change(:status) to get_field(:status). I follow why (the guard drops :status from attrs when fenced, so get_change would miss it), but it changes the unbounded path too: a turn that is already terminal with reply_text: nil will now re-run _unsafe_turn_reply_text/1 — a conversation lookup plus every log event for the turn plus block assembly — on each later update, where before it ran once. The comment that used to explain "the write that materialises the reply, not every later update" was deleted with it; the new behaviour deserves its own.
• valid_session_id?/1 is ~r/\A[A-Za-z0-9_-]+\z/. That refuses a dot or a colon, which several session-id formats use. If the restriction is deliberate (it reads like path-traversal defence, and the test uses "../other"), say so; otherwise this will reject a legitimate identity at bind time and land the execution in the uncertain dead-end above.
• decisions/0046 says 4,632 tests; on main today that number has moved. Not worth chasing, but the ADR's stale_after: 2026-09-14 is three days out — it will need a refresh before this merges regardless.

On the stack

For the record, since this is the base: #1745 and everything above it are forked from 59c5ebc8 and do not contain this PR's head commit. Since that fork point main has landed the execution-limit admission campaign (#1787–#1793), which ships its own Fountain.Conversations.ExecutionLimits, users.execution_limits and an execution_allowances table. #1745 adds a second, earlier-timestamped migration for the same users.execution_limits column. I have put the detail on #1745; it does not change anything here, but this PR cannot be merged as "the base" of the stack until the stack is actually rebased onto it.

Happy to re-review quickly once the fence has a way out.

[jhgaylor]

Pushed c0216625 addressing both blockers.

1. The fence now has two exits, neither of which authorizes a provider write.

• ExecutionGuard._unsafe_retire_unresolved/2 ages a row out of awaiting_identity or uncertain. It keeps last_error, so the trail still says the operation was never confirmed — written off, not erased. A session that really did survive is the SandboxReaper's to find, the same as every unbounded turn's. The scheduler that calls it belongs to the deadline supervisor in Enforce journal deadlines independently of conversation actors #1746; ADR 0046's integration table now names that sweep.
• reset_sandbox(force: true) resets past an unresolved execution and retires the journals bound to that machine with it. It deliberately does not override a running turn — that one ends on its own, and interrupting it is a different operation.

On why forcing is safe rather than merely permitted: the row is terminated in the same transaction, and current_binding?/1 already requires a live sandbox of the same id, name and provider, so no attempt can reach a replacement built under a reused name. Destroying the machine is the strongest termination available, so recording it as one is honest. _unsafe_retire_sandbox/1 runs inside the caller's transaction and records nothing itself — an audit insert that failed there would abort the reset it belongs to (ADR 0013) — so the trail is written by audit_retired/1 after commit.

2. terminated_at and the billed duration now agree. _unsafe_retire_home/1 no longer passes its own timestamp. A caller that already retired the row under its machine lock keeps its stamp; stamp_terminated_at/1 fills the gap for one that did not. The regression asserts duration_ms == terminated_at - inserted_at against a 1.1 s stubbed destroy — it fails with left: 0, right: 1000 on the old code.

Also addressed, as comments rather than behaviour: why _unsafe_write_turn/3's per-write lookup is deliberate, why maybe_put_reply_text/2 moved to get_field and what that changes on the unbounded path (it is the same work _unsafe_backfill_reply_texts/0 does, on the same rows, for the same reason), and why valid_session_id?/1 is narrower than what a provider might send.

Verification. Nine new regressions. Each of the three fixes was reverted in turn to confirm its test fails: the age-out reverts to 4 failures, force: to 1, the timestamp to left: 0, right: 1000. Full core suite from apps/fountain: 6 doctests, 4,961 tests, 0 failures, 2 skipped. mix format --check-formatted, mix compile --warnings-as-errors and mix credo --strict (7,258 mods/funs, no issues) all clean — credo's ownership check wanted an # ownership: line within ten of the new _unsafe_ call, which it now has. okf validate decisions valid; index regenerated; stale_after moved to 2026-09-21.

The stack note stands: #1745 and up are forked from 59c5ebc8 and now need rebasing onto this head as well. I'm working that next — the collision with the admission campaign is the substantive part, and I've left the detail on #1745.

[BinaryBourbon]

Reviewed c02166253087d4eccef94c2cae66abf415fb421c (the head when I started; if it has moved since, this covers that SHA only). CI is green on it across all six partitions, static analysis, coverage, the release/contract job and CI required.

Both blockers are answered. Approving.

The absorbing states now have exits, and neither of them authorizes a provider write. _unsafe_retire_unresolved/2 is the first preference I asked for — an obligation with an age. It writes awaiting_identity and uncertain off to stopped, keeps last_error so the trail still says the operation was never confirmed, and the four new regressions cover the two entry paths, the release of both the conversation fence and the machine fence, and the sweep leaving completed/stopped alone. The property that convinces me it is safe rather than merely convenient is one the tests do not name: a retired row is stopped, not completed, so prior_connection/1 in _unsafe_register still refuses :connection_retired on that connection id. Giving up on the obligation does not hand the same connection to a later turn — it only stops the conversation being hostage to it. That is the right line.

reset_sandbox(force: true) is the second, and the reasoning in its docstring holds: the sandbox row is terminated in the same transaction that retires the journals, current_binding?/1 requires a live row of the same id, name and provider, so no attempt can reach a replacement built under a reused name. It correctly still refuses a running turn — _unsafe_running_turns_elsewhere/2 is outside the force? guard — and sandbox_reset_test.exs asserts exactly that. _unsafe_retire_sandbox/1 runs inside the caller's transaction and records nothing; audit_retired/1 writes the trail after commit, which is ADR 0013 followed rather than cited.

terminated_at and duration_ms agree, and the fix is real rather than cosmetic. _unsafe_retire_home/1 no longer passes its own stamp; the reset's in-transaction stamp survives, and the second update_sandbox/2 finds nothing to change. I checked the other caller: _unsafe_destroy_home/1 (agent delete, conversations.ex:2948) does not pre-stamp, so stamp_terminated_at/1 genuinely fills that gap — the new comment describes a caller that exists. The regression compares metadata["duration_ms"] against the row's own span, which is the assertion that fails on the old code for the stated reason rather than by luck.

The four smaller items are all answered as comments or ADR text, and stale_after moved with the index regenerated.

Findings below are non-blocking. I did not re-run the three revert proofs; I read the tests and relied on CI for the pass.

---

Findings

1. conversations.ex:2996 / decisions/0046-durable-turn-deadlines.md:106 — the "owner's lever" is not reachable by an owner, and the ADR says it ships now.

DELETE /api/sandboxes/:id (sandbox_controller.ex:95) is the only owner-facing reset door, and it passes Audited.attribution(conn) and nothing else. No console surface, no admin surface, no force parameter anywhere. The docstring says "It is the owner's lever for a fence nothing else can clear"; ADR 0046 line 106 says "the second needs no scheduler and ships with the journal". Both describe behaviour no owner can reach.

Failure scenario: the incident the ADR describes happens after the stack activates — a provider never acknowledges a stop, the row sits uncertain, the shared persistent home is fenced. The owner calls the documented recovery, gets 409 sandbox_mid_turn, and the lever the ADR promised is reachable only from a remote shell on a prod pod. Bounded, because the ageing sweep is now the real exit — which is why this is a finding and not a block — but the two sentences claim more than the code does, and this repo's rule is that a docstring must not describe unbuilt behaviour as existing. Either wire a force on the delete (seven gates, so probably a PR of its own) or soften those two sentences to say the door is owed and add it to the integration list.

2. conversations.ex:2948 — the agent-delete path destroys a home without retiring its journals.

_unsafe_destroy_home/1 calls _unsafe_retire_home/1 directly, bypassing do_reset_sandbox/2 and therefore ExecutionGuard._unsafe_retire_sandbox/1. Open rows bound to that machine survive its destruction with _unsafe_sandbox_open?/1 still true against a terminated sandbox. It converges once #1746 exists — a ready row is picked by _unsafe_due/1, current_binding?/1 is false against the terminated row, it goes uncertain, then ages out — so nothing is permanently stuck. But reset and agent-delete are siblings and only one of them cleans the journal; worth either the same call or a line saying why the long way round is fine here.

3. execution_guard.ex:383 — the sweep's return value cannot distinguish a row it retired from one it skipped.

The double-check inside with_execution/2 returns {execution, nil, nil} when the row moved between the id scan and the lock, and transaction/1 turns that into {:ok, execution} — the same shape as a retirement. The tests only see [] because the outer query filters, so the race is untested and invisible. #1746's scheduler will want to log how many obligations it wrote off; as written it has to re-read state to find out. _unsafe_recover_submissions/2 has the same shape, so this is consistency rather than regression — a distinguishable skip return would help both.

4. Not addressed, and you did not claim it was: record_sandbox_usage/2 under the advisory lock. The usage_events insert plus the PostHog mirror still runs inside the transaction holding pg_advisory_xact_lock(4316, …), with no line in the docstring saying so. Cheap and non-blocking, as I said last time.

---

Stack

The note stands and you have it: #1745 (84cb45f3) is still based on feat/durable-turn-deadlines but forked below this head, and #1746 onward chain from it. Nothing here contradicts what the children assume — this commit adds two functions and one option and changes no signature they call — but ADR 0046 now puts the ageing sweep in the deadline supervisor's row, so #1746 owes a scheduler for _unsafe_retire_unresolved/2 as well as the worker. That is the obligation this approval is resting on.