Enforce journal deadlines independently of conversation actors

[jhgaylor]

A blocked conversation callback must not prevent a turn deadline from expiring. This adds a supervised coordinator with separate expiration and termination pools, task-local timeouts, and persisted recovery that never replays an uncertain provider write. It terminates only the journaled provider session after an ownership-checked claim.

Stacked on #1745. Draft; public bounded execution remains disabled. Trusted session identity, durable deadline events, and the remaining actor/sandbox lifecycle integration still block activation. Local task death does not prove remote termination or stopped billing.

Validation: full precommit passes 4,671 tests and 6 doctests, zero failures, including 38 worker/journal regressions. Tests cover blocked actors, saturated termination slots, competing coordinators, timeout/restart, shutdown, and the recorded provider target. No live provider operations occurred. ADR 0046 and the evidence artifact preserve prior shutdown failures and their corrections.

Tracks #1732. No SDK surface changes beyond the parent; sdk-no-release applies.

Part of #1732 — held. Superseded on main by #1773–#1793; the residual hunks are being re-cut as focused PRs tracked by #1864, which keeps these frozen until each has a replacement or an explicitly linked deferral. Do not close: they are the reference for that mapping.

[BinaryBourbon]

Review: the coordinator shape is right; it should not be on by default, and it has no way out of uncertain

The separation this PR is built on is the correct one — a blocked provider call in the termination pool cannot starve expiry, each task carries its own :timer.kill_after, async_nolink under Fountain.TaskSupervisor is the supervised form this repo requires, and terminate/2 gives short journal writes a bounded grace before a brutal kill. _unsafe_due_deadlines/2 and _unsafe_ready_terminations/1 splitting the old _unsafe_due/2 is a clean way to express it.

Requesting changes on two points.

1. It runs by default, and there is no runtime switch

defp execution_deadline_children do
  if Application.get_env(:fountain, :execution_deadline_worker_enabled, true),
    do: [Fountain.Conversations.ExecutionDeadlineWorker],
    else: []
end

The default is true and only config/test.exs turns it off, so every prod and dev node starts a coordinator that runs two queries per second against turn_executions forever. Nothing in config/runtime.exs reads it, so an operator cannot turn it off without a rebuild.

Two asks:

• Gate it on execution limits actually being configured — the rest of this stack is careful to ship inert (enforced_controls/1 returns [], ADR 0046 says "no API or scheduler activates bounded turns yet"), and a per-second poll on every replica is the one part that is not.
• Give it a runtime.exs key regardless, so it can be turned off in production without a deploy. That also needs a row in docs/configuration.md or config_reference_test will fail.

A 1 s tick is also much tighter than anything comparable here (SandboxReaper, CreditExpirer). Deadlines are absolute and already durable, so a 5–15 s tick would cost nothing in correctness and a lot less in load. If 1 s is deliberate, a sentence saying why would help.

2. Every failed provider termination is permanent

defp call_terminator(terminator, attempt) do
  terminator.(attempt)
rescue
  _ -> {:error, :termination_unconfirmed}
catch
  _, _ -> {:error, :termination_unconfirmed}
end

That lands the row in uncertain, and _unsafe_ready_terminations/1 selects state == "ready" only, so nothing ever retries it. _unsafe_recover_submissions/2 only moves submitted → uncertain, which makes the absorbing state easier to reach, not easier to leave. The 10 s :timer.kill_after does the same thing to any provider call that outlives it.

The downstream cost is on #1744, where I have written it up in full: a row in uncertain keeps _unsafe_fenced?/1 true, refuses every new turn on the conversation, and makes reset_sandbox/2 return :sandbox_mid_turn forever. So on this branch a single slow terminate_session bricks a conversation and its shared home with no operator lever. A bounded retry with backoff, or an age after which the obligation is written off, would close it; either way it belongs in this PR, because this is the component that manufactures the state.

Smaller

• @providers maps all four providers, but ExecutionTransport in #1748 refuses anything but "sprites" and #1749 rolls back :provider_not_supported for the rest, so three of those four entries are unreachable. Either drop them or say in the moduledoc that they are staged for later — as written it reads like runner/E2B/Daytona are supported.
• start_candidates/3 does Enum.take(ids, @pool_size - used). It is safe today because used can never exceed @pool_size, but a negative count takes from the end of the list rather than returning [], which is a nasty way to find out if that invariant ever changes. Enum.take(ids, max(@pool_size - used, 0)) costs nothing.
• @batch_size 100 against a guard of limit in 1..100 — fine, but exactly at the boundary. Worth a comment so nobody bumps one without the other.

[jhgaylor]

Force-pushed 2462d426. Rebased onto the re-cut #1745 and both blockers addressed.

It is off unless an operator asked for it. The child list defaulted to true, so every prod and dev node polled turn_executions twice a second forever, and nothing in runtime.exs read the key — it could not be stopped without a rebuild. Now:

• default off, with runtime.exs deriving the default from whether FOUNTAIN_EXECUTION_LIMITS sets a host ceiling;
• FOUNTAIN_EXECUTION_DEADLINE_WORKER overrides either way, so an incident can stop the poll without a deploy;
• the tick is 5 s and configurable via FOUNTAIN_EXECUTION_DEADLINE_INTERVAL_MS. A deadline is absolute and durable, so lateness costs precision, not safety.

Both variables have docs/configuration.md rows — config_reference_test requires that, and it passes.

A failed termination is written off rather than fenced forever. This was the one I cared about: call_terminator/2 rescues every failure to {:error, :termination_unconfirmed} and _unsafe_ready_terminations/1 selects only "ready", so nothing looked at the row again. One slow terminate_session fenced its conversation and its shared home permanently and took reset_sandbox/2 with it. The recovery tick now also calls _unsafe_retire_unresolved/2 (added on #1744): past an hour, awaiting_identity and uncertain retire to stopped with last_error intact.

I did not add a retry, deliberately. One persisted attempt authorizes exactly one provider write. Re-arming a lost attempt would either replay an operation whose outcome is unknown, or require assuming terminate_session/3 is idempotent across a session that may already have been replaced. Giving up on a cleanup Fountain cannot confirm is the smaller claim, and ADR 0046 now records the rejected alternative alongside the sweep. If you want the retry instead, that is a real argument — but it needs the no-replay invariant relaxed explicitly rather than by accident.

Smaller items. @providers is now %{"sprites" => :sprites}: the other three were unreachable (ExecutionTransport refuses them, admission rolls back :provider_not_supported) and the map implied support that does not exist. Enum.take(ids, max(@pool_size - used, 0)), so used <= @pool_size staying true is not load-bearing.

One thing I had to restore. The re-cut of #1745 dropped 7db76444 with the rest of that branch, and this worker needs it — Managoat.Sandbox.terminate_session/3 arrives in managoat_sandbox 0.3.0, and managoat_runner 0.2.0 pins sandbox ~> 0.2.0, so the bump is the whole set. It is back as its own commit below the coordinator, because it moves the ACP stack (acp 0.4.0, runtimes 0.4.1, runner 0.2.2) and that is not the coordinator's change. Worth your eye: that is the largest thing in this PR by risk.

Verification. Five new regressions. Reverting the age-out wiring fails 2; reverting the default-off fails 1. Fountain.Application.execution_deadline_children/0 is @doc false public so the supervision test asserts on the list the supervisor actually reads rather than on the config it was just handed. Full core suite on the bumped dependency set: 6 doctests, 4,988 tests, 0 failures, 2 skipped. mix format --check-formatted, mix compile --warnings-as-errors, mix credo --strict (7,290 mods/funs, no issues), okf validate decisions valid, all three prose gates clean (docs-style.py's two findings are pre-existing in files this PR does not touch).

Moving on to #1747.

[BinaryBourbon]

Review: both asks answered, and the ageing exit is the right shape

Re-reviewed at 033eb63. The two blocking points are closed and closed in the way I would have argued for.

It is off unless an operator asked. execution_deadline_children/0 defaults to false, runtime.exs derives the default from whether FOUNTAIN_EXECUTION_LIMITS gives a host ceiling, FOUNTAIN_EXECUTION_DEADLINE_WORKER overrides either way, and both variables have rows in docs/configuration.md. The tick is 5 s and configurable. The two supervision tests assert the child list itself rather than a flag, including the unset case, which is the assertion that would actually catch a default flipping back.

A failed termination is written off, not fenced forever. _unsafe_retire_unresolved/2 on the recovery tick, past an hour, stopped with last_error intact. The reasoning for ageing out rather than retrying — one persisted attempt authorizes exactly one provider write — is right, and it is now in ADR 0046 as a rejected alternative rather than as an omission. Of the three new regressions, "the same attempt is never submitted to the provider twice" is the one that earns its keep: it proves the no-retry rule instead of restating it.

@providers down to Sprites and Enum.take(max(_, 0)) both done.

One thing I would still like written down somewhere

The default derives from the host ceiling only, but users.execution_limits is a real per-account ceiling with its own writer (Accounts.update_execution_limits/3). An operator who gives one account a ceiling and sets no host ceiling gets bounded turns with no coordinator on any node: nothing expires a deadline, and nothing ages an obligation out — the dead-end this PR just closed, reachable by configuration instead of by a slow provider.

The docs row does say "Set true if you give an account a ceiling but no host ceiling", so the burden is entirely on the operator reading that sentence. Cheapest closure later is to make it not a sentence: refuse at admission (or warn at boot) when an allowance can resolve for an account and the worker is off. Not blocking — enforced_controls/1 is still [], so nothing can admit a bounded turn today — but it is the one path left where the fence outlives its sweep.

Smaller, take it or leave it: @batch_size 100 and _unsafe_retire_unresolved/2's limit in 1..100 still sit exactly on each other's boundary with nothing saying so.

Approving.