Skip to content

Design memo: one owner per conversation lifecycle verb #2175

Description

@jhgaylor

The 2026-09-14 complexity review rated the Conversations subsystem (about 20,500 lines, the two most-churned files in the repo) the biggest engineering priority in the codebase and named the defect: the lifecycle verbs start, wake, reapply, interrupt, terminate, resume, retire and stop have no single owner. This is the design memo for giving each verb one owner. It was written read-only against main at 74ad641f, re-run 2026-09-15 against 4224258d after the #2176 credential stack landed (the re-run added one interacting rule and re-mapped every conversations.ex citation; nothing else moved), and filed as eleven stage issues under stack:2175 on 2026-09-15. Nothing is implemented. The full memo is at the bottom, under the fold.

Headline findings

  • The verb table the review hand-assembled was wrong in four places. DetachedRequest.resume_prompt/3 is a pure JSON builder, redaction.ex defines no terminate, start_or_resume/attach/release were missing, and execution_guard.ex (29 _unsafe_ functions) was omitted although it defines the durable half of five verbs.
  • Interrupt has four ad-hoc doors into ExecutionGuard._unsafe_interrupt; the plan reduces that to two owned ones.
  • resume is four unrelated functions sharing a word, and retire is per-thing. Neither gets a single owner; both get renamed by meaning.
  • execution_* is three concerns in five files (policy plus allowance; journal plus scheduler; transport). No rename while ADR 0046 stays Proposed.
  • No fifth from_state/into_state pair. The new owners act on ids, rows and a handle, not server fields.

Ownership table

Verb Owner Server keeps
start new Conversations.Launch nothing
wake new Conversations.Wake nothing
reapply existing Conversations.Reapply handle_call({:refresh_configuration, ..}), the :reapply_prompt continue
interrupt new Conversations.Interruption interrupt_turn/1
terminate + release new Conversations.Termination (absorbs Lifecycle.retire_terminated_sandbox) terminate_machine, terminate_kept_machine, finish_termination, OTP terminate/2
resume stays split by meaning resume_acp_connection
retire stays per-thing retire_bounded_turn delegate
stop stop_cotenants joins MachineEvents beside its receiver stop_acp_peer delegate

PR plan

Same shape as #1369: function families, each under ~400 lines, the thirteen conversation_server_*_test.exs files unchanged, a defdelegate left behind so no caller moves. Chains 2→3→4→5 and 7a→7b→7c; 1 and 6 land independently. The size pin is lowered only in PR 8.

# Issue Moves Depends on
1 #2209 Termination client, Lifecycle.retire_terminated_sandbox and the lifecycle audit (audit_lifecycle out of the server, four actions into @must_audit) → termination.ex none
2 #2210 Wake a: probe, suspended resume, resume_and_wake, start_conversation_serverwake.ex none
3 #2211 Wake b: wake_conversation_for, fresh sandbox, cotenant move → wake.ex 2
3b #2212 The cotenant cast has one sender, in MachineEvents none
4 #2213 Interruption client, wake_for_interrupt, _unsafe_interrupt_turn + _unsafe_idle_interrupted_turninterruption.ex 3
5 #2214 Journal doors call Interruption/Termination instead of ExecutionGuard; ADR 0046 status amended 1, 4
6 #2215 Reapply family → reapply.ex none
7a #2216 Launch: start_or_resume_conversation, resume_channel, channel rotation → launch.ex none
7b #2217 Launch: fresh start_conversation, reservation, fail_initial_start; one child spec builder 7a (2 for the spec)
7c #2218 Launch: attach_conversation, check_attachable, check_attach_capacity 7b
8 #2219 Lower the pin; refresh Fountain.Audit moduledoc all

Independent and startable now: 1, 2, 3b, 6, 7a. Chains: 2→3→4→5 (5 also needs 1) and 7a→7b→7c. 8 last. Rules for every stage are in the comment of 2026-09-15 below.

Open decisions that need a human

  1. Should interrupting a dead server ever provision? Today it can (conversations.ex:5528-5530 under purpose: :interrupt) and no test exercises that arm. The cheaper answer reconciles the orphan row only. A behaviour change, so it is not in the plan.
  2. ADR 0046's status. It says "Proposed; not shipped" while the journal is on every verb's path, gated off. Amend before PR 5, which makes the journal doors explicit.
  3. Where the machine lives. ADR 0023 keeps the handle in the conversation server with no SandboxServer. A machine-owner process is the other credible home for the retirement and deadline rules; choosing it would change PRs 1 and 3.
  4. Whether Lifecycle keeps its name once it holds only the idle/ceiling policy and park/destroy.
  5. Retarget the seven controller and worker call sites, or keep the delegates. Retargeting changes names docs/ cites.

Second evaluation, against 5da458f7

Two sessions ran this evaluation independently; the second wrote against the remove-avatar-generation tree (main at 5e499e49 plus the avatar removal), so its conversations.ex line numbers past about 1,000 are up to 133 higher than on 74ad641f. Both memos are under the fold below. They agree on the owners, on no fifth from_state/into_state pair, on leaving execution_* alone, on the delegates and on the chains. What the second one adds, each spot-checked against 74ad641f:

  • The four lifecycle audits are recorded by a private GenServer client function, not the context. audit_lifecycle/5 at conversation_server.ex:428 records conversation.prompted, interrupted, terminated and released, and none of the four appears in audit_guardrail_test.exs's @must_audit. That is the CLAUDE.md rule ("mutations audit in the context") already broken, and it proposes a Conversations.LifecycleAudit shared by Interruption, Termination and the prompt door, landed in PR 1 with the four actions added to @must_audit.
  • release is a ninth verb (ADR 0023's terminate-and-keep) with its own client half, journal door and caller in team.ex; the first memo folded it into Termination without naming it.
  • The Horde child spec is built three times (conversations.ex:3637, :5806, rehydrator.ex:139); its PR 7b collapses them to one builder in Launch.
  • last_resumed_at has two writers (conversations.ex:5777 and conversation_server.ex:1122); Wake should keep it.
  • create_conversation (conversations.ex:931) is Team's door with no gate and no audit, and _unsafe_sandbox_kept_on_terminate? has no production caller.
  • No cross-tenant 404 test exists for prompt, interrupt, terminate or release; the isolation tests cover GET and DELETE only.
  • Three more open decisions: whether the terminate fence, admin reap, destroy_home and Deletion's teardown join Termination as a second tranche (its own tracker); whether the controllers retarget to Conversations.* or keep ConversationServer.* as the door, given Mimic rejects a pin on send_prompt/4; and the *_after_retirement names, which mean "after the journal interrupt" and mislead.

Where the two disagree, and a suggested resolution:

Point Memo against 74ad641f Memo against 5da458f7 Suggest
execution_* three concerns (policy + allowance, journal + scheduler, transport) two (policy, journal with two drivers) Immaterial to the plan; neither renames or merges
Interrupt row halves (_unsafe_interrupt_* in conversations.ex) stay in the context move into Interruption Move them; the owner should be the only writer of the interrupted turn row
Cotenant cast (stop_cotenants, tell_cotenants) moves with Wake b (PR 3) its own PR 5 into MachineEvents, independent Separate PR; it never touches the server and lands alone
Child specs stay three collapse to one in PR 7b Collapse; three copies of one shape is the ownership defect in miniature
Lifecycle audit left where it is LifecycleAudit in PR 1 Do it in PR 1; it is the one CLAUDE.md rule the current shape already violates

Memos

The first memo's appendix links (appendix-verb-inventory.md, appendix-tests.md) point at files beside it under work/conversation-lifecycle-ownership/ on the reviewer's machine, not in the repo. Paths are relative to apps/fountain/lib/fountain/ unless prefixed; server.ex is conversations/conversation_server.ex and ctx.ex is conversations.ex.

memo.md, against 74ad641 (about 2,500 words)

Evaluation memo, 2026-09-14; read-only. Every file:line is against
origin/main at 74ad641f. Paths are relative to apps/fountain/lib/fountain/
unless prefixed; server.ex = conversations/conversation_server.ex,
ctx.ex = conversations.ex. Appendices:
verb inventory, tests.

1. Verb inventory

The prompt's table was hand-assembled and is wrong in four places (detail in
the appendix): detached_request.ex:191 resume_prompt/3 is a pure JSON
builder, not a verb; redaction.ex has no terminate, only an ETS owner's
start_link/1 (redaction.ex:39); start_or_resume_conversation
(ctx.ex:3202), attach_conversation (ctx.ex:4557) and
release_conversation (server.ex:367) were missing; and the largest
omission is conversations/execution_guard.ex (29 _unsafe_ functions),
which defines the durable half of interrupt, terminate, release, retire and
expire (_unsafe_interrupt :91, _unsafe_release_parent :121,
_unsafe_complete :428, _unsafe_expire :702, _unsafe_retire_unresolved
:872).

Verb Definitions today Callers (every door)
start ctx.ex:3535/3540/3545 (three clauses), :4557 attach_conversation, :3202/3204/3232 start_or_resume_conversation, :5804 start_conversation_server; turn_machine.ex:1289 start_acp_peer, :812 start_metrics; provision_watchdog.ex:35; execution_transport.ex:39 _unsafe_start API conversation_controller.ex:556, openai_controller.ex:500, agui_controller.ex:208; team.ex:238/389/764, team/schedules.ex:303; queue drain sandbox_queue.ex:373. The rehydrator bypasses all of them and calls Horde.DynamicSupervisor.start_child itself (rehydrator.ex:139-145). No LiveView caller
wake ctx.ex:5506 wake_conversation, :5510 wake_conversation_for (purpose :work/:interrupt), :5617 wake_for_interrupt, :5745 wake_suspended_sandbox, :5769 resume_and_wake, :5823 create_fresh_sandbox_and_start, :5961 move_cotenants, :6047 mark_old_sandbox_terminated server.ex:105 (send_prompt on a registry miss) and :203 (interrupt_dead); every prompt door reaches it through send_prompt (conversation_controller.ex:832, openai_controller.ex:517, agui_controller.ex:225, team.ex:354, ctx.ex:5335)
reapply ctx.ex:1286, :1379 do_reapply_conversation, :1319 announce_reapply; reapply.ex owns the rebuild rule and update_identity/4 (:216); server :406 refresh_configuration, handle_call :1397-1420, handle_continue({:reapply_prompt,..}) :590 conversation_controller.ex:675 only; the server reaches the continue from kick_turn (server.ex:2140)
interrupt client server.ex:179, :202 interrupt_dead; actor :2078/2080 interrupt_turn, handle_call(:interrupt) :1305/1309; durable execution_guard.ex:91; rows ctx.ex:2328 _unsafe_interrupt_turn, :2422 _unsafe_idle_interrupted_turn conversation_controller.ex:909; internally from both terminate arms (server.ex:1898, :1913), MachineEvents.gone (:1472), {:execution_retired, id} (:1513). ExecutionGuard._unsafe_interrupt has four doors: server.ex:183, :315, :617, ctx.ex:1636
terminate client server.ex:309, :323 terminate_after_retirement; actor handle_call({:terminate_conv,..}) :1365, :1880 prepare_termination, :1893 terminate_kept_machine, :1912 terminate_machine, :1923 finish_termination; OTP server.ex:2022, execution_transport.ex:361, execution_deadline_worker.ex:133; rows ctx.ex:1233, fence :4108 conversation_controller.ex:874; team.ex:305; accounts/deletion.ex:258; ctx.ex:198 (reaper, system:sandbox_reaper), :1646 (delete cascade), :4081 (system:home_reset)
resume four unrelated things: connection.ex:110 (reuse an idle ACP peer), ctx.ex:3236 resume_channel, :5769 resume_and_wake (provider resume), :5334 resume_after_request server.ex:2333 from run_turn; start_or_resume_conversation; wake_suspended_sandbox; answer_permission_request (:5086)
retire bounded_turn.ex:39, lifecycle.ex:343/345 retire_terminated_sandbox, execution_transport.ex:315/317 (private), execution_guard.ex:492 retire_orphan, :872, ctx.ex:6047, _unsafe_retire_home server.ex:2073interrupt_turn, drop_connection (:2405), abandon_session_restart (:2369), :1513; server.ex:334; transport timers; worker tick (execution_deadline_worker.ex:98-113)
stop lifecycle.ex:570 stop_cotenants, connection.ex:278/280 stop_peer, execution_deadline_worker.ex:203 stop/2, {:stop, :normal, _} in ~15 server paths lifecycle.ex:459/553 and ctx.ex:6032 tell_cotenants (two senders of one cast, received at machine_events.ex:67); connection.ex:203/246, reattachment.ex:155/182, server.ex:2479
release (ninth verb, ADR 0023's terminate-keep-machine) server.ex:367, handle_call(:release_conv) :1379/1383, ctx.ex:1189execution_guard.ex:121 team.ex:691 open_fresh_conversation

2. Interacting rules

In every case the row-level decision is made once, and the trigger and the
consequence are re-derived at each door.

  1. Wake during interrupt. interrupt/2 first retires any bounded
    journal (server.ex:183), then on a registry miss calls
    wake_for_interrupt (:202), which is wake_conversation_for with
    purpose: :interrupt (ctx.ex:5510-5527). The purpose only skips the
    allowance check; the :create_new arm stays reachable, so interrupting an
    orphan whose sprite is gone provisions a new sprite (ctx.ex:5823).
    The :interrupt call then queues behind handle_continue(:provision) and
    times out to {:error, :provisioning} (server.ex:143-160). Three
    places: guard, purpose flag, server call. No test covers :create_new
    under :interrupt.
  2. Terminate during provisioning. Four independent handlers: the live
    call times out behind provision (server.ex:143); the no-server path
    writes rows and fences (server.ex:326-335, lifecycle.ex:345); the
    watchdog fails rows then kills the server (provision_watchdog.ex:57-100);
    and the provision continue itself now recognises :retired,
    :sandbox_reset_pending and :configuration_changed after the provider
    call (server.ex:951-967, fix(conversations): handle retirement with other validation errors #2122). The row guard is single
    (ctx.ex:359 prevent_sandbox_revival inside update_sandbox), which is
    why Handle retired-sandbox update rejection without crashing conversation actors #1766 could be fixed at all; the four consequences are not.
  3. Reapply against a live turn. Refused at four layers: the context
    refuses running (ctx.ex:1543 assert_reapplicable); the server refuses
    refresh_configuration with a turn (server.ex:1403); admission compares
    configuration_revision and, since fix(conversations): refuse fenced prompts before configuration reload #2129, checks the journal before
    answering :configuration_changed (ctx.ex:1868-1880); and the actor
    rebuilds itself through {:continue, {:reapply_prompt, …}}
    (server.ex:2140, :590). The revision is one rule; the four checks are
    four places that must agree on what "live" means.
  4. Retire with a cotenant on the shared sandbox (ADR 0023). One decision
    function, _unsafe_sandbox_held_by_other? (ctx.ex:102), inside one
    fence (ctx.ex:4152-4155, answers :sandbox_kept). But the fence is
    reached from three callers with three different follow-ups:
    terminate_kept_machine (server.ex:1893), retire_terminated_sandbox
    (lifecycle.ex:345-380), and the reaper's _unsafe_reap_sandbox
    (ctx.ex:174-200) which does not use the fence at all and decides by
    whereis/1. Park and destroy use a fourth predicate, busy_elsewhere?
    (lifecycle.ex:286, server.ex:1947). Cotenants are told by two senders
    of the same cast (lifecycle.ex:570, ctx.ex:6032).
  5. Deadline expiry crossing a reattach (ADR 0046). The provision
    continue retires any journal before reattaching (server.ex:617-621):
    a bounded journal means the actor stops instead of reattaching. Expiry
    itself has five triggers that all land in ExecutionGuard: the worker
    tick (execution_deadline_worker.ex:98), the transport's :deadline
    timer (execution_transport.ex:203, "a wakeup hint"), the per-message
    actor gate (server.ex:1500, bounded_turn.ex:26), the terminal-stage
    writer (execution_guard.ex:715), and actor completion (fix(conversations): consult execution journal during actor completion #2133,
    _unsafe_end_actor_turn :360). Handled once at the row; five wake-ups.
  6. Capacity queue draining into a start (ADR 0042). One path: the drain
    replays start_or_resume_conversation (sandbox_queue.ex:373), poked
    from update_sandbox (ctx.ex:249; ADR 0042 §5). The interaction is the
    error vocabulary: the queue hard-codes which lifecycle errors are
    transient (sandbox_queue.ex:58 @transient_errors), and
    Workers.TeamScheduleRun snoozes on the same list by copy. A new refusal
    from start or wake is terminal to the queue until someone remembers.

Same shape, not in the prompt: the :already_started hand-off is written
three times (ctx.ex:5538, :5570, :5891).

3. Ownership table

The #1369 test: the server keeps a wrapper only where what happens next is
the process's own (a handle, a timer on self(), an ordering against a row
update). Everything else is a function over explicit arguments.

Verb Owner Server keeps Reason
start new Conversations.Launch (fresh, attach, start_or_resume, resume_channel, reservation, fail_initial_start, channel rotation) nothing zero process state; Horde.start_child is a call, not the child's own act
wake new Conversations.Wake (probe, suspended resume, fresh + cotenant move, server start, wake_for_interrupt) nothing same; the three probe outcomes are one decision table today split across maybe_reuse_sandbox (:5641), probe_sandbox (:5680) and the case at :5528
reapply Conversations.Reapply (exists: takes reapply_conversation, do_reapply_conversation, announce_reapply, assert_reapplicable) handle_call({:refresh_configuration,..}), {:continue, {:reapply_prompt,..}} the reload orders against the actor's own configuration_revision and handle
interrupt new Conversations.Interruption (client interrupt/2, interrupt_dead, the journal call, audit) interrupt_turn/1 cancel/EOF/stop act on current_command, acp_peer, the quiet timer
terminate + release new Conversations.Termination (client halves, terminate_after_retirement, retire_terminated_sandbox from Lifecycle, _unsafe_finish_conversation_termination, _unsafe_release_conversation, audit_lifecycle) terminate_machine, terminate_kept_machine, finish_termination, OTP terminate/2 destroy needs state.handle; finish_termination orders the row write against the reply
resume stays split, renamed by meaning: Connection.resume stays; resume_channel → Launch; resume_and_wake → Wake; resume_after_request stays with detached requests resume_acp_connection the four are genuinely different verbs sharing a word
retire BoundedTurn.retire stays; retire_terminated_sandbox → Termination; transport's private retire stays; mark_old_sandbox_terminated → Wake retire_bounded_turn delegate each is a different thing being retired
stop stop_cotenants + tell_cotenantsMachineEvents (sender beside receiver); stop_peer stays Connection; worker stop stays stop_acp_peer delegate one cast, one module

from_state/into_state. No. The four pairs (connection.ex:48,
output.ex, pending.ex, turn_machine.ex:119) exist because those families
own fields of the state map. The new owners act on conv_id, rows and a
handle; the actor halves left in the server already go through the
TurnMachine and Connection pairs (server.ex:2091-2097, :2405-2418). A
fifth pair would hand the lifecycle owner state it does not own.

4. The execution_* family

Three concerns in five files, not five and not one:

  • Policy and its storage. execution_limits.ex (pure typed ceilings,
    also read by accounts/user.ex and config/runtime.exs) and
    execution_allowance.ex (the saved allowance schema). Referenced from
    ctx.ex, turn_launch.ex, connection.ex. Stays; not a lifecycle verb.
  • The durable journal and its scheduler. execution_guard.ex (1,238
    lines, 31 _unsafe_ functions) and execution_deadline_worker.ex. This
    is the durable half of interrupt, terminate, release, retire and expire,
    and today the server reaches into it directly at server.ex:183, :315,
    :617 and ctx.ex:1636. The ownership table makes Interruption and
    Termination the only callers of _unsafe_interrupt and
    _unsafe_release_parent; the guard keeps its name and file.
  • The transport. execution_transport.ex, a GenServer owning one Sprites
    command. Stays; its retire and terminate/2 are the process's own.

No rename now: ADR 0046 is Proposed and the feature is off unless
FOUNTAIN_EXECUTION_LIMITS is set (decisions/0046:17-30,
application.ex:263), so churn here buys nothing before #1864 settles
activation. What changes is the call graph: four ad-hoc doors into the
journal become two owned ones.

5. Migration path

Same shape as #1369: function families, each PR under ~400 lines, the
thirteen conversation_server_*_test.exs files (8,122 lines) unchanged, a
delegate left behind so no test's call site moves. The pin is 2549 against a
file of 2493 (conversation_server_size_test.exs:59); every step holds or
lowers it, and only the last PR lowers the pin.

# Moves Files Proves unchanged Depends on
1 Termination client: terminate_conversation, terminate_after_retirement, release_conversation, audit_lifecycle (~110 lines) + Lifecycle.retire_terminated_sandbox (~40) server.ex, lifecycle.ex, new termination.ex termination_client_test, termination_fallback_test, termination_fence_policy_test, termination_actor_fence_test, termination_cluster_test, release_fence_test, release_fence_actor_test, conversation_audit_test "conversation.terminated", audit_guardrail_test none
2 Wake, part a: maybe_reuse_sandbox, probe_*, wake_suspended_sandbox, resume_and_wake, start_conversation_server (~170) ctx.ex, new wake.ex conversations_wake_test :62-207, conversations_start_test :15-55, wake_policy_test, saved_allowance_wake_test none
3 Wake, part b: wake_conversation_for, create_fresh_sandbox_and_start, mark_old_sandbox_terminated, move_cotenants + follow/strand/tell_cotenants (~230); stop_cotenants joins MachineEvents ctx.ex, wake.ex, lifecycle.ex, machine_events.ex conversations_wake_test :225-393, wake_race_test, conversation_server_shared_sandbox_test, machine_gone_binding_test, lifecycle_actions_test 2
4 Interruption client: interrupt/2, interrupt_dead, wake_for_interrupt (~60) server.ex, ctx.ex, new interruption.ex conversation_controller_test :1202-1230, interruption_admission_isolation_test, bounded_lifecycle_test :102, conversation_server_test interrupt cases 3 (wake_for_interrupt calls the private wake_conversation_for)
5 Journal doors: server.ex:183/315/617, ctx.ex:1636 call Interruption/Termination instead of ExecutionGuard (~30, net zero) server.ex, ctx.ex bounded_lifecycle_test, execution_guard_test, conversation_server_provision_retirement_test 1, 4
6 Reapply: reapply_conversation, do_reapply_conversation, announce_reapply, reapply_*, assert_reapplicable (~330) ctx.ex, reapply.ex conversation_reapply_test, inference_reapply_test, reapply_check_test, reapply_fingerprint_test, conversation_server_refresh_test, legacy_wake_reapply_test, audit_guardrail_test :74 none
7a Launch: start_or_resume_conversation, resume_channel, channel rotation helpers (~200) ctx.ex, new launch.ex conversation_controller_test create/channel cases, channel_allowance_lock_order_test, sandbox_queue tests, team_test, openai/agui controller tests none
7b Launch: start_conversation fresh clause, reserve_initial_conversation, fail_initial_start, home_or_new (~300) ctx.ex, launch.ex conversations_start_test :57-310, launch_commit_boundary_test, launch_credential_set_test, sandbox_mode_test, initial_start_failure_test 7a
7c Launch: attach_conversation, check_attachable, check_attach_capacity (~150) ctx.ex, launch.ex attach_test, sandbox_mode_test, orphaned_home_test 7b
8 Lower the pin to the file's real length; refresh Fountain.Audit moduledoc and @deliberately_silent names size test, audit.ex the pin test all

Chains: 2→3→4→5 and 7a→7b→7c; 1 and 6 land independently. Every PR leaves
defdelegates behind for the public names the controllers, team.ex,
deletion.ex and the queue call, so no caller moves with a definition. Three
more risk rows (the pin, the prompt-free child spec, the actor vocabulary)
are in the appendix.

6. Risk table

Invariant Where a move can break it Test today No test
Tenant scoping: every user-facing fetch scoped; _unsafe_ only after an adjacent scoped fetch with a comment Wake's first fetch is _unsafe_get_conversation with a prose promise that callers scoped (ctx.ex:5511-5515); moving it to a module with a public name invites a direct caller termination_fence_policy_test:75 (cross-tenant fence), conversations_start_test:240 (foreign vault) no guard that an _unsafe_ read is not the first fetch on a request path; the rule lives in CLAUDE.md prose only
Audit in the context, never inside a transaction, success only audit_lifecycle records after call_server returns (server.ex:428); reapply records after the source lock closes (ctx.ex:1305-1307); a move that puts either inside with_sandbox_lock loses the row silently audit_guardrail_test (@must_audit, arity list :244 names start_conversation, delete_conversation, the fence), conversation_audit_test :96-125, termination_client_test:46 the arity list omits terminate_conversation, interrupt, release_conversation, wake_conversation; nothing asserts "outside the transaction"
Fire-and-forget through Task.Supervisor, never Task.async Checkpoints.maybe_create_async uses Task.start (checkpoints.ex:80); the transport and worker use async_nolink (execution_transport.ex:343, execution_deadline_worker.ex:190); a moved wake that adds an unawaited Task.async for the cotenant broadcast would link to the request none no guardrail greps for Task.async; DataCase only waits for supervised tasks
In-flight turns finish after the credit gate closes (ADR 0030 d.6) the gate sits at four doors: start (ctx.ex:3562), attach (:4575), wake reuse and fresh (:5538, :5839), initial prompt via TurnMachine.gate (server.ex:1445, turn_machine.ex:1073); a move that adds check_spend to interrupt, terminate or resume_acp_connection refuses a turn the balance already funded ee/test/fountain/credits_enforcement_test.exs:90,100,116 (refused at the door) nothing asserts a running turn completes once the balance is zero
Lock order advisory → conversation → sandbox; fence commits before provider I/O; no provider call under a lock Termination and Wake both hold the order in comments (ctx.ex:4155, :3240, :3726); the in_transaction? refusals (server.ex:310, lifecycle.ex:477, ctx.ex:4109) must move with their verbs termination_attach_order_test, reset_admission_order_test, retirement_admission_order_test, lifecycle_fence_test:92, termination_client_test:56, launch_commit_boundary_test none for a new lock site
Interrupt before destroy (#2137); orphaned_at stays nil on an explicit terminate terminate_machine and terminate_kept_machine both interrupt first (server.ex:1898, :1913); a merged "teardown" helper that drops the connection first re-opens #1707 #2137's seven regressions in termination_actor_fence_test / turn_parent_actor_test
OTP terminate/2 order: redaction delete → key revoke → orphan (best-effort last) server.ex:2023-2033; a move that folds the orphan into Termination's row write runs it inside a transaction conversation_server_shutdown_revoke_test, orphan_binding_isolation_test
The queue's transient-error vocabulary matches the lifecycle's refusals sandbox_queue.ex:58; a renamed wake error becomes terminal to a queued run queue tests (sandbox_queue_*) no test that the list covers every retryable error start/wake can return

7. Open decisions that need a human

  1. Should interrupting a dead server ever provision? Today it can
    (ctx.ex:5528-5530 under purpose: :interrupt). The cheaper answer
    reconciles the orphan row only (ctx.ex:2543 _unsafe_orphan_turn). A
    behaviour change, so not in the PR plan.
  2. ADR 0046's status. It says "Proposed; not shipped"
    (decisions/0046:17), yet the journal is on every verb's path (interrupt,
    terminate, release, delete, provision) with the feature merely gated off.
    CLAUDE.md's rule cuts both ways; the ADR under-describes what is built.
    Amend before PR 5, which makes the journal doors explicit.
  3. Where the machine lives. ADR 0023 records that no SandboxServer
    exists and the advisory lock plus a query carries the role, "revisit if
    the handle and sprite env ever need to move into one owner"
    (decisions/0023:108-116). The table above keeps the handle in the
    conversation server. Confirm that stays the decision: a machine-owner
    process is the other credible home for rules 4 and 6 in section 2, and
    choosing it would change PRs 1 and 3.
  4. Whether Lifecycle keeps its name. After PR 1 and 3 it holds only the
    idle/ceiling policy and park/destroy. Reclaim would say what it does; a
    rename touches the reaper and three ADRs, so it is a separate call.
  5. Retarget the callers or keep the delegates. Retargeting the seven
    controller and worker call sites is one PR, but it changes names docs/
    cites.
memo-5da458f7.md, against 5da458f (about 2,550 words)

Design memo, 2026-09-14. Read-only; nothing here is implemented.

Base. Every file:line is against the working tree at 5da458f7
(main at 5e499e49 plus the avatar removal). Paths are relative to
apps/fountain/lib/fountain/; ctx = conversations.ex, srv =
conversations/conversation_server.ex, guard =
conversations/execution_guard.ex, lc = conversations/lifecycle.ex,
reaper = workers/sandbox_reaper.ex. origin/main moved to 74ad641f
during this evaluation (#2162, #2166 took 133 net lines out of ctx), so
ctx numbers past ~1,000 are up to 133 lower there. memo.md beside this
file is a sibling memo another session wrote against that commit; the last
section says where the two disagree.

Appendices A to C hold the per-definition tables (callers, state read and
written); D the interacting rules and the seven commits; E the #1369
history, execution_* and the tests.

1. Verb inventory

Verb Definitions today Same name, different thing?
start ctx:3598 (attach to a given sandbox) and ctx:3603 (provision) clauses of start_conversation; ctx:3260 start_or_resume_conversation (channel layer, every API door); ctx:4615 attach_conversation; ctx:1013 create_conversation (Team's door, no gate, no audit); ctx:5889 start_conversation_server. The Horde child spec is built three times: ctx:3695, ctx:5891, rehydrator.ex:139 Attach vs provision is real: attach writes idle, takes no reservation, and the server starts later on the wake path (ctx:4685). The three child specs are one shape
wake ctx:5591 wake_conversation and ctx:5702 wake_for_interrupt both call ctx:5595 wake_conversation_for; the purpose flag only skips the allowance preflight (ctx:5604). Called from srv:105 (registry miss) and srv:203 Real difference is precondition (running only) and return (pid vs row); code difference is one flag. Under :interrupt the :create_new arm still provisions (ctx:5671). No audit of its own; conversation.prompted is srv:119
reapply ctx:1344 (transaction, audit ctx:1364); Reapply helpers (reapply.ex:90/128/216/267); srv:406 refresh_configuration (a re-provision, srv:1413); srv:590 {:reapply_prompt} Four names for three phases and a stage called configuration. Reapply.mount_skills runs on every reattach (reattachment.ex:16), reapply or not
interrupt client srv:179 (audit srv:198), srv:202 interrupt_dead, srv:1303, srv:2076 interrupt_turn; journal guard:91 _unsafe_interrupt; row halves ctx:2386 and ctx:2480 Three writes named interrupt. For a bounded turn the journal ends the turn row before the actor hears (guard:992). No interrupted conversation status exists (conversation.ex:23). ctx:2601 _unsafe_orphan_turn writes the same turn status plus orphaned_at, which billing excludes (turn.ex:27)
terminate client srv:309; srv:323 (no server: update_conversation at srv:333, then lc:345 retire_terminated_sandbox); live srv:1363 → 1874 (fence ctx:4166) → 1910 terminate_machine or 1891 terminate_kept_machine → 1922 → ctx:1291; OTP srv:2020; audit srv:342 Two writers of status = terminated (srv:333 unguarded, ctx:1291 binding-checked). Terminate vs release (srv:367) vs lc:509 destroy (conversation stays idle) vs ctx:174 reap vs ctx:4129 destroy_home. ctx:4001 _unsafe_sandbox_kept_on_terminate? has no production caller
resume five meanings: connection.ex:110 reuse an idle peer (R1); detached_request.ex:191 + ctx:5392 open a new turn (R2); ctx:3294 resume_channel lookup plus labels (R3); ctx:5854 resume_and_wake provider resume (R4); reattachment.ex:202 + Rehydrator (R5) R1 and R5 are resumes; R2 and R3 borrow the word. R4 stamps last_resumed_at (ctx:5863) and the server stamps it again (srv:1119)
retire T1 a sandbox row: lc:345, ctx:4277, ctx:4530, the reaper. T2 a journal: bounded_turn.ex:39, execution_transport.ex:315, guard:492/872. T3 a name: srv:323 and ctx:1699 *_after_retirement mean "after the T2 journal interrupt" Two objects, one word; T3 misleads
stop connection.ex:278 stop_peer; lc:570 stop_cotenants (a cast also sent by ctx:6120); guard:992 (journal intent); execution_deadline_worker.ex:203 (the only provider stop); 24 {:stop, :normal, _} returns in srv Five different things
release (ninth verb) srv:367, srv:1377/1381, ctx:1247 → guard:121; caller team.ex:691 ADR 0023's terminate-and-keep

Two cross-cutting facts. The audit for prompted, interrupted,
terminated and released is recorded by srv:428-448 audit_lifecycle, a
GenServer client function, and none is in audit_guardrail_test's
@must_audit. No Task.async sits on any verb path; the watchdog,
rehydrator and checkpoints use bare spawn/Task.start
(provision_watchdog.ex:39, application.ex:54, checkpoints.ex:80).

2. Interacting rules

The pattern in all six: the row decision is made once, but the
trigger and the consequence are re-derived at every door.

  1. Wake during interrupt: several. An interrupt with no server must
    know the registry (srv:38), the sandbox row (ctx:5723-5741), the
    journal (srv:617 re-runs _unsafe_interrupt in the new server) and the
    reattach outcome (reattachment.ex:202-249, orphan at ctx:2601) before
    interrupt_turn (srv:2076-2109) has anything to act on. An interrupt
    on a provisioning server is refused by the 30 s timeout as
    :provisioning (srv:135-147), not queued.
  2. Terminate during provisioning: several. Four actors retire a
    pending/starting row: the no-server path (lc:343-377), the watchdog
    (provision_watchdog.ex:47-100), the reaper (reaper:118-150) and
    fail_initial_start (ctx:3786); the attempt defends itself at two
    checkpoints (srv:794-803, 934-976) plus provisioning.ex:914. The
    live arm cannot run during provisioning at all (srv:135-147). Only the
    row guard is single (ctx:359, ctx:257 claim_sandbox since fix(conversations): handle retirement with other validation errors #2122).
  3. Reapply against a live turn: several. "No running turn" is evaluated
    twice in one call (ctx:1598-1638, 1580-1596); the revision fence is
    in admission (ctx:1913-1926, where fix(conversations): refuse fenced prompts before configuration reload #2129 orders the journal fence ahead
    of it); the reload lives in four server places (srv:1395-1417,
    2135-2137, 590-595, 715-718). The revision is set once
    (srv:695-701).
  4. Retire with a cotenant (ADR 0023): several. "Is anyone else on this
    machine" is five queries (ctx:102, 2216, 2269, 2163,
    reaper:176/262); "keep or destroy" is decided in the fence
    (ctx:4222), reclaim (srv:1945 + lc:286), the reaper
    (reaper:220-345) and reset (ctx:4380-4390); cotenants are told by
    three senders (lc:570, ctx:6120, ctx:4466) in two shapes
    (machine_events.ex:13, :70). No lease column exists
    (sandbox.ex:28-69).
  5. Deadline expiry crossing a reattach (ADR 0046): once for the outcome,
    several for the trigger.
    Every terminal write resolves in
    guard:664 complete/3, so expiry and late completion cannot disagree
    (fix(conversations): consult execution journal during actor completion #2133). Retirement is decided by four callers (worker guard:702,
    per-message gate srv:1498 + bounded_turn.ex:39, the provision
    continue srv:617, orphan recovery guard:471) and the fence is read at
    seven doors (ctx:1922, 1970, 4390, 5603, guard:142, 183,
    rehydrator.ex:134). A bounded turn never reaches Reattachment
    (srv:617-621).
  6. Capacity queue draining into a start (ADR 0042): once for the
    mechanism, several for what it must know.
    One poke (ctx:339), one
    drainer, one fenced claim (sandbox_queue.ex:288); but the
    transient-error list (sandbox_queue.ex:58, copied at
    team/schedules.ex:206) must match seven producers (srv:135, 1273,
    ctx:1357, 1958, 3305, 5778, guard:137), and two "active" sets
    differ (quotas.ex:41 vs reaper:69).

The seven commits are the symptom: each taught one module another's
internals (#2137 is one line, srv:1911, teaching the destroy arm the
interrupt path's two-half ending). Appendix D has one paragraph per commit.

3. Ownership table

The #1369 test: the server keeps a wrapper only where what happens next is
the process's own. By that test the defect is the client halves in
srv (interrupt/2, terminate_conversation/2, release_conversation/2,
the miss branch of send_prompt/4, audit_lifecycle/5): they decide with
rows and the registry, not process state. Move those, keep the actor
halves, and make each owner the sole caller of its journal door.

Verb Owner Server keeps Reason
start new Conversations.Launch: both start_conversation clauses, attach_conversation, start_or_resume_conversation, resume_channel, reserve_initial_conversation, fail_initial_start, create_conversation, one child-spec builder init/1, handle_continue(:provision) zero process state; Horde.start_child is a call the child never sees
wake new Conversations.Wake: wake_conversation_for, the probe table (ctx:5613, 5723, 5751, 5766 are one decision in four functions), wake_suspended_sandbox, resume_and_wake, create_fresh_sandbox_and_start, the cotenant moves, wake_for_interrupt do_reattach/6 wake decides which machine; the reattach is the process's own
reapply Conversations.Reapply (exists) absorbs reapply_conversation, do_reapply_conversation, assert_reapplicable, announce_reapply handle_call({:refresh_configuration, _}), {:reapply_prompt} the reload orders against the actor's own configuration_revision and handle
interrupt new Conversations.Interruption: client interrupt/2, interrupt_dead, the guard:91 call, both row halves (ctx:2386, 2480) interrupt_turn/1, composing the existing Connection and TurnMachine pairs cancel, EOF and peer stop act on current_command, acp_peer, the quiet timer
terminate + release new Conversations.Termination: both client halves, terminate_after_retirement, lc:345, ctx:1291, ctx:1247; the fence ctx:4166 in a second tranche (§7) terminate_machine, terminate_kept_machine, finish_termination, OTP terminate/2 destroy needs state.handle; finish_termination orders the row write against the reply
resume by meaning: R1 stays Connection, R5 stays Reattachment, R3 → Launch, R4 → Wake, R2 stays DetachedRequest resume_acp_connection/5 five things share a word; one owner would be false
retire T1 retire_terminated_sandbox → Termination; the reset-family retirements (ctx:4277, 4530) stay with reset; T2 stays BoundedTurn/guard retire_bounded_turn/1 delegate two objects
stop stop_cotenants + tell_cotenantsMachineEvents, beside their receiver; stop_peer stays Connection; journal stop stays guard the {:stop, :normal, _} returns one cast, one module, one sender

Audit. audit_lifecycle/5 becomes Conversations.LifecycleAudit,
shared by Interruption, Termination and the prompt door, so the four
lifecycle actions sit under the context as CLAUDE.md requires and
audit_guardrail_test can name them.

from_state/into_state: no. The four pairs (turn_machine.ex:119,
pending.ex:50, connection.ex:48, output.ex:38) exist because those
families own fields of the state map. The verb owners act on an id, rows
and a handle; the actor halves that stay already use the TurnMachine and
Connection pairs (srv:2088-2095, srv:2422). A fifth pair would hand a
verb module state it does not own.

4. The execution_* family

Two concerns in five files, not five and not one. Policy is
ExecutionLimits (pure) plus ExecutionAllowance, a 62-line schema whose
every branch delegates to Limits (execution_allowance.ex:38/48). The
journal is ExecutionGuard, the only writer of turn_executions
(turn_execution.ex:7), with two drivers that exist so its decisions
survive a dead actor: ExecutionTransport per command,
ExecutionDeadlineWorker as a singleton. Inside the second concern the
duplication is real: ceilings resolved twice (ctx:3340-3358,
guard:1159-1172) and skipped a third time (execution_allowance.ex:48);
one deadline with two clocks (guard:65, execution_transport.ex:203/310);
"may this turn still run" answered three ways, split by lock cost rather
than concern (execution_transport.ex:310, guard:288, guard:339); the
task-pool boilerplate copied (execution_deadline_worker.ex:133/186,
execution_transport.ex:339/361); "Sprites only" encoded three times
(guard:54, execution_transport.ex:43, execution_deadline_worker.ex:58).

All five are under ADR 0046, still Proposed, and ship inert
(execution_limits.ex:58 returns []; the worker is off in
config/test.exs:42). The ownership table changes the call graph, not the
files: the guard is the durable half of interrupt, terminate, release and
retire, and its four ad-hoc doors (srv:183, 315, 617, ctx:1694)
become two owned callers plus the actor's own provision continue. No rename
or merge now; folding the drivers' boilerplate is gated on activation
(#1864).

5. PR plan

Same shape as #1369 (appendix E §A): one family per PR, under ~300 moved
lines plus the new module's own tests, the thirteen
conversation_server_*_test.exs files unchanged, and a defdelegate at
every public name so no caller moves with a definition. A new module needs
a .credo.exs:78-100 exempt_paths row and, if it publishes stages, an
events_test.exs:22 @sources line, the only guard edits #1369 made. Pin
2549 against 2500 lines; PRs 1, 4 and 8 shrink srv; the pin is lowered
once, in PR 9 (conversation_server_size_test.exs:18-31).

# Moves Files Proves behaviour unchanged After
1 Termination client srv:309-345, 367-378, 428-448 (→ LifecycleAudit), lc:343-375 srv, lc, new termination.ex, lifecycle_audit.ex the six termination_*_test files, release_fence(_actor)_test, conversation_audit_test:96-125, ee termination_billing_test; add the lifecycle actions to @must_audit
2 Wake a: probe ctx:5723-5800, suspended wake 5830-5873, server start 5889-5906 ctx, new wake.ex conversations_wake_test, wake_policy_test, saved_allowance_wake_test
3 Wake b: ctx:5595-5680, fresh path 5908-6010, cotenants 6044-6140, wake_for_interrupt ctx, wake.ex wake_race_test, conversation_server_shared_sandbox_test, machine_gone_binding_test 2
4 Interruption srv:179-207, ctx:2386-2430, 2480-2509 srv, ctx, new interruption.ex controller interrupt cases, interruption_admission_isolation_test, bounded_lifecycle_test, turn_machine_test 3
5 MachineEvents takes lc:570 and ctx:6120 lc, ctx, machine_events.ex lifecycle_actions_test, machine_gone_binding_test
6 Reapply absorbs ctx:1344-1656 ctx, reapply.ex the four *reapply*_test files, legacy_wake_reapply_test, audit_guardrail_test:74
7a Launch: channel layer ctx:3260-3420 ctx, new launch.ex controller create/channel cases, sandbox_queue tests, team_test
7b Launch: fresh clause ctx:3593-3820, reservation 3858-3905, the one child spec (ctx:3695, 5891, rehydrator.ex:139 collapse) ctx, launch.ex, rehydrator.ex, wake.ex conversations_start_test, launch_commit_boundary_test, launch_credential_set_test, sandbox_mode_test 7a, 2
7c Launch: attach ctx:4615-4780, the create_conversation door ctx, launch.ex attach_test, sandbox_mode_test, conversation_creation_seam_test 7b
8 Journal doors srv:183, 315, ctx:1694 call Interruption/Termination; srv:617 stays srv, ctx bounded_lifecycle_test, execution_guard_test, conversation_server_provision_retirement_test 1, 4
9 Lower the pin; optional caller retarget; rename *_after_retirement*_after_journal_interrupt size test, controllers the pin test all

Chains: 1→8; 2→3→4→8; 7a→7b→7c, with 7b also after 2. Independent: 5
and 6. PRs 2, 3, 5, 6 and 7 never touch srv.

6. Risk table

Invariant How a move breaks it Test today No test
Tenant scoping; _unsafe_ only after an adjacent scoped fetch Wake, Interruption and Termination take a bare id and read _unsafe_ first (ctx:5600, srv:430); as public modules they invite a direct caller cross_tenant_isolation_test:41-63 (GET, DELETE), conversation_controller_test:395/1508 no cross-tenant 404 test on prompt, interrupt, terminate or release. Credo's UnsafeCallOwnership checks any non-exempt path
Audit in the context, never inside a transaction, success only audit_lifecycle moves; inside with_sandbox_lock it loses rows silently; the reapply audit is after the lock by comment only (ctx:1362) audit_guardrail_test, conversation_audit_test lifecycle actions absent from @must_audit; nothing asserts Repo.in_transaction? is false at record time
Task.Supervisor for fire-and-forget a Task.async fan-out of the cotenant casts links to the request data_case.ex:55/73-95 drain no source guard
In-flight turns finish after the gate closes (ADR 0030 d.6) adding check_spend to interrupt or terminate; nothing on the finish path calls it (appendix B §4) ee credits_enforcement_test:90-130, termination_billing_test no positive test that a running turn completes at zero balance
The pin only ratchets down PRs 1, 4, 8 touch srv size test lowered only in PR 9
Lock order and the six in_transaction? refusals srv:310, lc:477, ctx:4107/4130/4167, deletion.ex:205 must move with their verbs the three *_order_test files, lifecycle_fence_test, launch_commit_boundary_test none for a new lock site
Interrupt before destroy (#2137) a merged teardown helper that drops the connection first reopens the orphaned_at gap termination_billing_test
Queue error vocabulary a renamed wake or start error is terminal to a queued run (sandbox_queue.ex:58) queue tests no test that the list covers every retryable error

7. Open decisions that need a human

  1. Should interrupting a dead server ever provision? It can today
    (ctx:5671 under :interrupt); reconciling the orphan row (ctx:2601)
    is cheaper. A behaviour change, so not in the plan.
  2. Where the public verbs live after the delegates. Retarget the
    controllers to Conversations.*, or keep ConversationServer.* as the
    door; Mimic rejects pin send_prompt/4 to the server (appendix E §C.7).
  3. Termination's second tranche. Whether the fence (ctx:4166), admin
    reap (ctx:174), destroy_home (ctx:4106) and Deletion's teardown
    join it. Needs its own tracker.
  4. ADR 0046's status. The journal is on every verb's path and the ADR
    says Proposed; amend before PR 8.
  5. last_resumed_at has two writers (ctx:5863, srv:1119). Wake
    should keep it.
  6. Lifecycle's name after PRs 1 and 5: policy plus park and destroy.

The sibling memo

memo.md (filed as #2175) agrees on the owners, on no
fifth from_state pair, on leaving execution_* alone, on the delegates
and on the chains. It differs in five places: three execution_* concerns
against two here; no LifecycleAudit; the interrupt row halves stay in
ctx; the cotenant cast moves inside Wake b; the child specs stay three.

🤖 Generated with Claude Code

https://claude.ai/code/session_01E8jqureiZocSQsfX87rXvc

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important planned work; not an emergency.area:conversationsConversation lifecycle, turns, streaming, recovery and orchestration.area:sandboxSandbox providers, machine identity, capabilities and lifecycle.lang:elixirElixir code or ecosystem.needs:decisionRequires a product, business or architecture decision before proceeding.scope:largeWork requiring broader investigation, design or coordinated changes; scope, not urgency.type:trackerCoordinates child deliverables; not a separate implementation task.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions