Skip to content

Record trusted sandbox provider identity - #1820

Merged
jhgaylor merged 1 commit into
mainfrom
feat/provider-identity-recording
Sep 11, 2026
Merged

jhgaylor merged 1 commit into
mainfrom
feat/provider-identity-recording

Conversation

@jhgaylor

@jhgaylor jhgaylor commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

A provider name alone cannot identify which sandbox instance was observed. Add an optional provider-issued ID and an internal writer that locks and rechecks tenant/provider/name, binds once, and refuses retired rows or conflicting identities. General sandbox changesets cannot set it; rollback refuses to discard recorded IDs.

Extracted from #1754 onto #1796: six files, +466 lines (278 test lines plus ADR/index). This is recording infrastructure. It adds no lifecycle caller or provider-write authorization; those need a separate durable operation protocol.

Validation: 35 focused tests, including two observed PostgreSQL lock interleavings; full mix precommit --seed 828329 passes (4,900 tests +6 doctests). Disposable-database migration proof covers legacy null backfill, refusal of populated rollback, empty rollback and reapply. OKF validation/index and staged secret scan pass. CI passes (run).

Part of #1864

@jhgaylor jhgaylor left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no changes requested, two things worth a decision.

The write path is careful in the ways that matter. Taking the advisory lock and FOR UPDATE before rechecking {user_id, provider, sprite_name}, refusing failed/terminated rows, treating a re-bind of the same id as a no-op rather than an error, and rolling back on a different id — that's the full set. Audit.record sitting outside the transaction is right (0013: a rescued audit insert doesn't survive a transaction, it aborts the enclosing one), and the docstring saying so means the next person won't helpfully move it inside. actor: "system:sandbox_identity" is in the closed vocabulary.

I verified the claim that general changesets can't set it: provider_instance_id is absent from Sandbox.changeset/2's cast list, so the only writer is _unsafe_bind/2. Good — that's the property the whole module rests on.

Two things I'd like you to decide rather than change on my say-so:

1. The down migration raises rather than rolls back. Once any row has an identity, down is a hard stop and the only way past it is manual SQL. That's a deployment hazard for a column whose contents are re-derivable — the provider still knows the id, and a later capture re-reads it. Refusing rollback to protect data you can ask the provider for again feels like the wrong side of that trade. If the intent is "don't silently lose evidence during an incident", a log line plus the drop gets you most of it without wedging a rollback.

2. Nothing calls it. SandboxIdentity has zero callers outside its own tests — I grepped. I'm not treating that as a blocker, because landing storage ahead of its caller is exactly how you keep the follow-up small and that's the point of this stack. It's also honestly labelled: ADR 0049 is Proposed and says outright that worker integration is separate, which is the discipline the ADR rules ask for and that this kind of PR usually skips. Just worth being deliberate that the unique index on (provider, provider_instance_id) is a constraint we're committing to before anything exercises it.

Minor: _unsafe_capture/2 and _unsafe_bind/2 have @doc but no @spec, unlike most of the surrounding context modules.

@BinaryBourbon BinaryBourbon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — detailed review in the comment above.

@BinaryBourbon BinaryBourbon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Storage plus one trusted writer, and the ADR is honest about what it is not — "this records an observation, it does not authorize a later provider mutation" is exactly the caveat the 2026-07 audit exists to force, and it is in the ADR body, the @moduledoc and the index row.

What I verified:

  • The identity cannot arrive from tenant input. provider_instance_id is absent from Sandbox.changeset/2's cast/3 list, so an ordinary update drops it silently. The test drives that both ways — before binding and after — with string and atom keys.
  • Provider I/O is outside every lock. _unsafe_capture/2 refuses an open transaction before calling Managoat.Sandbox.get/1, and the happy-path test asserts refute Repo.in_transaction?() from inside the stubbed provider call. _unsafe_bind/2 repeats the refusal so a direct caller cannot get around it.
  • The write is genuinely fenced. Advisory lock, then FOR UPDATE, then a recheck of tenant/provider/name and status, then a write only on nil. ^id re-binding is idempotent and does not re-audit; a different id is :provider_identity_changed. The lock test is the part that earns this — it drives a real PostgreSQL row-lock wait, confirms two distinct backends, waits on pg_blocking_pids, and asserts the post-wait recheck rejects. That is the test most people skip and then ship the race.
  • The partial unique index and the changeset constraint agree. create unique_index(:sandboxes, [:provider, :provider_instance_id], where: ...) produces sandboxes_provider_provider_instance_id_index, which is what unique_constraint([:provider, :provider_instance_id]) infers, and the retired-owner test proves the {:error, changeset} return survives the enclosing transaction rather than aborting it.
  • The audit follows the commit and records provider only — no id, no name. Correct under ADR 0013's "never record values."

Three things to fix or answer, none of which I would hold the merge for:

  1. 4316 is a bare literal here. It is @sandbox_lock_namespace in conversations.ex, whose own comment says the namespaces "must never be mistaken for one another." Taking the same namespace with the same phash2(sandbox.id) key is deliberate and correct — it is what serializes a binding against do_reset_sandbox/2 and turn admission — but that intent is now invisible at the third site. Please reference the named attribute, or add a comment saying which lock this is joining and why.

  2. unboxed_run has no precedent in this repo. SandboxIdentityLockTest is the first place that writes real rows outside the SQL Sandbox into fountain_test, which is shared across worktrees and CI partitions. The after blocks clean up, but Repo.delete!(Repo.get!(Sandbox, observed.id)) raises if the row is already gone and would mask the real failure while leaking the users. Wrapping the teardown in delete_all by id would make it unconditional. Worth a sentence in the module doc explaining why unboxed is unavoidable here, so the next person does not copy the pattern for a test that does not need it.

  3. The down migration is one-way in practice. It raises whenever any row carries an identity, and by design nothing ever clears one. So after the first bind in production, mix ecto.rollback past this migration fails permanently — including a rollback you want for an unrelated reason. Refusing to discard data is defensible; making the release irreversible is a separate decision. Is that intended? #1768 has the identical shape one PR over, which is why I am asking here rather than treating it as a nit.

decisions/index.md is refreshed in the same PR and okf validate is green, as the bundle requires. No caller yet is fine — the ADR says so explicitly and the tracker sequences integration separately.

jhgaylor added a commit that referenced this pull request Sep 11, 2026
Review of #1768 found the fence was a dead end. `update_sandbox/2` refused
every write to a fenced row, and its own docstring says "every sandbox status
change in the system goes through here" — which is true, and most of those
callers match `{:ok, _}`. Two reaper queries were filtered against this;
nothing else was. Reproduced all three before fixing:

  reap:         {:RAISED, MatchError}
  delete_agent: {:RAISED, MatchError}
  park:         {:RAISED, MatchError}

So one Sprites timeout left a `ready` row nobody could reap (500 from
/admin/sandboxes), whose agent could not be deleted (500), which crashed its
ConversationServer at idle, and which held a quota slot for good. With
`SANDBOX_CAP_FLOOR` at 2 that is half a small tenant's budget, and the only
documented remedy was "the operator must reconcile", with no lever to do it.

The rule is now that the fence stops the machine being re-used, not finished
off. `update_sandbox/2` lets a write through when the resulting status is
terminal, and refuses anything else. That covers the reset's own confirmed
destroy, an operator reap, agent deletion, account deletion and a server
giving up — each of which is the fence ending correctly.

Reaping is therefore the supported way out, and it already exists in the
console and the admin API; it just crashed. No new surface.

Park is the one reachable non-terminal writer, so it skips a fenced machine
instead, matching the reaper queries, in both `Lifecycle.park/4` and
`HomeCheckpoint.on_park/1` — there is no checkpoint worth taking of a disk
that is meant to be gone.

Also from the review:

- An unconfirmed reset committed the fence and cleared every
  `runtime_session_id` on the machine, then recorded nothing. `sandbox.reset`
  stays for the confirmed destroy; `sandbox.reset_requested` now records the
  fence, so whoever reconciles it can see who asked, when and why.
- The status gate had tightened from "not terminated/failed" to
  `ready`/`suspended` with no note. Documented in the docstring and in
  docs/concepts/sandboxes.md, with the reason: a machine still under
  construction has no disk to replace.
- The `down` migration refused whenever any row carried evidence, and a
  *successful* reset keeps its `reset_requested_at` by design — so the first
  reset in production would have been the last time the migration could be
  reversed. It now drops the column, which is the correct rollback semantic
  (the code being rolled back to has no fence concept), and logs each machine
  that had an unconfirmed delete so the operator can check it. Verified on a
  throwaway database with a fenced row present; `id::text` because a raw
  query hands back the 16-byte UUID and the Logger formatter raises on it.
  #1820 has the same one-way `down` and is worth the same look.
- `check_attachable/4` now formats like the clause beneath it.

The existing test asserting `update_sandbox(home, %{status: "terminated"})`
is refused asserted the bug; it now covers a `suspended` write, which still
is. New coverage: reap, agent deletion, a failed write, park-is-skipped, both
audit rows, and the pending/starting refusal.

apps/fountain: 994 tests, 0 failures across conversations, quotas, the reaper
and the audit guardrail. credo --strict, format, and the three prose gates
clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181T6GGULcZfUa6dQtgNwtY
jhgaylor added a commit that referenced this pull request Sep 11, 2026
Review of #1768 found the fence was a dead end. `update_sandbox/2` refused
every write to a fenced row, and its own docstring says "every sandbox status
change in the system goes through here" — which is true, and most of those
callers match `{:ok, _}`. Two reaper queries were filtered against this;
nothing else was. Reproduced all three before fixing:

  reap:         {:RAISED, MatchError}
  delete_agent: {:RAISED, MatchError}
  park:         {:RAISED, MatchError}

So one Sprites timeout left a `ready` row nobody could reap (500 from
/admin/sandboxes), whose agent could not be deleted (500), which crashed its
ConversationServer at idle, and which held a quota slot for good. With
`SANDBOX_CAP_FLOOR` at 2 that is half a small tenant's budget, and the only
documented remedy was "the operator must reconcile", with no lever to do it.

The rule is now that the fence stops the machine being re-used, not finished
off. `update_sandbox/2` lets a write through when the resulting status is
terminal, and refuses anything else. That covers the reset's own confirmed
destroy, an operator reap, agent deletion, account deletion and a server
giving up — each of which is the fence ending correctly.

Reaping is therefore the supported way out, and it already exists in the
console and the admin API; it just crashed. No new surface.

Park is the one reachable non-terminal writer, so it skips a fenced machine
instead, matching the reaper queries, in both `Lifecycle.park/4` and
`HomeCheckpoint.on_park/1` — there is no checkpoint worth taking of a disk
that is meant to be gone.

Also from the review:

- An unconfirmed reset committed the fence and cleared every
  `runtime_session_id` on the machine, then recorded nothing. `sandbox.reset`
  stays for the confirmed destroy; `sandbox.reset_requested` now records the
  fence, so whoever reconciles it can see who asked, when and why.
- The status gate had tightened from "not terminated/failed" to
  `ready`/`suspended` with no note. Documented in the docstring and in
  docs/concepts/sandboxes.md, with the reason: a machine still under
  construction has no disk to replace.
- The `down` migration refused whenever any row carried evidence, and a
  *successful* reset keeps its `reset_requested_at` by design — so the first
  reset in production would have been the last time the migration could be
  reversed. It now drops the column, which is the correct rollback semantic
  (the code being rolled back to has no fence concept), and logs each machine
  that had an unconfirmed delete so the operator can check it. Verified on a
  throwaway database with a fenced row present; `id::text` because a raw
  query hands back the 16-byte UUID and the Logger formatter raises on it.
  #1820 has the same one-way `down` and is worth the same look.
- `check_attachable/4` now formats like the clause beneath it.

The existing test asserting `update_sandbox(home, %{status: "terminated"})`
is refused asserted the bug; it now covers a `suspended` write, which still
is. New coverage: reap, agent deletion, a failed write, park-is-skipped, both
audit rows, and the pending/starting refusal.

apps/fountain: 994 tests, 0 failures across conversations, quotas, the reaper
and the audit guardrail. credo --strict, format, and the three prose gates
clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0181T6GGULcZfUa6dQtgNwtY
@jhgaylor
jhgaylor changed the base branch from fix/serialize-credit-pricer-tests to main September 11, 2026 07:32
@jhgaylor
jhgaylor force-pushed the feat/provider-identity-recording branch 2 times, most recently from 5be8a0e to 70755ea Compare September 11, 2026 07:36
Signed-off-by: Jake Gaylor <jhgaylor@gmail.com>
@jhgaylor
jhgaylor force-pushed the feat/provider-identity-recording branch from 70755ea to 7d5c24a Compare September 11, 2026 07:58
@jhgaylor
jhgaylor added this pull request to the merge queue Sep 11, 2026
Merged via the queue into main with commit a1298e3 Sep 11, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants