fix(contexts): mount writeable contexts from an exact durable snapshot, not a stale executor cache

[federicodeponte]

Problem

A worker writing to a shared, writeable context ("memory") during a run did not reliably see other workers' / prior runs' writebacks. Reported by a cloud user (Nima).

Root cause

The writeback -> durable-storage path is wired and works. The gap is on the mount side: a writeable (mutable) context is mounted from context_dir(name) on the executor's long-lived local disk, which is only hydrated from durable storage when empty (hydrate_if_missing / _cloud_context_dir both skip-if-nonempty). So once a container has a pack on disk, a writeback made on another executor/replica, or an operator edit made through the separate API container, is never re-pulled and the run mounts a stale mutable pack. Writeable contexts are exactly the ones whose source of truth is the durable store, not the cache.

Design reviewed with Codex (adversarial): overwriting the live cache in place was rejected (it races with the worker process's synchronous writeback and the existing downloader is overlay-only, resurrecting deleted files). The accepted design is an immutable per-run snapshot.

Fix

Add a set_context_mount_snapshot_hook seam (mirrors the existing hydration/refresh hooks). writeable_mount_source(name):

• OSS/single-tenant (no hook): yields context_dir(name) unchanged (local disk IS the durable store).
• Hosted (hook set): yields a fresh EXACT snapshot of the pack materialized from durable storage into a temp dir, removed on exit. The long-lived canonical cache is never mounted or mutated, so a concurrent writeback cannot be read torn.

Both mount paths use it for writeable local contexts: e2b_driver._upload_contexts_to_sandbox and agent_capabilities.stage_context_packs. Read-only contexts keep the cheap local cache.

Properties:

• Exact: a file deleted upstream is absent from the snapshot, so a subsequent writeback cannot resurrect it.
• Fail-closed: a listing/download failure aborts the mount as retryable rather than mounting stale mutable data.

The cloud registers the hook (download the exact Storage contents into the temp dir) and bumps this submodule; that PR is separate.

Out of scope (documented)

Full cross-process concurrent-writeback correctness (two runs writing the same pack simultaneously) still needs a per-workspace_id/context_name lock or generation/CAS + delta writeback. Pre-existing last-writer-wins behavior is unchanged here.

Tests

apps/api/tests/test_writeable_context_mount_snapshot.py (7 tests): seam with/without hook, exact snapshot + cleanup, fail-closed propagation, decline-fallback, and both mount paths staging the durable snapshot (not the stale cache). Fail-before/pass-after by construction (cache=v1, durable=v2 -> mounts v2).

Existing context suite: 64 passed, no regressions.

🤖 Generated with Claude Code