Repository navigation
Add an opt-in bubblewrap sandbox for the whole run - #1925
Draft
marcusr-amd wants to merge 2 commits into
Draft
marcusr-amd wants to merge 2 commits into
marcusr-amd wants to merge 2 commits into
Conversation
Code metrics gate: PASSEDJudged: the 3 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
116 findings in files this change does not touchOutside this change (informational): 116
...and 91 more; the job log lists every row.
Thresholds and sources: |
A run inherits whatever the host happens to hold: $HOME, credentials,
unrelated checkouts, ambient caches. Nothing stops an optimization session
from reading or depending on any of it, which makes a result hard to trust
and hard to reproduce somewhere else.
HYPERLOOM_SANDBOX_MODE=bwrap re-executes the process inside bubblewrap
before the CLI dispatches, so every subprocess a command later spawns is a
descendant of the sandbox. Re-exec rather than per-spawn wrapping because
subprocesses are created from more than seventy call sites -- benchmarks,
specialists, the kernel lane, installers -- and a contract spread that
thinly is one nobody can hold. The spawn sites are untouched.
The contract mirrors the Codex sandbox next door: validate the mode, probe
the real namespace capability rather than trusting that a bwrap binary
exists, and fail closed. A run that asked to be isolated and cannot be is
an error, never a quiet downgrade -- and an unrecognised mode is rejected
rather than defaulted, so a typo cannot silently disable isolation. The
default is off, so existing runs are bit-for-bit unchanged.
Four of the binds encode failures that give no useful diagnostic, and are
commented in place so nobody has to rediscover them:
* /sys must be bound. ROCm reads GPU topology from /sys/class/kfd, and
without it torch.cuda.device_count() returns 0 -- no error, no warning,
simply no GPUs.
* TMPDIR must point inside the sandbox. ROCm JIT-compiles its blit
kernels at device init; when the compiler cannot write its temporaries
the process dies with SIGSEGV and prints nothing at all. An inherited
TMPDIR naming an unbound host path is enough to trigger it.
* The interpreter distribution is bound at the path pyvenv.cfg declares,
unresolved. uv points a venv at a version-alias directory which is
itself a symlink; binding the resolved target leaves the alias absent
and the interpreter unreachable. Its parent is bound, not the bin dir,
because the interpreter needs its sibling lib/.
* The sandbox HOME is deliberately not under /home. A tmpfs over the real
home shadows ~/.local/share/uv/python, taking the interpreter with it.
The importing tree travels with the process too: a source checkout on
PYTHONPATH sits outside the venv, and without binding it the re-executed
process silently imports a different Hyperloom -- whichever one the venv
happens to contain. Running other code than the caller's, quietly, is worse
than failing.
The venv is mounted read-only; only paths named in
HYPERLOOM_SANDBOX_WRITABLE are writable. Installed packages are not a
deliverable, so a write to them is a mistake worth surfacing.
Verified on gfx1151 against a ROCm nightly venv: inside the sandbox
~/.ssh is unreadable and the host checkouts are invisible, while torch sees
the GPU and runs, Ray brings up a cluster and schedules a GPU actor on it,
writes to the declared root land on the host, and a write into the venv is
refused. 31 tests cover mode resolution, every bind above, the capability
probe, and each path that must fail closed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The code-metrics gate fails on any literal written three or more times in a file it judges, and this module wrote four: the mode name "off", and the bwrap flags --tmpfs, --ro-bind and --setenv. Naming them is the fix the gate asks for, and it reads better here anyway -- the bind list is now a sequence of named operations rather than a wall of strings that differ by one character. With the flag named, three consecutive single-path --ro-bind blocks were only repetition, so they collapse into one loop over the paths. Argument order and the resulting invocation are unchanged. Also joins two wrapped raise statements that ruff format wanted on one line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
marcusr-amd
force-pushed
the
marcusr/sandbox-reexec
branch
from
October 19, 2026 18:56
a74b648 to
576fcb8
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A run inherits whatever the host happens to hold —
$HOME, credentials, unrelated checkouts, ambient caches — and nothing stops an optimization session from reading or depending on any of it. That makes a result hard to trust and hard to reproduce elsewhere.HYPERLOOM_SANDBOX_MODE=bwrapre-executes the process inside bubblewrap before the CLI dispatches, so every subprocess a command later spawns is a descendant of the sandbox.Default is
off; existing runs are bit-for-bit unchanged.Why re-exec rather than wrapping each spawn
Subprocesses are created from more than seventy call sites across
src/— benchmarks, specialists, the kernel lane, installers. A contract spread that thinly is one nobody can hold, and it would need re-checking on every new spawn site. Re-executing once covers all of them by construction and leaves those call sites untouched.Contract
Mirrors the Codex sandbox in
common/codex_session.py: validate the mode, probe the real namespace capability rather than trusting that abwrapbinary exists, and fail closed. A run that asked to be isolated and cannot be is an error, never a quiet downgrade. An unrecognised mode is rejected rather than defaulted, so a typo cannot silently disable isolation.The venv is mounted read-only; only paths named in
HYPERLOOM_SANDBOX_WRITABLEare writable. Installed packages are not a deliverable, so a write to them is a mistake worth surfacing rather than absorbing.Four binds that encode silent failures
Each is commented in place, because none produces a useful diagnostic:
/systorch.cuda.device_count()returns 0. No error, no warning — ROCm reads GPU topology from/sys/class/kfd.TMPDIRinside the sandboxTMPDIRnaming an unbound host path is enough.pyvenv.cfgdeclares, unresolveduvpoints a venv at a version-alias directory that is itself a symlink; binding the resolved target leaves the alias absent. Its parent is bound, not thebindirectory — the interpreter needs its siblinglib/.HOMEoutside/home~/.local/share/uv/python, taking the interpreter with it.The importing tree travels with the process too: a source checkout on
PYTHONPATHsits outside the venv, and without binding it the re-executed process silently imports a different Hyperloom — whichever one the venv happens to contain. Running other code than the caller's, quietly, is worse than failing.Scope and limits
This is an isolation boundary, not a security boundary against hostile code: without
--unshare-netthe network is shared, and the GPU devices are bound in deliberately.Test plan
ruff checkclean.torch 2.14.0+rocm10.2.0a20261015): inside the sandbox~/.sshis unreadable and host checkouts are invisible; torch sees the GPU and a matmul runs; Ray brings up a cluster and schedules a GPU actor on the Radeon 8060S, then shuts down cleanly; writes to a declared writable root land on the host; a write into the venv is refused withRead-only file system.Note for reviewers
Everything above was measured on a single machine with one GPU. The bind list is derived from observed failures rather than from documentation, so the four entries in the table are the ones most likely to need adjusting on other hardware — and each fails silently, which is why they carry their reasoning inline.