diff --git a/.azure-pipelines/templates/SDK.Integration.Test.Job.yml b/.azure-pipelines/templates/SDK.Integration.Test.Job.yml index b2b32dbc6..d3242d2a2 100644 --- a/.azure-pipelines/templates/SDK.Integration.Test.Job.yml +++ b/.azure-pipelines/templates/SDK.Integration.Test.Job.yml @@ -102,6 +102,20 @@ jobs: sudo apt-get install -y -qq lxc lxc-utils dnsmasq-base iptables bubblewrap slirp4netns displayName: Install LXC, Bubblewrap, and slirp4netns + # LXC enforces network policy from the policy itself, and the default is + # deny-all, so every LXC run installs an egress chain even when the + # request carries no network section. A bridged veth only reaches + # FORWARD while br_netfilter delivers bridged packets to iptables, and + # without it the backend refuses to report success for a policy it + # cannot enforce. Tolerated rather than required, because a hosted pool + # may forbid loading modules; when it does, the LXC tests fail exactly + # as they already would. + - script: | + sudo modprobe br_netfilter || echo "br_netfilter unavailable; LXC policy enforcement is unreachable on this pool" + sudo sysctl -w net.bridge.bridge-nf-call-iptables=1 || true + sudo sysctl -w net.bridge.bridge-nf-call-ip6tables=1 || true + displayName: Enable bridge netfilter + - script: sudo MXC_SKIP_LXC_NETWORK_TESTS=1 MXC_DEBUG=${{ parameters.debug }} npm test workingDirectory: $(integrationDirectory) displayName: npm test (sudo, with LXC) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 6eca4ebb3..e7c6fdd1c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -186,7 +186,7 @@ The Rust workspace (`src/`) implements multiple sandboxing backends behind the ` | Hyperlight | `wxc-exec.exe` | Windows | `backends/hyperlight/common/src/lib.rs` — Hyperlight + Unikraft micro-VM backend | | IsolationSession | `wxc-exec.exe` | Windows | `backends/isolation_session/common/src/` — feature-gated behind `isolation_session`, experimental, uses the in-proc `Windows.AI.IsolationSession.Preview` `IsoSessionOps` API. Supports both one-shot (single-invocation lifecycle, via `ScriptRunner`) and state-aware (multi-invocation provision/start/exec/stop/deprovision, via `StatefulSandboxBackend`) modes. Rejects all filesystem policy (`readwritePaths`/`readonlyPaths`/`deniedPaths`) at every phase with `policy_validation` — the backend has no host-folder-sharing primitive. Likewise rejects any supplied `ui` policy at every phase on both surfaces (as `policy_validation` on the state-aware surface; one-shot discards the typed variant and surfaces `backend_error` with the reason in the message): the isolation session isolates the *host's* UI from contained code but does not deny it UI capabilities (window creation, GDI and the session's own clipboard all work inside it), so no `ui` posture is truthful here — there is no value combination that could be accepted instead, which is why there is no acknowledgment-style gate as there is for `network`. The check is presence-based via `ContainerPolicy::ui_specified` (twin of `network_specified`) because `UiPolicy`'s defaults are full lockdown, making an explicit lockdown `ui` indistinguishable by value from an absent one. An omitted `ui` is accepted and applies no restriction — the schema's default-deny reading does not hold on this backend. One-shot additionally rejects `lifecycle.destroyOnExit=false` and `lifecycle.preservePolicy=true` — the in-proc API has no session-lifetime knob, and the default `destroyOnExit=true` matches actual behavior so it is accepted; the state-aware parser already rejects the whole `lifecycle` section. The full per-phase honor matrix for both surfaces is in `docs/isolation-session/state-aware-rust.md`. The container's network is unrestricted (outbound open; a process inside can listen on a localhost-reachable port) and MXC has no primitive to filter or deny it, so provision (and one-shot) accept ONLY the canonical unrestricted-network acknowledgment — `network.defaultPolicy=allow` + `network.allowLocalNetwork=true`, no host rules, no proxy, default enforcement — and refuse anything else (including an absent policy, which defaults to the unenforceable deny) with `policy_validation`; post-provision phases reject any supplied network policy (fixed at provision, tracked via `ExecutionRequest.network_specified`) and inherit an absent one. State-aware provision accepts an optional `appId` (a packaged app must pass its Package Family Name in the `PFN:` format, e.g. `PFN:Contoso.App_8wekyb3d8bbwe`; an unpackaged app may pass any string), carried verbatim inside the returned `sandboxId`; the one-shot surface takes no backend configuration at all (a stray `experimental.isolation_session` payload is accepted and ignored). Streams stdout/stderr, forwards stdin, and switches to ConPTY mode when wxc-exec's stdout is a TTY for `spawnSandbox` parity. | | WSLc | `wxc-exec.exe` | Windows | `backends/wslc/common/src/` — feature-gated behind `wslc`, experimental, uses the WSLc SDK (`wslcsdk.dll`, loaded at runtime) to run Linux containers in a WSL2 VM. Supports both one-shot (`WSLContainerRunner`, via `ScriptRunner` + streaming `SandboxBackend`) and state-aware (`state_aware.rs` `WslcStateAwareRunner`, via `StatefulSandboxBackend`) modes. Because the WSLc SDK has **no cross-process re-attach**, state-aware keeps the session (VM) + container warm across separate `wxc-exec` phase processes behind a persistent per-user daemon (`wxc-wslc-daemon.exe`, `backends/wslc/daemon/`) that owns the live `WslcSession`/`WslcContainer` handles; phase processes are thin named-pipe clients (`daemon_client.rs`). The daemon runs all SDK calls on one apartment-affine worker thread (so exec is currently serialized across sandboxes — see `docs/wsl/wslc-state-aware.md`). Honors `readwritePaths`/`readonlyPaths` at provision (→ container volumes) + `network.defaultPolicy` (`Block`→`None`, `Allow`→`Bridged`; networking is all-or-nothing — no per-host filtering, since the container lacks `CAP_NET_ADMIN`); rejects `deniedPaths` nested under a mount and rejects proxy/host-filtering at provision. exec honors `network.proxy` **url-form only** (injected as `HTTP_PROXY`/`HTTPS_PROXY`); start/stop/deprovision reject all policy. ID prefix `wslc` (`wslc:<32-hex>`). Idle-timeout is env-overridable via `MXC_WSLC_DAEMON_IDLE_TIMEOUT_SECS`/`MXC_WSLC_DAEMON_IDLE_POLL_SECS`. See `docs/wsl/wslc-state-aware.md`. | -| LXC | `lxc-exec` | Linux | `core/lxc/src/main.rs` + `backends/lxc/common/` | +| LXC | `lxc-exec` | Linux | `core/lxc/src/main.rs` + `backends/lxc/common/` — supports both **one-shot** (single-invocation lifecycle) and **state-aware** (multi-invocation provision/start/exec/stop/deprovision, via `StatefulSandboxBackend`) modes. State-aware lives in `backends/lxc/common/src/state_aware.rs` (`LxcStateAwareRunner`) and needs **no daemon**: the container outlives each phase process in the LXC runtime, so `sandboxId` is just `lxc:` and every phase re-acquires the handle with `LxcContainer::new`. In a firewall mode the host-side veth name is derived from the container name and pinned by a container-global `lxc.hook.start-host` installed **before** start, so the iptables FORWARD chain is scoped before the interface exists; the hook resolves the container's peer interface from `$LXC_PID` and renames it to that name, and exits nonzero — aborting the start — if it cannot find one. The interface set is read from liblxc (`lxc-info -c`) rather than by parsing the container's config, so an `lxc.include` that declares interfaces elsewhere is resolved rather than refused, and no interface *index* is read at all, so a container numbering its interface `lxc.net.3` is enforced exactly as one using `lxc.net.0`. Start is refused when the container declares anything other than exactly one interface, or does not declare that interface's type as `veth` (an undeclared type is refused too, since absence is not evidence of a veth). Teardown derives that same name rather than asking `lxc-info`, which keeps reporting the name liblxc recorded before the hook renamed it, and is ownership-scoped, so a concurrent start's chain is left alone; mount cleanup clears only MXC-marked `lxc.mount.entry` lines. Inbound default-deny (`IngressManager`) is installed on both paths; on the state-aware path it goes on after `container.start()` because the chain lives in the container's network namespace and needs its init PID, and a failure there rolls the start back. Start, stop, and deprovision each take a per-sandbox `flock` in the LXC root, so a teardown cannot strip a start's firewall in the window before the container runs. See `docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md`. | | Seatbelt | `mxc-exec-mac` | macOS | `core/mxc_darwin/src/main.rs` + `backends/seatbelt/common/` — uses macOS App Sandbox (Seatbelt) profiles for process containment. Requires schema `0.7.0-alpha`+. Supports `network.proxy` via the same cooperative env-var model as Bubblewrap (injects `HTTP_PROXY`/`HTTPS_PROXY` into the sandbox, reusing `wxc_common::unix_proxy_coordinator`; `builtinTestServer` spawns the shared `unix-test-proxy`). See `docs/macos-support/seatbelt-backend.md`. | | Bubblewrap | `lxc-exec` | Linux | `backends/bubblewrap/common/src/bwrap_runner.rs` — unprivileged sandboxing via Linux user namespaces and `bwrap`. Experimental — requires `--experimental`. Uses shared filesystem/network policy fields; per-host network filtering via `NetworkIptablesManager` from `backends/lxc/common`. For schema 0.8+, proxy mode uses a private network namespace with rootless `slirp4netns` routing and a default-DROP egress chain that permits only loopback and the translated proxy endpoint (installed via `nsenter` + `iptables`/`ip6tables` from a supervisor that holds the namespaces); it requires `slirp4netns`, util-linux `unshare`/`nsenter` and `iptables`/`ip6tables` on PATH and fails validation if any is unavailable. `iptables`/`ip6tables` must also resolve to the `nf_tables` backend, unless `/run/xtables.lock` is writable by the calling user: the legacy backend opens that lock unconditionally, and the rules are installed by an unprivileged same-uid supervisor that cannot open a root-owned one — `validate` refuses such a host rather than letting the supervisor die at the first rule. The host proxy endpoint is rewritten to slirp's gateway `10.0.2.2`, so `127.0.0.1`/`0.0.0.0`/`::` are translated while `::1` is rejected (an IPv6-loopback listener cannot accept the gateway's IPv4 connection). Schema 0.6/0.7 and absent-version requests retain the legacy shared-network proxy behavior. See `docs/bwrap-support/bubblewrap-backend.md`. | @@ -278,7 +278,7 @@ The workspace is organized into six top-level directories under `src/`: - `learning_mode_windows` (`backends/learning_mode/windows`) is a Windows-only backend support crate for the AppInfo-brokered Learning Mode APIs in `processmodel.dll`. It runtime-resolves the Learning Mode trace and process security-environment exports, owns their typed handle/lifecycle wrappers, decodes sealed ETL traces through `learning_mode_core`, and process-scopes guarded WPR retention with Windows Trace Relogger using exact job-attested PID/creation/exit `FILETIME` ranges. It depends on `wxc_common` plus `learning_mode_core`; runner integration consumes it from the AppContainer backend layer. The trace contract is `HRESULT Start` + retryable `HRESULT Stop` + infallible `Close`: `Stop` never consumes the trace handle, and every started trace must be closed exactly once (closing without stopping is the early-exit discard path). The process security-environment contract is `HRESULT Create` + infallible by-value `Close` and consumes a PSEC 1.0 FlatBuffer, not the legacy SBOX buffer; generated PSEC bindings live in `core/generated/process_security_environment_specification`. - `plm` (`host/plm`) is the Windows-only legacy WPR Learning Mode helper. Public `plm.exe` is `asInvoker`: ETL analysis and every caller-selected file path stay under the caller token. It self-elevates only hidden fixed WPR operations; the retained elevated guardian accepts authenticated attach and stop/analyze/discard controls over unique local PID-checked named pipes and uses the compiled-in profile from protected fixed-volume ProgramData scratch. WPR's host-wide source ETL never crosses the privilege boundary: after terminal job tracking, the guardian relogs a separate ETL containing only supported Learning Mode events from exact handle-attested process generations, analyzes that filtered ETL, and returns its bytes only when explicitly retained. Relogging failure transfers no trace. Successful authenticated stop/discard disarms the child before releasing the PLM singleton. Owner death, pipe break, or another uncertain control failure preserves the recovery marker and deliberately leaves WPR untouched for administrator recovery. - `wxc`, `lxc`, and `mxc_darwin` are thin binary crates (`wxc-exec` / `lxc-exec` / `mxc-exec-mac`) that wire up CLI args (`clap`), load/validate config, handle maintenance modes (`--probe`, `--delete`, `--setup-*`, `--audit`), and **delegate all backend dispatch to `mxc_engine`**. They contain no `match request.containment` of their own. `wxc-exec` additionally owns the Windows Ctrl-C / DACL-cleanup / telemetry orchestration around the engine call. Its `--audit` compatibility workflow synthesizes allow-mode `captureDenials` with ETL retention, then generates policy-authoring artifacts from the canonical denials document returned by the selected engine backend. -- `mxc_engine` is the **single execution engine** — the one home for "given an `ExecutionRequest`, run it". It owns: run-to-completion backend selection (`run` / `resolve_runner`, covering **all** backends, incl. the Windows ProcessContainer BaseContainer/AppContainer BFS/DACL fallback tiers via `appcontainer_common::dispatcher::dispatch_with_fallback`, and every experimental backend, feature-gated); streaming (`spawn` → `Box`); state-aware lifecycle dispatch (`run_state_aware`, including Windows Sandbox and IsolationSession); host probing (`platform_support` / `PlatformSupport`); and config building (`build_request` / `build_request_with_containment`, `SandboxPolicy` + sections, `available_tools_policy`/`user_profile_policy`/`temporary_files_policy`). It depends on the backend crates (cfg-split: appcontainer/windows_sandbox lifecycle/isolation_session/wslc/nanvix on Windows, bubblewrap/lxc/nanvix on Linux, seatbelt on macOS) so it can't live in `wxc_common`. Both the executor binaries and `mxc-sdk` call into it. `ResolvedRunner` carries the boxed runner plus (Windows only) the optional `DaclManager` guard, so `wxc-exec` can park the guard for its signal handler. +- `mxc_engine` is the **single execution engine** — the one home for "given an `ExecutionRequest`, run it". It owns: run-to-completion backend selection (`run` / `resolve_runner`, covering **all** backends, incl. the Windows ProcessContainer BaseContainer/AppContainer BFS/DACL fallback tiers via `appcontainer_common::dispatcher::dispatch_with_fallback`, and every experimental backend, feature-gated); streaming (`spawn` → `Box`); state-aware lifecycle dispatch (`run_state_aware`, covering Windows Sandbox, IsolationSession, LXC, and WSLc); host probing (`platform_support` / `PlatformSupport`); and config building (`build_request` / `build_request_with_containment`, `SandboxPolicy` + sections, `available_tools_policy`/`user_profile_policy`/`temporary_files_policy`). It depends on the backend crates (cfg-split: appcontainer/windows_sandbox lifecycle/isolation_session/wslc/nanvix on Windows, bubblewrap/lxc/nanvix on Linux, seatbelt on macOS) so it can't live in `wxc_common`. Both the executor binaries and `mxc-sdk` call into it. `ResolvedRunner` carries the boxed runner plus (Windows only) the optional `DaclManager` guard, so `wxc-exec` can park the guard for its signal handler. - `mxc-sdk` is the **public Rust SDK** — a thin facade over `mxc_engine`. Build a `SandboxRequest` with `build_request`, then either `run(request)` (run-to-completion; returns an `Output` with the `WaitOutcome`, captured `stdout`/`stderr`, warnings, and optional structured output metadata) or `spawn_sandbox(request)` (returns a `Sandbox` handle for live bidirectional stdio — `take_stdin`/`take_stdout`/`take_stderr`, `kill()`, `wait()` returning a `WaitOutcome` (`Exited(i32)` / `TimedOut`) as `io::Result`, `output_metadata()` after terminal completion, or `wait_with_output()`). It re-exports the engine's config-building surface (`build_request`, `build_request_with_containment` + `Containment`/`WslcSection`, `mxc_sdk::policy::{SandboxPolicy sections}`, discovery helpers) and `platform_support`; `mod sandbox` (wrapping the engine's `SandboxProcess` in `Sandbox`) is its only local module. No pty is ever allocated. Streaming supports Seatbelt (macOS), Bubblewrap (Linux), Windows ProcessContainer (AppContainer + BaseContainer), and WSLC (Windows, experimental — needs the crate's `wslc` feature plus `SandboxRequest::set_experimental(true)`; no stdin and `id() == 0`, since the WSLC SDK exposes neither); other backends return `ErrorCode::UnsupportedContainment`. - The lower-level execution surface lives in `wxc_common::sandbox_process`: the `SandboxBackend` trait (`validate` + `spawn(request, logger, StdioMode) -> Box` + a `diagnose_exit` hook) and the generic `Runner` adapter that bridges any `SandboxBackend` to the run-to-completion `ScriptRunner` (via `spawn(StdioMode::Inherit)` then `wait()`). `SandboxProcess::output_metadata()` carries backend-produced structured outputs after terminal teardown without writing to process-global stdio. `StdioMode::Pipes` hands the caller live stdin/stdout/stderr (what the `mxc-sdk` streaming path uses); `StdioMode::Inherit` lets the child inherit the host's stdio (what the executor binaries use, preserving the TTY under a pty). `SandboxBackend` is implemented for Seatbelt, Bubblewrap, Windows ProcessContainer, and WSLC (on `wslc_common::WSLContainerRunner` itself, which shares one container lifecycle — `start_container` — between its streaming `SandboxBackend` and run-to-completion `ScriptRunner` impls, differing only in where the WSLC SDK's output callbacks write). - `mxc_ffi` (`ffi/mxc_ffi`, `crate-type = ["cdylib", "staticlib", "lib"]`) is a flat, panic-safe **C ABI over `mxc-sdk`** for language bindings. `mxc_run(policyJson, command, out)` runs a sandbox to completion, filling a `#[repr(C)] MxcRunResult` (status + exit_code + timed_out, owned stdout/stderr/output-metadata C strings, and an `MxcErrorDetail` carrying the failure message plus the failing API call and its platform status); every entry point is `catch_unwind`-wrapped so a panic becomes a status code, never an unwind. Its `build.rs` runs **csbindgen** to generate the C# P/Invoke (`sdk/dotnet/Microsoft.Mxc.Sdk/Native/NativeMethods.g.cs`), gated behind the crate's **`dotnetsdk`** feature (off by default, so the whole-workspace backend build matrix doesn't compile csbindgen). The generated file is **not committed** (gitignored); the C# csproj regenerates it at build time and `scripts/check-dotnet-bindings-codegen.js` runs the codegen in CI and asserts the expected entry points are produced. The C ABI is **not a stable external contract** (native + binding are co-versioned and generated together; see the crate docs). It exposes three surfaces: **run-to-completion** (`mxc_run`), **streaming** (`mxc_spawn` → opaque `MxcSandbox` handle; `mxc_stream_read`/`write`/`flush`, `mxc_sandbox_take_stdin`/`stdout`/`stderr`, `mxc_sandbox_id`/`try_wait`/`wait`/`kill`/`output_metadata_json`/`free`, in `src/streaming.rs`), and the **state-aware lifecycle** (`mxc_state_aware` for the envelope phases + `mxc_state_aware_exec` returning a live streaming handle, in `src/state_aware.rs`). All four `.rs` files are csbindgen inputs in `build.rs` (the shared `MxcErrorDetail` lives in `src/error_detail.rs`); the `MXC_STATUS_*` space already reserves the state-aware phase codes. diff --git a/.github/workflows/SDK.Integration.Test.Job.yml b/.github/workflows/SDK.Integration.Test.Job.yml index fc8495e37..db08912d6 100644 --- a/.github/workflows/SDK.Integration.Test.Job.yml +++ b/.github/workflows/SDK.Integration.Test.Job.yml @@ -180,6 +180,21 @@ jobs: sudo systemctl start lxc-net sudo systemctl is-active --quiet lxc-net + # LXC enforces network policy from the policy itself, and the default is + # deny-all, so every LXC run installs an egress chain even when the + # request carries no network section. A bridged veth only reaches + # FORWARD while br_netfilter delivers bridged packets to iptables, and + # without it the backend refuses to report success for a policy it cannot + # enforce, so every LXC test fails at the backend probe. + - name: Enable bridge netfilter + if: matrix.os_label == 'linux' + shell: bash + run: | + set -euo pipefail + sudo modprobe br_netfilter + sudo sysctl -w net.bridge.bridge-nf-call-iptables=1 + sudo sysctl -w net.bridge.bridge-nf-call-ip6tables=1 + - name: Restore execute permission on mxc-exec-mac if: matrix.os_label == 'macos' shell: bash diff --git a/docs/lxc-support/lxc-backend.md b/docs/lxc-support/lxc-backend.md index bdca9afb4..b6662eb2d 100644 --- a/docs/lxc-support/lxc-backend.md +++ b/docs/lxc-support/lxc-backend.md @@ -82,6 +82,49 @@ The `distribution` and `release` fields control which LXC template is used to cr | `debian` | `bookworm`, `trixie` | Stable, well-tested | | `fedora` | `39`, `40` | Modern packages | +### State-aware lifecycle configuration + +The table above describes the **one-shot** surface, where the two fields sit in +a top-level `lxc` section. The state-aware lifecycle carries the same two +fields in the backend's own sub-object instead, under +`experimental.lxc.provision`, alongside the `provision` phase that consumes +them: + +```json +{ + "phase": "provision", + "containment": "lxc", + "experimental": { + "lxc": { + "provision": { + "distribution": "alpine", + "release": "3.23" + } + } + } +} +``` + +`provision` is the only phase that takes LXC-specific configuration. `start`, +`exec`, `stop`, and `deprovision` carry no `experimental.lxc` payload — they +route by the `sandboxId` returned from `provision`, and their cross-cutting +`filesystem` and `network` policy comes from the top-level sections. + +| Rule | Enforced by | Diagnostic when violated | +|------|-------------|--------------------------| +| Both `distribution` and `release` are required | runtime | `LXC distribution and release are required` | +| `experimental.lxc.provision` must be present on the provision phase | runtime | `experimental.lxc.provision with distribution and release is required` | +| Each field must be a string | schema and parser | ``Invalid configuration at `experimental.lxc.provision.distribution`: invalid type: ... expected a string`` | +| Only one backend section may appear, and it must match `containment` | runtime | `Multiple containment backends configured: ... Only one backend section is allowed; remove the unused section(s)` | + +Both fields are optional in the wire model, exactly as they are in the one-shot +`lxc` section, so the schema accepts a provision section that omits them and the +backend is what refuses it. That split is deliberate: the `experimental` block +is intentionally permissive and the schema "is an editor/CI convenience, never +the gate" (`docs/schema-codegen.md`), so requirements that a caller must satisfy +live in the parser and the backend. See `docs/versioning.md` for the +single-backend-section rule and its graduation path. + ### Process Environment and Working Directory The `process.cwd` and `process.env` fields from the standard schema are honored inside the container: @@ -111,7 +154,12 @@ Filesystem policies are enforced via bind mounts in the container configuration: Network policy has two independent halves: outbound (egress) filtering on the host, described first, and inbound (ingress) filtering inside the container, described under [Inbound (ingress) policy](#inbound-ingress-policy). -Both halves require `enforcementMode` to be `firewall` or `both`. Under the default `capabilities` mode, MXC installs no iptables rules at all, so `defaultPolicy`, `allowedHosts`, and `blockedHosts` are parsed but never take effect. +The egress half is installed whenever the policy requires firewall enforcement: +`defaultPolicy` is `"block"` (including its default when omitted), +`allowedHosts` or `blockedHosts` is non-empty, or `proxy` is enabled. The +ingress half is installed for every LXC run, regardless of that predicate. LXC +accepts `enforcementMode` but ignores its value; it does not disable policy +enforcement. Outbound policies are enforced with parallel `iptables` and `ip6tables` chains scoped to the container's virtual ethernet (veth) interface: @@ -220,13 +268,14 @@ Egress firewall state is torn down automatically with best-effort removal of the ### Inbound (ingress) policy -Inbound filtering is a separate chain from the egress chains above, and it lives **inside the container's own network namespace** rather than on the host. Every command is issued through `nsenter -t -n`, so the container's init PID is mandatory. When a firewall enforcement mode is requested and MXC cannot discover that PID, the run is aborted rather than started with inbound enforcement silently disabled. This is LXC-specific, and the Bubblewrap comparison is policy-dependent rather than absolute: Bubblewrap gives the sandbox its own network namespace via `--unshare-net` when the default policy is `block` with no `allowedHosts`, no `blockedHosts`, and no proxy, and shares the host's namespace otherwise. It installs no inbound chain in either case — under `--unshare-net` because nothing outside the sandbox can reach in, and when the namespace is shared because an inbound chain there would be host-wide. +Inbound filtering is a separate chain from the egress chains above, and it lives **inside the container's own network namespace** rather than on the host. Every command is issued through `nsenter -t -n`, so the container's init PID is mandatory for every run. If MXC cannot discover that PID, the run is aborted rather than started with inbound enforcement silently disabled. This is LXC-specific, and the Bubblewrap comparison is policy-dependent rather than absolute: Bubblewrap gives the sandbox its own network namespace via `--unshare-net` when the default policy is `block` with no `allowedHosts`, no `blockedHosts`, and no proxy, and shares the host's namespace otherwise. It installs no inbound chain in either case — under `--unshare-net` because nothing outside the sandbox can reach in, and when the namespace is shared because an inbound chain there would be host-wide. Every `iptables`/`ip6tables` subprocess is spawned with `LC_ALL=C` and `LANG=C`. Teardown decides whether a non-zero exit means "already absent" by matching iptables' own diagnostic text, and that text is localized, so an unpinned locale would turn a benign already-absent result on a non-English host into a fatal error and abort every fresh install. -The rows below describe the `firewall` and `both` enforcement modes. `networkEnforcementMode` defaults to `capabilities`, and under that mode the ingress path installs nothing at all — the same gate that skips the egress chains skips this one, so inbound is unfiltered and `allowLocalNetwork: true` is accepted rather than refused. Inbound default-deny is a property of the firewall enforcement modes, not of every LXC run. +The rows below apply to every LXC run. Inbound default-deny is unconditional; +it does not depend on the egress policy or `enforcementMode`. -| Policy (`networkEnforcementMode`: `firewall` or `both`) | Implementation | +| Network policy | Implementation | |--------|---------------| | `allowLocalNetwork: false` (default) | Container `INPUT` chain drops new inbound connections | | `allowLocalNetwork: true` | **Not yet implemented.** Firewall setup fails with an explicit not-yet-implemented error rather than falling back to an unenforced accept | @@ -265,15 +314,8 @@ unreachable and the firewall rule would never match. `{ "builtinTestServer": true }` is rejected for the same reason, as is a `url` whose host is a loopback literal. -Two further constraints are enforced at parse time, both rejections rather than -silent corrections: +One further constraint is enforced at parse time: -- **`enforcementMode` must be `firewall` or `both`.** Under the default - `capabilities` mode no iptables rules are installed, so the proxy env vars - would be injected while direct egress stayed open — a config that reads as - deny-all-except-proxy and enforces neither half. MXC refuses it rather than - auto-promoting the mode, so a stated enforcement level is never silently - rewritten. - **The `url` must not carry credentials.** LXC passes the proxy URL to `lxc-attach` as a `--set-var` argument, and process arguments are world-readable through `/proc//cmdline`, so inline `user:pass@` would be diff --git a/docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md b/docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md index cb121aaf5..7ff845c62 100644 --- a/docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md +++ b/docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md @@ -236,7 +236,7 @@ wire format and have different roles: | Field | Where it appears | Source | Purpose | |---|---|---|---| | `sandboxId` | State-aware wire envelope (§7); SDK return value from `provisionSandbox` | System-generated by the backend | Opaque routing identifier; must be passed to subsequent state-aware calls | -| `containerId` | One-shot wire envelope (per `docs/schema.md`) | Caller-supplied (or auto-generated random hex) | Human-readable label, used as e.g. AppContainer profile name | +| `containerId` | One-shot wire envelope (per `docs/schema.md`); **LXC** state-aware `provision` | Caller-supplied (or auto-generated random hex) | Human-readable label, used as e.g. AppContainer profile name or LXC container name | State-aware non-provision calls carry `sandboxId` on the request; provision returns it on the response. A state-aware request **may** also carry `containerId` — the parser @@ -244,6 +244,19 @@ preserves it into the request the backend receives — but it is inert for backe do not use it as a label, and it is never a routing key on the state-aware path. One-shot calls carry `containerId` (when present); they do not carry `sandboxId`. +**LXC exception.** LXC is the one backend whose state-aware `provision` also accepts +`containerId`, because its durable state *is* the named container and `lxc-ls` / +`lxc-attach` address it by name. Consequences callers must understand: + +- The returned `sandboxId` is `lxc:`, so the name is not opaque. +- Provision is **adopt-or-create**: if a container with that name already exists it is + reused rather than created, and `provisionMetadata.created` reports which happened. +- Deprovision **destroys the container unconditionally**, including one it adopted + rather than created. MXC retains no caller-side state between phases, so deprovision + cannot distinguish an adopted container from a created one. Callers that pass a + `containerId` for a pre-existing container therefore hand MXC ownership of it. +- Omit `containerId` to get a generated name and avoid adopting anything. + ## 6. TypeScript SDK The SDK adds five new functions, exported from `@microsoft/mxc-sdk` alongside the existing @@ -1688,6 +1701,43 @@ unconditionally by the in-guest agent). | `network` | rejected | rejected | rejected | rejected | rejected | | `ui` | rejected | rejected | rejected | rejected | rejected | +For LXC, filesystem path lists and `network` are applied at **start** — the container is +created empty at provision and its mounts and iptables chain are installed just before +it runs — and rejected at every other phase. Only a *non-empty* path list counts as a +filesystem policy, so a `filesystem` block whose lists are all empty is accepted at any +phase. `ui` is not consulted anywhere in the LXC backend; `LxcStartConfig` does not +expose it, so SDK callers cannot pass it, but a raw-JSON caller that sends it is not +rejected. + +| Field | provision | start | exec | stop | deprovision | +|---|---|---|---|---|---| +| `filesystem` (non-empty path lists) | rejected | applied | rejected | rejected | rejected | +| `network` | rejected | applied | rejected | rejected | rejected | +| `ui` | ignored | ignored | ignored | ignored | ignored | + +> **LXC `network` constraints.** `network.proxy` and `allowLocalNetwork` are +> rejected at start. LXC enforces network policy through iptables whenever +> `defaultPolicy` is `"block"`, either host list is non-empty, or proxy is +> enabled. The value of `enforcementMode` is accepted but ignored by LXC; it +> does not disable enforcement. The veth is not discovered after start: the name +> is derived from the container name and pinned by a container-global +> `lxc.hook.start-host` installed before start, so the chain is scoped before the +> interface exists. The hook resolves the container's peer interface from +> `$LXC_PID` and renames it to that name; if it cannot find one it exits +> nonzero, which aborts the start rather than leaving the container unfiltered. +> The interface set comes from liblxc rather than from the container's own +> config file, so an `lxc.include` that declares interfaces elsewhere is +> resolved rather than guessed at, and no interface index is read at all — a +> container numbering its only interface `lxc.net.3` is enforced exactly as one +> using `lxc.net.0`, as is one at any index liblxc accepts. Start fails instead +> when that pin cannot be trusted to cover the container's traffic — when the +> container declares anything other than exactly one interface, or does not +> declare that interface's type as `veth`. An undeclared type is refused +> alongside a wrong one, because absence is not evidence of a veth, and a +> `macvlan` or `phys` interface would take a hook pinned to a veth name that +> never appears while its traffic ran unfiltered. `removeRulesOnExit` is not +> part of the LXC surface (see `LxcNetworkConfig`). + - **Compile-time enforcement at the SDK.** Each per-(backend, phase) Config (§6.1) declares only the cross-cutting fields the matrix marks as `applied` for that phase *and* that the runtime currently honors. TypeScript rejects callers passing fields diff --git a/schemas/dev/mxc-config.schema.0.8.0-dev.json b/schemas/dev/mxc-config.schema.0.8.0-dev.json index 67e674da7..f0f7bbbaa 100644 --- a/schemas/dev/mxc-config.schema.0.8.0-dev.json +++ b/schemas/dev/mxc-config.schema.0.8.0-dev.json @@ -201,6 +201,17 @@ ], "description": "IsolationSession backend config (Windows)." }, + "lxc": { + "anyOf": [ + { + "$ref": "#/definitions/LxcExperimental" + }, + { + "type": "null" + } + ], + "description": "LXC backend config (Linux)." + }, "seatbelt": { "anyOf": [ { @@ -401,6 +412,43 @@ }, "type": "object" }, + "LxcExperimental": { + "description": "LXC backend config under the experimental surface. Carries only the per-phase state-aware nesting for the phases that take config (`provision`); the one-shot LXC surface is the stable top-level `lxc` section, so this type is named apart from it rather than shared with it. `start`, `exec`, `stop`, and `deprovision` take no per-phase config payload.", + "properties": { + "provision": { + "anyOf": [ + { + "$ref": "#/definitions/LxcProvisionPhase" + }, + { + "type": "null" + } + ], + "description": "State-aware provision-phase configuration." + } + }, + "type": "object" + }, + "LxcProvisionPhase": { + "description": "Provision-phase LXC configuration (state-aware lifecycle), nested under `experimental.lxc.provision`. Names the container image to create.\n\nFilesystem mounts and network policy derive from the top-level `filesystem` and `network` sections, not from here. It is its own type rather than a shared one because a shared type would advertise its fields on every phase in the generated schema.", + "properties": { + "distribution": { + "description": "Distribution image (e.g. `alpine`).", + "type": [ + "string", + "null" + ] + }, + "release": { + "description": "Distribution release (e.g. `3.23`).", + "type": [ + "string", + "null" + ] + } + }, + "type": "object" + }, "Network": { "additionalProperties": false, "description": "Network access policy.", diff --git a/sdk/node/README.md b/sdk/node/README.md index 3cc567902..4356575af 100644 --- a/sdk/node/README.md +++ b/sdk/node/README.md @@ -240,7 +240,7 @@ capability names are reserved and must not be added directly to For long-lived sandboxes where you provision once, exec many times, and tear down at the end (e.g. agentic loops), use the state-aware lifecycle. -> **Backend support:** the state-aware lifecycle is currently implemented for `isolation_session`, `windows_sandbox`, and `wslc` (all Windows-only; all still experimental, so every call must pass `{ experimental: true }`). The one-shot spawn APIs (`spawnSandbox` / `spawnSandboxFromConfig`) are the supported path for every other backend. +> **Backend support:** the state-aware lifecycle is currently implemented for `isolation_session`, `windows_sandbox`, and `wslc` (all Windows-only; all still experimental, so every call must pass `{ experimental: true }`) and `lxc` (Linux-only; not experimental). The one-shot spawn APIs (`spawnSandbox` / `spawnSandboxFromConfig`) are the supported path for every other backend. ```typescript import { @@ -404,10 +404,10 @@ spawnSandboxFromConfig(config, options?, workingDirectory?, env?) → IPty | Chi spawnSandbox(script, policy, options?, workingDirectory?, containerName?, env?) → IPty spawnSandboxAsync(script, policy, ...) → Promise<{ stdout, stderr, exitCode }> -// State-aware lifecycle (currently `isolation_session`, `windows_sandbox`, and `wslc` — all Windows-only) +// State-aware lifecycle (`isolation_session`, `windows_sandbox`, and `wslc` — Windows-only, experimental; `lxc` — Linux-only) // `config` on provisionSandbox is required for backends whose provision config -// has a required member (isolation_session: the network acknowledgment) and -// optional otherwise (windows_sandbox, wslc). +// has a required member (isolation_session: the network acknowledgment; lxc: +// distribution and release) and optional otherwise (windows_sandbox, wslc). provisionSandbox(containment, config, options?) → Promise startSandbox(sandboxId, config?, options?) → Promise execInSandbox(sandboxId, config, options?) → IPty // streaming diff --git a/sdk/node/src/generated/wire.ts b/sdk/node/src/generated/wire.ts index 3a58357fb..3de1d3a9a 100644 --- a/sdk/node/src/generated/wire.ts +++ b/sdk/node/src/generated/wire.ts @@ -78,6 +78,10 @@ export interface Experimental { * IsolationSession backend config (Windows). */ isolation_session?: IsolationSession | null; + /** + * LXC backend config (Linux). + */ + lxc?: LxcExperimental | null; /** * Seatbelt backend config (pre-promotion alias). */ @@ -188,6 +192,34 @@ export interface Lxc { release?: string | null; } +/** + * LXC backend config under the experimental surface. Carries only the per-phase state-aware nesting for the phases that take config (`provision`); the one-shot LXC surface is the stable top-level `lxc` section, so this type is named apart from it rather than shared with it. `start`, `exec`, `stop`, and `deprovision` take no per-phase config payload. + */ +export interface LxcExperimental { + /** + * State-aware provision-phase configuration. + */ + provision?: LxcProvisionPhase | null; + [k: string]: unknown; +} + +/** + * Provision-phase LXC configuration (state-aware lifecycle), nested under `experimental.lxc.provision`. Names the container image to create. + * + * Filesystem mounts and network policy derive from the top-level `filesystem` and `network` sections, not from here. It is its own type rather than a shared one because a shared type would advertise its fields on every phase in the generated schema. + */ +export interface LxcProvisionPhase { + /** + * Distribution image (e.g. `alpine`). + */ + distribution?: string | null; + /** + * Distribution release (e.g. `3.23`). + */ + release?: string | null; + [k: string]: unknown; +} + /** * Network access policy. */ diff --git a/sdk/node/src/index.ts b/sdk/node/src/index.ts index 61e1a4b14..d1deb7f15 100644 --- a/sdk/node/src/index.ts +++ b/sdk/node/src/index.ts @@ -107,6 +107,13 @@ export { IsolationSessionStopConfig, IsolationSessionDeprovisionConfig, IsolationSessionProvisionMetadata, + LxcProvisionConfig, + LxcNetworkConfig, + LxcStartConfig, + LxcExecConfig, + LxcStopConfig, + LxcDeprovisionConfig, + LxcProvisionMetadata, WindowsSandboxProvisionConfig, WindowsSandboxStartConfig, WindowsSandboxExecConfig, diff --git a/sdk/node/src/state-aware-helper.ts b/sdk/node/src/state-aware-helper.ts index 5174ce424..b117efd96 100644 --- a/sdk/node/src/state-aware-helper.ts +++ b/sdk/node/src/state-aware-helper.ts @@ -20,13 +20,14 @@ export const WSLC_STATE_AWARE_VERSION = '0.8.0-alpha'; // Wire-format cross-cutting fields that live at the envelope's top level. // Anything else on a per-(backend, phase) Config is backend-specific and is // nested under `experimental..`. -export const CROSS_CUTTING_FIELDS = ['filesystem', 'network', 'ui', 'process'] as const; +export const CROSS_CUTTING_FIELDS = ['containerId', 'filesystem', 'network', 'ui', 'process'] as const; // Per-backend wire-format prefix. Each value mirrors the corresponding // Rust `Runner::ID_PREFIX` const and is the leading segment of a // `sandboxId` produced by that backend. Each future state-aware backend // declares its own `_ID_PREFIX` const here. export const ISOLATION_SESSION_ID_PREFIX = 'iso'; +export const LXC_ID_PREFIX = 'lxc'; export const WINDOWS_SANDBOX_ID_PREFIX = 'wsb'; export const WSLC_ID_PREFIX = 'wslc'; @@ -36,6 +37,7 @@ export const WSLC_ID_PREFIX = 'wslc'; // global constant. const DEFAULT_STATE_AWARE_VERSION: Record = { isolation_session: STATE_AWARE_VERSION, + lxc: STATE_AWARE_VERSION, windows_sandbox: STATE_AWARE_VERSION, wslc: WSLC_STATE_AWARE_VERSION, }; @@ -48,6 +50,7 @@ const DEFAULT_STATE_AWARE_VERSION: Record // `malformed_id`. export const BACKEND_TO_PREFIX: Record = { isolation_session: ISOLATION_SESSION_ID_PREFIX, + lxc: LXC_ID_PREFIX, windows_sandbox: WINDOWS_SANDBOX_ID_PREFIX, wslc: WSLC_ID_PREFIX, }; diff --git a/sdk/node/src/state-aware-types.ts b/sdk/node/src/state-aware-types.ts index 2334ec82b..2d3dd6d0a 100644 --- a/sdk/node/src/state-aware-types.ts +++ b/sdk/node/src/state-aware-types.ts @@ -19,7 +19,7 @@ export type Phase = 'provision' | 'start' | 'exec' | 'stop' | 'deprovision'; */ export type StateAwareContainmentBackend = Extract< ContainmentBackend, - 'isolation_session' | 'windows_sandbox' | 'wslc' + 'isolation_session' | 'lxc' | 'windows_sandbox' | 'wslc' >; /** @@ -94,6 +94,70 @@ export interface IsolationSessionDeprovisionConfig { version?: string; } +export interface LxcProvisionConfig { + /** Schema version (semver). */ + version?: string; + /** Optional externally assigned LXC container name. */ + containerId?: string; + /** Linux distribution for the container rootfs, e.g. "alpine" or "ubuntu". */ + distribution: string; + /** Distribution release version, e.g. "3.20" or "24.04". */ + release: string; +} + +/** + * Network policy accepted by LXC state-aware `start`. + * + * Narrowed to the fields an LXC caller can usefully set, rather than derived + * from `NetworkConfig`: + * + * - `proxy` is rejected at start (`apply_network_policy` returns a + * policy-validation error). + * - `removeRulesOnExit` is an SDK-only field emitted inside the top-level + * `network` object. Rust's `wire::Network` is `deny_unknown_fields`, so + * sending it fails the whole request. + * - `allowLocalNetwork` deserializes, but start rejects + * `allowLocalNetwork: true` with a policy-validation error + * (`reject_unenforceable_network_policy`). The container's inbound chain can + * only open a source range, and opening every source is broader than the + * local-network access requested. + * - `enforcementMode` is not offered. LXC enforces from `defaultPolicy`, + * `allowedHosts`, and `blockedHosts` alone (`apply_firewall_rules` never + * consults the mode), so exposing it would advertise a setting with no + * effect. The wire still accepts the field, so a `network` object shared + * with another backend keeps parsing. + */ +export interface LxcNetworkConfig { + defaultPolicy?: NetworkConfig['defaultPolicy']; + allowedHosts?: NetworkConfig['allowedHosts']; + blockedHosts?: NetworkConfig['blockedHosts']; +} + +export interface LxcStartConfig { + /** Schema version (semver). */ + version?: string; + /** Filesystem mounts to apply before starting the container. */ + filesystem?: FilesystemConfig; + /** iptables policy to install before the container starts. `proxy` is not supported by this backend. */ + network?: LxcNetworkConfig; +} + +export interface LxcExecConfig { + /** Schema version (semver). */ + version?: string; + process: ProcessConfig; +} + +export interface LxcStopConfig { + /** Schema version (semver). */ + version?: string; +} + +export interface LxcDeprovisionConfig { + /** Schema version (semver). */ + version?: string; +} + /** * IsolationSession's provision-phase metadata surfaced to the caller: the * per-instance agent user account name minted for this sandbox, the agent @@ -107,6 +171,11 @@ export interface IsolationSessionProvisionMetadata { ephemeralWorkspacePath: string; } +export interface LxcProvisionMetadata { + containerName: string; + created: boolean; +} + // WindowsSandbox per-(backend, phase) Configs. WindowsSandbox holds a single // active sandbox behind a persistent host-side daemon. Filesystem policy // (readwrite/readonly/denied HOST paths) is honored at provision and is @@ -251,6 +320,13 @@ type StateAwareConfigRegistry = DefineStateAwareConfigRegistry<{ stop: IsolationSessionStopConfig; deprovision: IsolationSessionDeprovisionConfig; }; + lxc: { + provision: LxcProvisionConfig; + start: LxcStartConfig; + exec: LxcExecConfig; + stop: LxcStopConfig; + deprovision: LxcDeprovisionConfig; + }; windows_sandbox: { provision: WindowsSandboxProvisionConfig; start: WindowsSandboxStartConfig; @@ -350,6 +426,10 @@ export type StateAwareMetadata = DefineStateAwareMetadataRegistry<{ provision?: IsolationSessionProvisionMetadata; // IsolationSession returns no metadata for start, stop, or deprovision. }; + lxc: { + provision?: LxcProvisionMetadata; + // LXC returns no metadata for start, stop, or deprovision. + }; // WindowsSandbox returns no metadata for any phase (provision yields only the // sandbox id). The key still participates so `StateAwareMetadata[C]` type- // checks for `C = 'windows_sandbox'`. `Record` has `keyof = diff --git a/sdk/node/src/state-aware.ts b/sdk/node/src/state-aware.ts index 3c1cfa51a..329ff70c7 100644 --- a/sdk/node/src/state-aware.ts +++ b/sdk/node/src/state-aware.ts @@ -102,9 +102,9 @@ export async function startSandbox( /** * Streams a script execution inside a started sandbox. Returns an * `IPty` for live stdout/stderr/exit handling, mirroring `spawnSandbox`. - * On dispatch failure the executor emits a single error envelope on stdout; - * the SDK does not parse it here — callers consuming `IPty.onData` see the - * raw bytes. Use `execInSandboxAsync` when typed-error throwing is needed. + * On dispatch failure the executor emits a single error envelope on stdout + * (§7.3); the SDK does not parse it here — callers consuming `IPty.onData` see + * the raw bytes. Use `execInSandboxAsync` when typed-error throwing is needed. */ export function execInSandbox( sandboxId: SandboxId, @@ -144,8 +144,9 @@ export function execInSandbox( /** * Buffered exec convenience. Resolves with `{stdout, stderr, exitCode}` * on script completion. Throws an `MxcError` (with the wire-format `code` - * field set) when the executor reports a dispatch failure (recognised by - * exit != 0 and stdout being a complete `{error}` envelope). + * field set) when the executor reports a dispatch failure, recognised by + * exit != 0 together with a complete `{error}` envelope on stdout, which §7.3 + * reserves for the response envelope in every backend. */ export async function execInSandboxAsync( sandboxId: SandboxId, @@ -163,6 +164,9 @@ export async function execInSandboxAsync const { stdout, stderr, exitCode } = await spawnAndCollect(envelope, options); if (exitCode !== 0) { + // The cross-backend contract (§7.3) reserves stdout for the response + // envelope in every phase, so a dispatch failure is always found there and + // no backend needs its own channel rule. const errorEnvelope = tryParseErrorEnvelope(stdout); if (errorEnvelope) { throw mxcErrorFromEnvelope(errorEnvelope.error); diff --git a/sdk/node/src/types.ts b/sdk/node/src/types.ts index 11c1cf5da..53781d8d7 100644 --- a/sdk/node/src/types.ts +++ b/sdk/node/src/types.ts @@ -210,9 +210,9 @@ export interface NetworkConfig { * TCP/UDP). Independent of `defaultPolicy`. (default: false) */ allowLocalNetwork?: boolean; - /** Hostnames or IP addresses/CIDR blocks to allow (firewall mode only) */ + /** Hostnames or IP addresses/CIDR blocks to allow (Windows and Bubblewrap require firewall mode) */ allowedHosts?: string[]; - /** Hostnames or IP addresses to block (firewall mode only) */ + /** Hostnames or IP addresses to block (Windows and Bubblewrap require firewall mode) */ blockedHosts?: string[]; /** Proxy configuration (supported on Windows ProcessContainer, Linux Bubblewrap, * macOS Seatbelt, and WSLC). On Bubblewrap/Seatbelt/WSLC it is a cooperative diff --git a/sdk/node/tests/integration/test-helpers.ts b/sdk/node/tests/integration/test-helpers.ts index 8240cf8f7..62e7a9cd9 100644 --- a/sdk/node/tests/integration/test-helpers.ts +++ b/sdk/node/tests/integration/test-helpers.ts @@ -277,6 +277,15 @@ export async function probeStateAwareRuntime { }); it('should force enforcementMode=firewall when host filtering is requested', () => { - // The LXC runner only invokes iptables when network_enforcement_mode is - // Firewall|Both (see lxc_common::network_iptables). Without this stamp, - // the parser would default to Capabilities and allowedHosts/blockedHosts - // would be silently dropped on the floor. + // The stamp stays for parity with the other Linux backends; LXC now + // enforces from the policy, so it is no longer load-bearing here. const config = createConfigFromPolicy({ version: '0.6.0-alpha', network: { allowOutbound: true, allowedHosts: ['example.com'] }, diff --git a/sdk/node/tests/unit/state-aware-types.test.ts b/sdk/node/tests/unit/state-aware-types.test.ts index d16b8332a..44415ab7c 100644 --- a/sdk/node/tests/unit/state-aware-types.test.ts +++ b/sdk/node/tests/unit/state-aware-types.test.ts @@ -12,6 +12,7 @@ import { ProvisionMetadataFor, ProvisionResult, SandboxId, + StartConfigFor, StartMetadataFor, StateAwareContainmentBackend, StopConfigFor, @@ -226,6 +227,70 @@ describe('IsolationSessionStopConfig and IsolationSessionDeprovisionConfig', () }); }); +describe('LxcStartConfig', () => { + it('accepts the network fields LXC actually enforces', () => { + const cfg: StartConfigFor<'lxc'> = { + version: '0.6.0-alpha', + filesystem: { readwritePaths: ['/workspace'] }, + network: { + defaultPolicy: 'block', + allowedHosts: ['example.com'], + blockedHosts: ['blocked.example.com'], + }, + }; + assert.strictEqual(cfg.network?.defaultPolicy, 'block'); + }); + + it('rejects network.proxy because the LXC runner rejects it at start', () => { + const cfg: StartConfigFor<'lxc'> = { + network: { + // @ts-expect-error — LXC state-aware start does not support network.proxy. + proxy: { builtinTestServer: true }, + }, + }; + assert.ok(cfg); + }); + + it('rejects removeRulesOnExit, which Rust would reject as an unknown field', () => { + // `wire::Network` is `deny_unknown_fields` and has no `removeRulesOnExit`, + // so emitting it inside the top-level `network` object fails the request. + const cfg: StartConfigFor<'lxc'> = { + network: { + // @ts-expect-error — SDK-only field; not part of the LXC wire surface. + removeRulesOnExit: true, + }, + }; + assert.ok(cfg); + }); + + it('rejects allowLocalNetwork, which the LXC backend never enforces', () => { + const cfg: StartConfigFor<'lxc'> = { + network: { + // @ts-expect-error — start rejects it with a policy-validation error. + allowLocalNetwork: true, + }, + }; + assert.ok(cfg); + }); + + it('rejects enforcementMode, which LXC never consults', () => { + const cfg: StartConfigFor<'lxc'> = { + network: { + // @ts-expect-error — enforcement follows the policy, not the mode. + enforcementMode: 'capabilities', + }, + }; + assert.ok(cfg); + }); + + it('enforces a restrictive policy from the policy fields alone', () => { + const cfg: StartConfigFor<'lxc'> = { + network: { defaultPolicy: 'block', allowedHosts: ['example.com'] }, + }; + assert.strictEqual(cfg.network?.defaultPolicy, 'block'); + }); +}); + describe('ConfigsForBackend', () => { it('selects the IsolationSession bundle for the isolation_session backend', () => { const bundle: ConfigsForBackend<'isolation_session'> = { diff --git a/sdk/node/tests/unit/state-aware.test.ts b/sdk/node/tests/unit/state-aware.test.ts index f74727167..d7aa50dff 100644 --- a/sdk/node/tests/unit/state-aware.test.ts +++ b/sdk/node/tests/unit/state-aware.test.ts @@ -443,6 +443,87 @@ describe('execInSandboxAsync', { skip: platformSkip }, () => { ); }); + it('throws the typed MxcError on an LXC dispatch failure with diagnostics on stderr', async () => { + // Channel separation, per the cross-backend contract (§7.3): the executor + // flushes its buffered diagnostics to stderr and writes exactly one + // envelope to stdout. stderr is informational and is never parsed, so the + // buffer cannot shadow the envelope no matter how many lines it runs to. + const fake = fakeSpawn({ + stdout: '{"error":{"code":"not_started","message":"sandbox is not started"}}\n', + stderr: 'lxc: preparing container\nlxc: policy validated\n', + exitCode: 1, + }); + _setSpawnImpl(fake.spawn); + const id = 'lxc:prov-1' as SandboxId<'lxc'>; + await assert.rejects( + () => execInSandboxAsync(id, { process: { commandLine: 'echo' } }, testOptions()), + (err: unknown) => err instanceof MxcError && err.code === 'not_started', + ); + }); + + it('returns an ExecResult when a script is killed by its timeout after streaming output', async () => { + // Characterization, not an endorsement. §7.3 models exec as "either the + // script's output (success) or exactly one envelope (failure)", but a + // timeout is a dispatch failure raised *after* the script has streamed, so + // stdout holds both and whole-string parsing matches neither. The caller + // still sees a nonzero exit, but not the typed reason for it. + // + // Every state-aware backend shares this: the executors all write the + // envelope to the same stdout the guest streamed to. Pinning it here means + // closing the contract gap has to be a deliberate change, not a silent one. + const fake = fakeSpawn({ + stdout: + 'partial output before the kill\n' + + '{"error":{"code":"backend_error","message":"Execution failed: script timed out after 5000ms"}}\n', + stderr: 'diagnostic: attaching to container\n', + exitCode: 3, + }); + _setSpawnImpl(fake.spawn); + const id = 'lxc:abc' as SandboxId<'lxc'>; + const result = await execInSandboxAsync( + id, + { process: { commandLine: 'slow' } }, + testOptions(), + ); + assert.strictEqual(result.exitCode, 3); + }); + + it('does not read stderr as an envelope for backends that relay the guest there', async () => { + // The stderr fallback rests on LXC merging guest stderr into stdout, and + // that reasoning is backend-specific. Windows Sandbox forwards guest + // FrameKind::Stderr frames straight to this stream + // (backends/windows_sandbox/lifecycle/src/state_aware.rs:408-414), and the + // isolation-session exec relays the guest to wxc-exec's own stdout and + // stderr. On both, the last stderr line can be the script's own -- so a + // script that ends with envelope-shaped stderr and exits nonzero would be + // turned into a thrown MxcError rather than the result it actually is. + const stderrLookingLikeADispatchFailure = + '{"error":{"code":"not_started","message":"the script printed this itself"}}\n'; + + for (const id of [ + 'wsb:abc' as SandboxId<'windows_sandbox'>, + 'iso:abc' as SandboxId<'isolation_session'>, + ]) { + const fake = fakeSpawn({ + stdout: 'script output\n', + stderr: stderrLookingLikeADispatchFailure, + exitCode: 7, + }); + _setSpawnImpl(fake.spawn); + const result = await execInSandboxAsync( + id as SandboxId<'windows_sandbox'>, + { process: { commandLine: 'noisy' } }, + testOptions(), + ); + assert.strictEqual( + result.exitCode, + 7, + `${id} must return the script's result, not throw its stderr as a dispatch failure`, + ); + assert.strictEqual(result.stderr, stderrLookingLikeADispatchFailure); + } + }); + it('closes the child stdin so a stdin-reading command sees EOF instead of hanging', async () => { // Regression for the buffered-exec hang: the Rust state-aware path waits // for stdin EOF before closing guest stdin, so spawnAndCollect must end() diff --git a/sdk/node/tests/unit/wire-conformance-state-aware.test.ts b/sdk/node/tests/unit/wire-conformance-state-aware.test.ts index 0ea7488c1..78918f4ba 100644 --- a/sdk/node/tests/unit/wire-conformance-state-aware.test.ts +++ b/sdk/node/tests/unit/wire-conformance-state-aware.test.ts @@ -46,12 +46,18 @@ import type { WslcExecConfig, WslcStopConfig, WslcDeprovisionConfig, + LxcProvisionConfig, + LxcStartConfig, + LxcExecConfig, + LxcStopConfig, + LxcDeprovisionConfig, } from '../../src/state-aware-types.js'; import type { Phase as WirePhase, IsolationSessionProvisionPhase as WireProvisionPhase, WslcProvisionPhase as WireWslcProvisionPhase, + LxcProvisionPhase as WireLxcProvisionPhase, } from '../../src/generated/wire.js'; import type { @@ -84,8 +90,10 @@ type _Phase = AssertTrue>; // `filesystem` is a lifted top-level wire field (like `network`): WSLc provision // surfaces it publicly but it maps to the envelope's top-level `filesystem`, not // under `experimental.wslc.provision`. Listing it here keeps the backend-key set -// limited to genuinely per-phase wire fields. -type LiftedPhaseKey = 'version' | 'process' | 'network' | 'filesystem'; +// limited to genuinely per-phase wire fields. `containerId` is lifted the same +// way for every backend -- `CROSS_CUTTING_FIELDS` in `state-aware-helper.ts` +// moves it to the envelope top level, so it never reaches a per-phase object. +type LiftedPhaseKey = 'version' | 'process' | 'network' | 'filesystem' | 'containerId'; type BackendKeys = Exclude; type WireKeys = keyof StripIndex; @@ -172,6 +180,44 @@ type _WslcProvisionWireKeysNonVacuous = AssertTrue< Equivalent, 'image' | 'imageTarPath'> >; +// --- LXC per-phase wire field-set conformance ------------------------------ + +// LXC is the third state-aware backend, so the oracle must cover it too or a +// wire-model change to the LXC surface would regenerate `wire.ts`, pass the +// codegen gate, and leave the SDK silently lagging with no CI signal. LXC's +// only per-phase wire object is provision (`distribution` / `release`); start, +// exec, stop, and deprovision have wire associated type `()` and must expose no +// backend-specific field. `filesystem`, `network`, and `containerId` are lifted +// top-level wire fields (see `LiftedPhaseKey`). +type _LxcProvisionPublicKeys = AssertTrue< + Equivalent, WireKeys>, never> +>; +type _LxcProvisionWireKeys = AssertTrue< + Equivalent, BackendKeys>, never> +>; +type _LxcProvisionFieldValueTypes = AssertTrue< + Equivalent< + PublicFieldValues, + WireFieldValues + > +>; + +type _LxcStartNoBackendKeys = AssertTrue, never>>; +type _LxcExecNoBackendKeys = AssertTrue, never>>; +type _LxcStopNoBackendKeys = AssertTrue, never>>; +type _LxcDeprovisionNoBackendKeys = AssertTrue< + Equivalent, never> +>; + +// Non-vacuity guards (see the isolation_session pins above): pin the derived key +// sets so a derivation bug fails the oracle rather than silently disabling it. +type _LxcProvisionKeysNonVacuous = AssertTrue< + Equivalent, 'distribution' | 'release'> +>; +type _LxcProvisionWireKeysNonVacuous = AssertTrue< + Equivalent, 'distribution' | 'release'> +>; + // --- delegation to the one-shot oracle (documented, asserted) -------------- // The per-phase configs must REUSE the public one-shot leaf types for their @@ -180,6 +226,7 @@ type _WslcProvisionWireKeysNonVacuous = AssertTrue< // assertions fail if that ever happens. type _ExecProcessReuse = AssertTrue>; type _WslcExecProcessReuse = AssertTrue>; +type _LxcExecProcessReuse = AssertTrue>; // Reference the assertion aliases so they read as intentionally load-bearing. export type StateAwareWireConformanceAssertions = [ @@ -204,6 +251,16 @@ export type StateAwareWireConformanceAssertions = [ _WslcProvisionKeysNonVacuous, _WslcProvisionWireKeysNonVacuous, _WslcExecProcessReuse, + _LxcProvisionPublicKeys, + _LxcProvisionWireKeys, + _LxcProvisionFieldValueTypes, + _LxcStartNoBackendKeys, + _LxcExecNoBackendKeys, + _LxcStopNoBackendKeys, + _LxcDeprovisionNoBackendKeys, + _LxcProvisionKeysNonVacuous, + _LxcProvisionWireKeysNonVacuous, + _LxcExecProcessReuse, ]; test('public state-aware SDK types conform to the generated wire schema (compile-time)', () => { diff --git a/src/Cargo.lock b/src/Cargo.lock index b494dd438..42586e6a7 100644 --- a/src/Cargo.lock +++ b/src/Cargo.lock @@ -1392,6 +1392,7 @@ dependencies = [ "mxc_engine", "nanvix_binaries", "nanvix_build_common", + "serde_json", "wxc_common", ] diff --git a/src/backends/lxc/common/src/filesystem_mounts.rs b/src/backends/lxc/common/src/filesystem_mounts.rs index 6dbfc1494..e11495673 100644 --- a/src/backends/lxc/common/src/filesystem_mounts.rs +++ b/src/backends/lxc/common/src/filesystem_mounts.rs @@ -206,7 +206,15 @@ pub fn configure_filesystem_mounts( policy: &ContainerPolicy, logger: &mut Logger, ) -> Result<(), String> { + // Derive the whole mount set first, then commit it in one config rewrite. + // liblxc accumulates `lxc.mount.entry` lines across restarts, so the + // previous run's MXC mounts have to go; committing the replacement one + // entry at a time meant a crash or a rejected path partway through left a + // durable config matching no policy anyone wrote. Only MXC's own entries + // are replaced -- baseline mounts the distribution template or the operator + // placed in the config carry no marker and survive. let mounts = resolve_mount_order(policy); + let mut entries: Vec = Vec::with_capacity(mounts.len()); // Container-side paths of every re-bound (rw/ro) mount, used to decide // whether a denied *directory* must be masked with a writable tmpfs so a @@ -236,7 +244,7 @@ pub fn configure_filesystem_mounts( "Adding rw bind mount: {} -> /{}", host_path, container_path )); - container.set_config_item("lxc.mount.entry", &mount_entry)?; + entries.push(mount_entry); } FsIntent::ReadOnly => { let mount_entry = format!( @@ -247,7 +255,7 @@ pub fn configure_filesystem_mounts( "Adding ro bind mount: {} -> /{}", host_path, container_path )); - container.set_config_item("lxc.mount.entry", &mount_entry)?; + entries.push(mount_entry); } FsIntent::Denied => { // Resolve the denied path through symlinks to its real host @@ -301,12 +309,12 @@ pub fn configure_filesystem_mounts( "Masking denied path: /{} ({})", container_path, create_type )); - container.set_config_item("lxc.mount.entry", &mount_entry)?; + entries.push(mount_entry); } } } - Ok(()) + container.replace_mxc_mount_entries(&entries) } /// Remove filesystem mount configuration. @@ -357,6 +365,55 @@ mod tests { assert!(validate_path("/mnt/shared").is_ok()); } + #[test] + fn configure_filesystem_mounts_replaces_not_accumulates() { + use wxc_common::logger::Mode; + + // Real config file so set_config_item/clear_config_item operate on disk. + let base = std::env::temp_dir().join(format!( + "mxc-fs-mounts-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0) + )); + let name = "box"; + std::fs::create_dir_all(base.join(name)).unwrap(); + let container = LxcContainer::new(name, Some(base.to_str().unwrap())); + std::fs::write(container.config_file_path(), "lxc.arch = amd64\n").unwrap(); + + let mut logger = Logger::new(Mode::Buffer); + + // First start: broad policy binds /host/broad. + let policy_a = ContainerPolicy { + readwrite_paths: vec!["/host/broad".to_string()], + ..Default::default() + }; + configure_filesystem_mounts(&container, &policy_a, &mut logger).unwrap(); + + // Second start (simulated restart): narrower policy binds /host/narrow. + let policy_b = ContainerPolicy { + readonly_paths: vec!["/host/narrow".to_string()], + ..Default::default() + }; + configure_filesystem_mounts(&container, &policy_b, &mut logger).unwrap(); + + let cfg = std::fs::read_to_string(container.config_file_path()).unwrap(); + assert!( + !cfg.contains("/host/broad"), + "restart must not inherit the previous run's mount, got:\n{cfg}" + ); + assert!( + cfg.contains("/host/narrow"), + "restart must apply the new policy's mount, got:\n{cfg}" + ); + // Non-mount config lines are preserved across the clear/reapply. + assert!(cfg.contains("lxc.arch = amd64")); + + let _ = std::fs::remove_dir_all(&base); + } + #[test] fn has_rebound_descendant_detects_nested_rebind_only() { let rebound = vec![ diff --git a/src/backends/lxc/common/src/lib.rs b/src/backends/lxc/common/src/lib.rs index 3dea29501..7852a440b 100644 --- a/src/backends/lxc/common/src/lib.rs +++ b/src/backends/lxc/common/src/lib.rs @@ -8,3 +8,4 @@ pub mod lxc_runner; pub mod network_ingress; pub mod network_iptables; pub mod signal_cleanup; +pub mod state_aware; diff --git a/src/backends/lxc/common/src/lxc_bindings.rs b/src/backends/lxc/common/src/lxc_bindings.rs index e32ded9fe..b9839640d 100644 --- a/src/backends/lxc/common/src/lxc_bindings.rs +++ b/src/backends/lxc/common/src/lxc_bindings.rs @@ -68,18 +68,121 @@ pub fn resolve_default_lxcpath() -> String { resolve_lxcpath_with_env(|k| std::env::var(k).ok(), current_euid) } +/// Environment variable stamped on every process an exec starts inside the +/// container, so a timeout can find its descendants and kill them. +/// +/// A timeout tears down `lxc-attach` on the host, which says nothing about the +/// processes the script started inside the container's PID namespace. The +/// container is persistent — it outlives the exec and lives until deprovision +/// — so those survivors keep holding CPU, memory, handles, and network inside a +/// sandbox the caller believes is idle, and the next exec shares the container +/// with them. +/// +/// The marker rides on the environment, which every `fork`/`exec` inherits, so +/// it reaches descendants at any depth. But the environment belongs to the +/// workload: anything that scrubs it escapes, so this cleans up after work that +/// is not trying to escape and is not a boundary against work that is. +/// `lxc-attach` joins the container's existing namespaces via `setns` and cannot +/// create one, so a per-exec PID namespace would have to be unshared by the +/// attached command itself — real, but a design that needs a live host to +/// validate, so #871 carries it rather than this improvising one. +/// +/// The name is **reserved**: `build_attach_args` drops any caller-supplied +/// entry that uses it, so a caller cannot set it to another exec's token and be +/// reaped by that exec's timeout. +#[cfg(any(target_os = "linux", test))] +const EXEC_MARKER_VAR: &str = "MXC_EXEC_ID"; + +/// Mint a token for one exec, unique to this process and this call. +/// +/// Uniqueness matters in both directions: two execs in the same container must +/// not reap each other, and a stale token from an earlier process must not +/// match anything live. +pub fn mint_exec_marker() -> String { + format!( + "{}-{}", + std::process::id(), + wxc_common::id::mint_random_token() + ) +} + +/// Build the post-binary argv for the `lxc-attach` that reaps an exec's +/// leftovers, given the marker value that exec was stamped with. +/// +/// The marker travels as a positional argument rather than being spliced into +/// the script, so nothing in it can be read as shell syntax. Only `cat` and +/// `kill` are used beyond shell builtins: `grep`'s `-a` and `-F` flags are not +/// dependable across the busybox and GNU userlands MXC containers are built +/// from. Command substitution drops the NULs that separate `environ` entries, +/// which concatenates neighbors but leaves each `KEY=VALUE` intact, so the +/// substring test still holds. +/// +/// The reaping shell is attached *without* the marker, so it cannot match +/// itself. +/// +/// `/proc/[0-9]*` is expanded once per pass, so a workload that forks after +/// the expansion has a child the pass never sees. The scan therefore stops +/// what it finds before killing anything: a stopped process cannot fork again, +/// so each pass shrinks the set that is still able to grow, and the loop +/// repeats until a pass discovers no marked process it had not already seen. +/// Only then is the collected set killed. The pass count is bounded so a +/// deliberate fork bomb cannot spin here forever. +/// +/// That is containment by convergence, not by construction. A per-exec cgroup +/// or PID namespace would make it race-free outright, and `lxc-attach` offers +/// neither. +#[cfg(any(target_os = "linux", test))] +fn build_reap_args(marker: &str) -> Vec { + vec![ + "--".to_string(), + "/bin/sh".to_string(), + "-c".to_string(), + "seen=\" \"; i=0; \ + while [ \"$i\" -lt 8 ]; do \ + found=0; \ + for d in /proc/[0-9]*; do \ + p=${d#/proc/}; \ + case \"$seen\" in *\" $p \"*) continue ;; esac; \ + e=$(cat \"$d/environ\" 2>/dev/null) || continue; \ + case \"$e\" in *\"$1\"*) \ + kill -STOP \"$p\" 2>/dev/null; \ + seen=\"$seen$p \"; \ + found=1 ;; \ + esac; \ + done; \ + [ \"$found\" -eq 0 ] && break; \ + i=$((i + 1)); \ + done; \ + for p in $seen; do kill -KILL \"$p\" 2>/dev/null; done; \ + exit 0" + .to_string(), + "_".to_string(), + format!("{}={}", EXEC_MARKER_VAR, marker), + ] +} + /// The keep-env argv shape, for tests that do not exercise env control. /// /// No production caller wants it, so outside `cfg(test)` this is dead code, /// and the workspace clippy lane runs with `-D warnings`. #[cfg(test)] -fn build_attach_args(env: &[String], working_directory: &str, command: &str) -> Vec { - build_attach_args_with_env_control(env, working_directory, command, false) +fn build_attach_args( + env: &[String], + working_directory: &str, + command: &str, + marker: Option<&str>, +) -> Vec { + build_attach_args_with_env_control(env, working_directory, command, false, marker) } /// Build the post-binary argv for `lxc-attach` (the args that follow the /// `-n NAME -P lxcpath` flags already appended by `lxc_command`). /// +/// `marker`, when present, is stamped into the child's environment as +/// [`EXEC_MARKER_VAR`] so a timeout can locate the whole process tree later; +/// see [`build_reap_args`]. It is only supplied when the caller set a timeout, +/// so the no-timeout argv is unchanged. +/// /// Extracted so the env / cwd / command layering is unit-testable without /// actually spawning `lxc-attach`. See [`LxcContainer::attach_run`] for /// the full contract. @@ -98,6 +201,7 @@ fn build_attach_args_with_env_control( working_directory: &str, command: &str, force_clear_env: bool, + marker: Option<&str>, ) -> Vec { // Loose upper bound; realloc-avoidance hint only. let mut args: Vec = Vec::with_capacity(env.len() + 8); @@ -113,13 +217,26 @@ fn build_attach_args_with_env_control( // `"BADENTRY"` are both silently skipped; embedded `=` in // VAL is fine because split_once stops at the first one. if let Some((key, _)) = kv.split_once('=') { - if !key.is_empty() { + // `EXEC_MARKER_VAR` is reserved. Dropping a caller's copy is + // not just tidiness: the marker is what a timeout kills by, so + // a caller that set this name to another exec's token would be + // reaped by that exec. The drop is unconditional because a + // concurrent exec's timeout can reap this one even when this + // one carries no marker of its own. + if !key.is_empty() && key != EXEC_MARKER_VAR { args.push(format!("--set-var={}", kv)); } } } } + // Stamped after the caller's entries so `--clear-env` still leads, and + // outside the `env.is_empty()` gate so a caller that set no env still gets + // a reapable exec. + if let Some(token) = marker { + args.push(format!("--set-var={}={}", EXEC_MARKER_VAR, token)); + } + args.push("--".to_string()); args.push("/bin/sh".to_string()); args.push("-c".to_string()); @@ -141,6 +258,174 @@ fn build_attach_args_with_env_control( args } +/// What liblxc says the container's network interfaces are. +/// +/// Read through `lxc-info -c` rather than by parsing the container's config +/// file, because `lxc.include` can declare interfaces that file never mentions. +/// liblxc has already resolved those includes, so its answer is the set that +/// will actually be brought up; a count taken from the file alone would +/// understate the container and let a policy claim to be enforced when it is +/// not. +#[derive(Debug, Default, PartialEq, Eq)] +pub struct NetInterfaceConfig { + /// How many interfaces liblxc will bring up. + pub count: usize, + /// The declared type of the container's only interface, present only when + /// `count == 1`. + /// + /// Enforcement refuses anything that is not a `veth`, so the type is read + /// to decide that. It comes from the same `lxc.net` read as the count, so + /// no separate indexed probe is needed and there is no window between + /// reading the count and reading the type. + pub sole_kind: Option, +} + +/// Interpret an `lxc-info -c ` answer as the values liblxc holds for that +/// key. +/// +/// liblxc prints the first value on a `key = value` line and any further values +/// bare on lines of their own. Each line is trimmed; a line beginning with +/// `key` followed by optional whitespace and `=` yields what follows, and any +/// other line is taken whole. Results are trimmed and empties dropped, so an +/// absent or empty key yields no values. +/// +/// The prefix is keyed on `key` rather than split on the first `=` anywhere in +/// the line: a continuation value can itself contain `=` (a hook command does), +/// and splitting on `=` would truncate it. +fn interpret_config_values(key: &str, stdout: &str) -> Vec { + stdout + .lines() + .map(|line| line.trim()) + .map(|line| { + line.strip_prefix(key) + .map(|rest| rest.trim_start()) + .and_then(|rest| rest.strip_prefix('=')) + .map_or(line, |value| value) + .trim() + }) + .filter(|value| !value.is_empty()) + .map(|value| value.to_string()) + .collect() +} + +/// Build the `lxc.hook.start-host` value that runs the veth-pin script. +/// +/// The script takes the desired host veth name as its sole argument, so the +/// value is the script path and the target name separated by a space. +fn veth_pin_hook_command(script_path: &str, target_veth: &str) -> String { + format!("{} {}", script_path, target_veth) +} + +/// Shell body of the `lxc.hook.start-host` hook that renames the container's +/// host-side veth to the name passed as `$1`. +/// +/// It finds the container's sole peered interface (a veth in the netns prints +/// `eth0@if`; `lo` has no `@if`, so it is naturally excluded), resolves that +/// host ifindex to its current name, and renames it. It is idempotent and fails +/// closed. +const VETH_PIN_SCRIPT: &str = r#"#!/bin/sh +set -e +target="$1" +i=$(nsenter -t "$LXC_PID" -n ip -o link | sed -n 's/^[0-9]*: [^:@]*@if\([0-9]*\):.*/\1/p' | head -n1) +[ -n "$i" ] || { echo "mxc: no peered interface in container netns" >&2; exit 1; } +c=$(ip -o link | sed -n "s/^$i: \([^:@]*\)[@:].*/\1/p" | head -n1) +[ -n "$c" ] || { echo "mxc: host ifindex $i not resolvable" >&2; exit 1; } +[ "$c" = "$target" ] || ip link set "$c" name "$target" +"#; + +/// Read an `lxc-info` run as "does this container exist?". +/// +/// Split out as a pure function so the three-way answer is testable without +/// `lxc-info` on the box. +/// +/// A nonzero exit is ambiguous on its own: `lxc-info` reports a container it +/// does not know that way, but so do a permission error, a transient runtime +/// failure, and a malformed config. "Defined" means LXC has a config file for +/// the container, so that file settles the ambiguity, and `try_exists` reports +/// `false` only when it can prove absence -- a directory it cannot read is an +/// `Err`, not a `false`. +/// +/// The layout assumption is `{lxc_path}/{name}/config`. If that is ever wrong +/// the failure lands on `Ok(true)` for the config probe and therefore on `Err` +/// here, which refuses the phase; the old code returned `Ok(false)` and +/// unfiltered a live container. Wrong in the safe direction. +fn interpret_defined_probe( + probe_succeeded: bool, + stderr: &str, + config: &std::path::Path, + name: &str, +) -> Result { + if probe_succeeded { + return Ok(true); + } + let detail = stderr.trim(); + match config.try_exists() { + Ok(false) => Ok(false), + Ok(true) => Err(format!( + "lxc-info failed for container {name:?} but its config file at {} is present, so \ + whether the container is defined is unknown: {detail}", + config.display() + )), + Err(e) => Err(format!( + "lxc-info failed for container {name:?} and its config file at {} could not be \ + checked ({e}), so whether the container is defined is unknown: {detail}", + config.display() + )), + } +} + +/// The `lxc-info -s` states that still have a live container behind them. +/// +/// `STOPPED` is the only state that does not. `FROZEN` and `FREEZING` have +/// processes that thaw straight back into a running container, so unfiltering +/// one is the same fail-open as unfiltering a running one. The transitional +/// states count as live for the same reason: stopping a container that is +/// already going down is harmless, and unfiltering one that is coming up is +/// not. +const LIVE_STATES: [&str; 7] = [ + "RUNNING", "FROZEN", "FREEZING", "THAWED", "STARTING", "STOPPING", "ABORTING", +]; + +/// Read `lxc-info -s` output as "is this container running?". +/// +/// Split out as a pure function so the three-way answer is testable without a +/// container on the box. Only a `State:` line answers the question; anything +/// else is an error rather than `false`, because callers treat a stopped +/// container as safe to unfilter and safe to skip stopping. Guessing +/// "stopped" from output we could not read is the one answer that turns a +/// broken probe into an unfiltered running container. +fn interpret_state_output(stdout: &str) -> Result { + for line in stdout.lines() { + let Some((key, value)) = line.split_once(':') else { + continue; + }; + if key.trim() == "State" { + let state = value.trim(); + if state == "STOPPED" { + return Ok(false); + } + if LIVE_STATES.contains(&state) { + return Ok(true); + } + return Err(format!( + "lxc-info -s named an unrecognized state {state:?}, so whether the container is \ + running is unknown" + )); + } + } + // No `State:` line. If a live-state name shows up anyway, answer in the + // safe direction rather than give up: reporting "running" only ever costs a + // refused operation, whereas reporting "stopped" is what unfilters a live + // container. + if LIVE_STATES.iter().any(|s| stdout.contains(s)) { + return Ok(true); + } + Err(format!( + "lxc-info -s named no state, so whether the container is running is unknown (output: {:?})", + stdout.trim() + )) +} + /// Safe wrapper around an LXC container. pub struct LxcContainer { name: String, @@ -201,19 +486,48 @@ impl LxcContainer { Ok(()) } - /// Check if the container exists. - pub fn is_defined(&self) -> bool { - let output = self.lxc_command("lxc-info").output(); - matches!(output, Ok(o) if o.status.success()) + /// Whether the container exists. + /// + /// `Err` means the probe could not answer, which is not evidence of + /// absence. Collapsing that into `false` reads "the probe broke" as "the + /// container is gone", which let deprovision skip the destroy and then + /// strip the firewall from a container that was still running. + /// + /// See [`interpret_defined_probe`] for why a nonzero exit is not an answer + /// on its own. + pub fn is_defined(&self) -> Result { + let output = self + .lxc_command("lxc-info") + .output() + .map_err(|e| format!("failed to run lxc-info: {e}"))?; + let config = self.config_file_path(); + interpret_defined_probe( + output.status.success(), + &String::from_utf8_lossy(&output.stderr), + std::path::Path::new(&config), + &self.name, + ) } - /// Check if the container is running. - pub fn is_running(&self) -> bool { - let output = self.lxc_command("lxc-info").arg("-s").output(); - match output { - Ok(o) => String::from_utf8_lossy(&o.stdout).contains("RUNNING"), - Err(_) => false, + /// Whether the container is running. + /// + /// `Err` covers both a probe that could not run and one whose output names + /// no state we recognize. Neither is evidence that the container is + /// stopped, and callers treat "stopped" as safe to unfilter or safe to + /// skip stopping -- so an unreadable probe must not answer `false`. + pub fn is_running(&self) -> Result { + let output = self + .lxc_command("lxc-info") + .arg("-s") + .output() + .map_err(|e| format!("failed to run lxc-info -s: {e}"))?; + if !output.status.success() { + return Err(format!( + "lxc-info -s failed: {}", + String::from_utf8_lossy(&output.stderr).trim() + )); } + interpret_state_output(&String::from_utf8_lossy(&output.stdout)) } /// Return the PID of the container's init process, or `None` if the @@ -254,6 +568,14 @@ impl LxcContainer { Self::run_status(cmd, "lxc-create") } + /// Marker comment written immediately above every `lxc.mount.entry` line + /// that MXC itself adds, so + /// [`replace_mxc_mount_entries`](Self::replace_mxc_mount_entries) can + /// rewrite only MXC's own mounts and leave baseline entries the distro + /// template or the user placed in the config untouched. It is a real LXC + /// comment (`#`), so liblxc ignores it when parsing the file. + const MXC_MOUNT_MARKER: &'static str = "# mxc-managed-mount"; + /// Set a configuration item on the container. /// /// Appends `key = value` to the container's config file. The error @@ -279,6 +601,276 @@ impl LxcContainer { }) } + /// Replace the container's config file with `contents` in one atomic step. + /// + /// [`std::fs::write`] truncates in place, so a signal, crash, or OOM between + /// the truncate and the last byte leaves the container's durable config + /// partial or empty. liblxc re-reads that file on every start, so a + /// half-written rewrite silently drops the entries a tightened policy + /// depends on -- the failure lands on the next start, far from the write + /// that caused it. Writing a sibling temporary and renaming it over the + /// target makes the swap atomic: a concurrent reader observes either the + /// whole old config or the whole new one, never a truncated prefix. + /// + /// The temporary is created beside the target so the rename stays inside one + /// filesystem, and it is flushed before the rename so the bytes are durable + /// before anything points at them. It carries the process id so two + /// processes rewriting the same config cannot clobber each other's + /// temporary, and it is removed on every failure path so a failed rewrite + /// leaves no residue. + /// + /// A rename swaps in a *new* inode, so the target's mode and ownership are + /// whatever the temporary had rather than what the operator set. That would + /// silently relax a hardened `0600` root-owned config to a umask-derived + /// `0644` on the first start, and would hand the file to the executor's uid + /// when it runs as root. The original's metadata is therefore captured + /// before the write and restored onto the temporary before the rename. The + /// temporary is opened `0600` so its contents are never briefly readable by + /// anyone the final mode would exclude; when there is no original to mirror, + /// the platform default is left alone rather than a policy being invented. + fn write_config_atomically(config_path: &str, contents: &str) -> std::io::Result<()> { + use std::io::Write; + + let temp_path = format!("{}.mxc-tmp-{}", config_path, std::process::id()); + let original = std::fs::metadata(config_path).ok(); + let write_temp = || -> std::io::Result<()> { + let mut file = Self::create_config_temp(&temp_path, original.is_some())?; + file.write_all(contents.as_bytes())?; + file.sync_all() + }; + if let Err(e) = write_temp() { + let _ = std::fs::remove_file(&temp_path); + return Err(e); + } + if let Some(ref meta) = original { + if let Err(e) = Self::mirror_config_metadata(&temp_path, meta) { + let _ = std::fs::remove_file(&temp_path); + return Err(e); + } + } + if let Err(e) = std::fs::rename(&temp_path, config_path) { + let _ = std::fs::remove_file(&temp_path); + return Err(e); + } + Ok(()) + } + + /// Open the rewrite temporary, restricted to the owner when there is an + /// existing config whose mode will be restored before the rename. + #[cfg(unix)] + fn create_config_temp(temp_path: &str, has_original: bool) -> std::io::Result { + use std::os::unix::fs::OpenOptionsExt; + + if !has_original { + return std::fs::File::create(temp_path); + } + std::fs::OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .mode(0o600) + .open(temp_path) + } + + #[cfg(not(unix))] + fn create_config_temp(temp_path: &str, _has_original: bool) -> std::io::Result { + std::fs::File::create(temp_path) + } + + /// Put the replaced config's mode and ownership onto its replacement. + /// + /// Ownership is restored only when it actually differs, so an unprivileged + /// executor rewriting a config it already owns is not failed by a `chown` + /// it never needed permission to make. + #[cfg(unix)] + fn mirror_config_metadata( + temp_path: &str, + original: &std::fs::Metadata, + ) -> std::io::Result<()> { + use std::os::unix::fs::MetadataExt; + use std::os::unix::fs::PermissionsExt; + + // Masked to the permission bits: the mode read back from a `Metadata` + // also carries the file-type bits, which are not this call's to set. + std::fs::set_permissions( + temp_path, + std::fs::Permissions::from_mode(original.mode() & 0o7777), + )?; + let temp_meta = std::fs::metadata(temp_path)?; + if temp_meta.uid() != original.uid() || temp_meta.gid() != original.gid() { + std::os::unix::fs::chown(temp_path, Some(original.uid()), Some(original.gid())) + .map_err(|e| { + std::io::Error::new( + e.kind(), + format!( + "could not restore the config's owner {}:{} onto its replacement, so \ + the rewrite was abandoned rather than silently changing who owns it: \ + {e}", + original.uid(), + original.gid() + ), + ) + })?; + } + Ok(()) + } + + #[cfg(not(unix))] + fn mirror_config_metadata( + temp_path: &str, + original: &std::fs::Metadata, + ) -> std::io::Result<()> { + std::fs::set_permissions(temp_path, original.permissions()) + } + + /// Remove every configuration line for `key` from the container's config + /// file. + /// + /// [`set_config_item`](Self::set_config_item) *appends* a `key = value` + /// line, and list-type keys such as `lxc.mount.entry` accumulate one line + /// per call. liblxc replays every occurrence when it parses the file at + /// start, so a caller that re-derives a list from policy on each start must + /// clear the previous run's lines first — otherwise a restart inherits + /// stale entries (e.g. mounts a tightened policy meant to drop). + /// + /// A line matches when the token before its first `=` (trimmed) equals + /// `key`, so `lxc.mount.entry` is matched but neighbouring keys like + /// `lxc.mount` are left intact, and `=` inside a value (e.g. + /// `create=dir`) is irrelevant. A missing config file is treated as + /// already-clear (`Ok`). + pub fn clear_config_item(&self, key: &str) -> Result<(), String> { + let config_path = self.config_file_path(); + let contents = match std::fs::read_to_string(&config_path) { + Ok(c) => c, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), + Err(e) => { + return Err(format!( + "Failed to read config to clear {}: {} (config file: {})", + key, e, config_path + )) + } + }; + + let mut out = String::with_capacity(contents.len()); + for line in contents.lines() { + let matches_key = line + .split_once('=') + .map(|(lhs, _)| lhs.trim() == key) + .unwrap_or(false); + if !matches_key { + out.push_str(line); + out.push('\n'); + } + } + + Self::write_config_atomically(&config_path, &out).map_err(|e| { + format!( + "Failed to rewrite config to clear {}: {} (config file: {})", + key, e, config_path + ) + }) + } + + /// Replace MXC's whole mount set in one atomic config rewrite. + /// + /// Each entry is written as [`MXC_MOUNT_MARKER`](Self::MXC_MOUNT_MARKER) on + /// its own line followed by `lxc.mount.entry = value`. liblxc treats the + /// marker as a comment and the entry exactly as if it had been added with + /// [`set_config_item`](Self::set_config_item). + /// + /// The set is the unit that matters: a container configured with half of + /// its policy's bind mounts is not a weaker sandbox, it is a different one. + /// Clearing and then appending each entry separately committed a config + /// per mount, so a crash, a signal, or a rejected path partway through left + /// a durable config that matched no policy anyone wrote. One rewrite means + /// a reader sees either the previous run's mounts or this run's, never a + /// prefix of this run's. + /// + /// Only MXC's own entries are replaced. Each is tagged with + /// [`MXC_MOUNT_MARKER`](Self::MXC_MOUNT_MARKER) on the line above it, which + /// liblxc treats as a comment; foreign `lxc.mount.entry` lines placed by the + /// distribution template or the operator carry no marker and survive + /// untouched. Clearing those instead would silently detach container + /// storage nobody asked us to manage. + /// + /// A missing config file is already free of MXC mounts, so an empty set + /// succeeds against one. A non-empty set does not: writing mount entries + /// into a config that liblxc never created would produce a container + /// definition with no template behind it. + pub fn replace_mxc_mount_entries(&self, values: &[String]) -> Result<(), String> { + let config_path = self.config_file_path(); + let contents = match std::fs::read_to_string(&config_path) { + Ok(c) => c, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + if values.is_empty() { + return Ok(()); + } + return Err(format!( + "Failed to set {} MXC mount entries: no config file at {}", + values.len(), + config_path + )); + } + Err(e) => { + return Err(format!( + "Failed to read config to rewrite MXC mounts: {} (config file: {})", + e, config_path + )) + } + }; + + let mut out = Self::strip_mxc_mount_entries(&contents); + for value in values { + out.push_str(Self::MXC_MOUNT_MARKER); + out.push('\n'); + out.push_str("lxc.mount.entry = "); + out.push_str(value); + out.push('\n'); + } + + Self::write_config_atomically(&config_path, &out).map_err(|e| { + format!( + "Failed to rewrite config with {} MXC mount entries: {} (config file: {})", + values.len(), + e, + config_path + ) + }) + } + + /// `contents` with every MXC-added mount line removed. + /// + /// A marker line and the `lxc.mount.entry` line immediately following it are + /// dropped together. An orphaned marker -- one left by a config written + /// before the rewrite became atomic -- is dropped on its own so stray + /// comments cannot accumulate. + /// + /// Pure so the line-pairing is testable without a container on the box. + fn strip_mxc_mount_entries(contents: &str) -> String { + let lines: Vec<&str> = contents.lines().collect(); + let mut out = String::with_capacity(contents.len()); + let mut i = 0; + while i < lines.len() { + let line = lines[i]; + if line.trim() == Self::MXC_MOUNT_MARKER { + let next_is_entry = lines + .get(i + 1) + .map(|l| { + l.split_once('=') + .map(|(lhs, _)| lhs.trim() == "lxc.mount.entry") + .unwrap_or(false) + }) + .unwrap_or(false); + i += if next_is_entry { 2 } else { 1 }; + continue; + } + out.push_str(line); + out.push('\n'); + i += 1; + } + out + } + /// Start the container. pub fn start(&self) -> Result<(), String> { Self::run_status(self.lxc_command("lxc-start"), "lxc-start") @@ -358,6 +950,12 @@ impl LxcContainer { /// /// `timeout: Some(d)` kills the child if it runs longer than `d` and /// returns `Err("script timed out after {ms}ms")`. + /// + /// `marker: Some(token)` stamps the exec so its container-side processes + /// can be reaped later; see [`mint_exec_marker`]. The caller owns the + /// token because a timeout is not the only way an exec ends early -- a + /// signal kills this process outright, and the watchdog needs the same + /// token to reap on its way out. #[cfg(target_os = "linux")] pub fn attach_run( &self, @@ -366,6 +964,7 @@ impl LxcContainer { env: &[String], force_clear_env: bool, timeout: Option, + marker: Option<&str>, ) -> Result<(i32, String, String), String> { use mxc_pty::{run_with_pty, PtyOptions, PtyOutcome, Signal}; @@ -377,6 +976,7 @@ impl LxcContainer { working_directory, command, force_clear_env, + marker, )); let options = PtyOptions { @@ -392,11 +992,60 @@ impl LxcContainer { PtyOutcome::TimedOut => { let ms = timeout.map(|d| d.as_millis()).unwrap_or(0); + + // Killing lxc-attach ended the caller's view of the work, not + // the work. Reap before reporting, and if the reap fails say + // so: a bare timeout message would tell the caller the script + // stopped when it may still be running. A reap that succeeds + // is not a containment guarantee either — see + // `reap_marked_processes` for what escapes it. + if let Some(token) = marker { + if let Err(e) = self.reap_marked_processes(token) { + return Err(format!( + "script timed out after {}ms, and its processes could not be \ + reaped from the container, so they may still be running: {}", + ms, e + )); + } + } + Err(format!("script timed out after {}ms", ms)) } } } + /// Kill every process in the container whose environment carries `marker`. + /// + /// Called on timeout by [`attach_run`](Self::attach_run) and on a fatal + /// signal by the cleanup watchdog. Reaching into the container's PID + /// namespace requires a second attach; the alternative the issue raised — + /// stopping and restarting the container — would discard the ingress chain + /// that lives in its network namespace and the rest of the start-time + /// enforcement, so it trades an orphaned process for an unfiltered sandbox. + /// + /// The guarantee is exactly what the sentence above says and no more: this + /// reaps processes *carrying the marker*, not every descendant of the exec. + /// The marker is inherited across `fork`/`exec`, so it reaches descendants + /// at any depth — but the environment belongs to the workload, and a + /// workload that scrubs it (`env -i`, `env -u MXC_EXEC_ID`, an explicit + /// `unsetenv`) drops off the list and survives the reap. Returning `Ok(())` + /// means every *marked* process was killed, not that the container is quiet. + /// + /// **This is hygiene, not containment.** Contained code is untrusted, so a + /// handle the workload can erase is not a boundary against it — it only + /// cleans up after work that is not trying to escape, which is the case that + /// actually leaks into the next exec today. A boundary needs kernel-owned + /// membership the workload cannot leave (a per-exec PID namespace or + /// cgroup); #871 carries that design and the host-dependent questions it + /// has to answer first, because a containment mechanism that silently fails + /// is worse than one documented not to be one. + #[cfg(target_os = "linux")] + pub(crate) fn reap_marked_processes(&self, marker: &str) -> Result<(), String> { + let mut cmd = self.lxc_command("lxc-attach"); + cmd.args(build_reap_args(marker)); + Self::run_status(cmd, "lxc-attach (reap)") + } + /// Stub for the workspace-wide clippy lane that runs on Windows. #[cfg(not(target_os = "linux"))] pub fn attach_run( @@ -406,15 +1055,39 @@ impl LxcContainer { _env: &[String], _force_clear_env: bool, _timeout: Option, + _marker: Option<&str>, ) -> Result<(i32, String, String), String> { Err("LxcContainer::attach_run is only supported on Linux".to_string()) } /// Stop the container. + /// + /// Graceful: `lxc-stop` asks init to shut down and waits. That is right for + /// an explicit lifecycle stop, and wrong for every rollback -- see + /// [`kill`](Self::kill). pub fn stop(&self) -> Result<(), String> { Self::run_status(self.lxc_command("lxc-stop"), "lxc-stop") } + /// Stop the container immediately, without waiting for a graceful shutdown. + /// + /// `lxc-stop` on its own waits up to 60 s for init to respond, and on + /// distros running systemd as PID 1 in an unprivileged userns init never + /// cleanly responds to SIGPWR at all -- so the wait can be the full timeout + /// and the stop can still fail. + /// + /// That is merely slow when a caller asked to stop a sandbox. It is a hole + /// when a rollback is stopping a container *because its isolation is not in + /// force*: the container keeps running, and keeps accepting traffic, for as + /// long as the graceful stop takes. Rollback paths use this instead, so the + /// exposure ends now rather than after a shutdown negotiation the guest can + /// decline. + pub fn kill(&self) -> Result<(), String> { + let mut cmd = self.lxc_command("lxc-stop"); + cmd.arg("-k"); + Self::run_status(cmd, "lxc-stop -k") + } + /// Destroy the container (removes rootfs and config). /// /// `lxc-destroy -f` already force-stops a running container; we used to @@ -430,10 +1103,128 @@ impl LxcContainer { } /// Get the path to the container's config file. - fn config_file_path(&self) -> String { + pub(crate) fn config_file_path(&self) -> String { format!("{}/{}/config", self.lxc_path, self.name) } + /// What liblxc says this container's network interfaces are. + /// + /// Provision adopts an existing container as readily as it creates one, and + /// an adopted container can carry more network interfaces than the single + /// `lxc.net.0` MXC configures for itself. A caller that filters egress needs + /// to know that before it claims to have filtered anything. + /// + /// The question goes to liblxc rather than to the config file because + /// `lxc.include` can add interfaces the file never mentions, and resolving + /// includes here -- relative paths and directory globs both -- would mean + /// reimplementing liblxc's own resolution and getting it subtly wrong. + /// liblxc answers for a stopped container, so the answer is available while + /// there is still time to act on it before start. + /// + /// The count and the sole interface's type both come from a single + /// `lxc.net` read, so there is no longer a window between reading the count + /// and reading the interface. This is not atomic with respect to the start + /// itself -- the config could still change before liblxc reads it -- only + /// with respect to this pair of observations. + /// + /// A probe that cannot run is an error rather than an empty answer: no + /// evidence of an interface is not evidence of no interface. + pub fn configured_net_interfaces(&self) -> Result { + let values = interpret_config_values("lxc.net", &self.query_config_item("lxc.net")?); + let count = values.len(); + // The type only matters when there is exactly one interface; every + // other count is refused upstream without reference to it. + let sole_kind = if count == 1 { + values.into_iter().next() + } else { + None + }; + Ok(NetInterfaceConfig { count, sole_kind }) + } + + /// Install the `lxc.hook.start-host` hook that pins the container's + /// host-side veth to `target_veth`. + /// + /// The hook runs after liblxc has created the veth pair and attached it to + /// the bridge but before the container's init execs, so the deterministic + /// name is in place before anything in the container can transmit. The hook + /// key is container-global -- it carries no `lxc.net.` index -- so + /// enforcement no longer depends on which index the interface uses. + /// + /// The script is written fresh every time so it cannot go stale, and made + /// executable. The hook entry is appended only when an identical one is not + /// already present; the key is never cleared, because a container's own + /// config may declare start-host hooks that clearing would destroy. + pub fn ensure_veth_pin_hook(&self, target_veth: &str) -> Result<(), String> { + let script_path = format!("{}/{}/mxc-veth-pin.sh", self.lxc_path, self.name); + + std::fs::write(&script_path, VETH_PIN_SCRIPT) + .map_err(|e| format!("Failed to write veth pin hook script {script_path}: {e}"))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + std::fs::set_permissions(&script_path, std::fs::Permissions::from_mode(0o755)) + .map_err(|e| format!("Failed to chmod veth pin hook script {script_path}: {e}"))?; + } + + if self.has_veth_pin_hook(target_veth)? { + return Ok(()); + } + self.set_config_item( + "lxc.hook.start-host", + &veth_pin_hook_command(&script_path, target_veth), + ) + } + + /// Whether this container's own config already pins its veth to + /// `target_veth`. + /// + /// A container carrying this hook has had its host-side veth renamed by the + /// time its init execs, because the hook runs before that and a nonzero + /// exit aborts the start. So the answer decides which name describes the + /// live interface: with the hook, the pinned name; without it, whatever + /// liblxc recorded when it created the pair. + /// + /// This asks the container rather than the host on purpose. Asking the host + /// whether an interface of the pinned name exists answers a different + /// question -- it cannot tell this container's interface from a stranger's + /// that happens to hold the name, and it turns a transient failure of the + /// probe into the wrong answer rather than into an error. + /// + /// It reuses the comparison `ensure_veth_pin_hook` writes with, so the + /// reader and the writer cannot drift apart. + pub fn has_veth_pin_hook(&self, target_veth: &str) -> Result { + let script_path = format!("{}/{}/mxc-veth-pin.sh", self.lxc_path, self.name); + let value = veth_pin_hook_command(&script_path, target_veth); + let existing = interpret_config_values( + "lxc.hook.start-host", + &self.query_config_item("lxc.hook.start-host")?, + ); + Ok(existing.iter().any(|present| present == &value)) + } + + /// Ask liblxc for one config key, with any `lxc.include` already resolved. + /// + /// A key liblxc does not hold is not a failure -- it reports that on stderr + /// and exits zero, leaving stdout empty -- so only a nonzero exit is treated + /// as one. + fn query_config_item(&self, key: &str) -> Result { + let output = self + .lxc_command("lxc-info") + .arg("-c") + .arg(key) + .output() + .map_err(|e| format!("failed to run lxc-info -c {key}: {e}"))?; + if !output.status.success() { + return Err(format!( + "lxc-info -c {} failed: {}", + key, + String::from_utf8_lossy(&output.stderr).trim() + )); + } + Ok(String::from_utf8_lossy(&output.stdout).to_string()) + } + /// Get the current system architecture string for LXC templates. fn current_arch() -> &'static str { #[cfg(target_arch = "x86_64")] @@ -459,6 +1250,101 @@ mod tests { None } + #[test] + fn an_atomic_config_rewrite_replaces_the_file_and_leaves_no_temporary() { + let dir = std::env::temp_dir().join(format!("mxc-atomic-write-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("test temp dir"); + let target = dir.join("config"); + let path = target.to_string_lossy().to_string(); + + std::fs::write(&target, "lxc.mount.entry = old\nlxc.mount.entry = stale\n") + .expect("seed the original config"); + LxcContainer::write_config_atomically(&path, "lxc.mount.entry = new\n") + .expect("rewrite must succeed"); + + assert_eq!( + std::fs::read_to_string(&target).expect("read the rewritten config"), + "lxc.mount.entry = new\n", + "the rewrite must fully replace the previous contents" + ); + + // The swap must not leave its sibling behind. liblxc reads the config + // directory, and a surviving *.mxc-tmp-* is a second copy of a policy + // that was meant to be replaced, not merely litter. + let leftovers: Vec = std::fs::read_dir(&dir) + .expect("list the config directory") + .filter_map(|e| e.ok()) + .map(|e| e.file_name().to_string_lossy().to_string()) + .filter(|name| name != "config") + .collect(); + assert!( + leftovers.is_empty(), + "a successful write must leave no temporary, found: {:?}", + leftovers + ); + + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn an_atomic_config_rewrite_reports_failure_without_touching_the_original() { + // A directory standing where the config should be makes both the + // temporary create and the rename fail. The original must survive an + // unwritable target rather than being truncated on the way to an error, + // which is the whole reason the write does not go in place. + let dir = std::env::temp_dir().join(format!("mxc-atomic-fail-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(dir.join("config")).expect("test temp dir"); + + let err = LxcContainer::write_config_atomically( + &dir.join("config").to_string_lossy(), + "lxc.mount.entry = new\n", + ); + assert!(err.is_err(), "writing over a directory must report failure"); + assert!( + dir.join("config").is_dir(), + "the failed write must not have replaced the target" + ); + + let _ = std::fs::remove_dir_all(&dir); + } + + #[cfg(unix)] + #[test] + fn an_atomic_config_rewrite_keeps_the_operator_s_permissions() { + use std::os::unix::fs::PermissionsExt; + + // A rename swaps in a new inode, so without explicit restoration a + // hardened config silently relaxes to whatever the umask allows the + // first time a start rewrites it. An operator who set 0600 gets to keep + // it. + let dir = std::env::temp_dir().join(format!("mxc-atomic-mode-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("test temp dir"); + let target = dir.join("config"); + let path = target.to_string_lossy().to_string(); + + std::fs::write(&target, "lxc.mount.entry = old\n").expect("seed the original config"); + std::fs::set_permissions(&target, std::fs::Permissions::from_mode(0o600)) + .expect("harden the original config"); + + LxcContainer::write_config_atomically(&path, "lxc.mount.entry = new\n") + .expect("rewrite must succeed"); + + let mode = std::fs::metadata(&target) + .expect("stat the rewritten config") + .permissions() + .mode() + & 0o777; + assert_eq!( + mode, 0o600, + "the rewrite must not widen the config's permissions" + ); + + let _ = std::fs::remove_dir_all(&dir); + } + #[test] fn lxcpath_honors_lxc_path_env() { let p = resolve_lxcpath_with_env( @@ -588,6 +1474,334 @@ mod tests { assert!(!c.lxc_path().is_empty()); } + #[test] + fn one_interface_reports_one() { + assert_eq!( + interpret_config_values("lxc.net", "lxc.net = veth\n\n").len(), + 1 + ); + } + + #[test] + fn a_state_line_answers_whether_the_container_is_running() { + assert_eq!( + interpret_state_output("State: RUNNING\n"), + Ok(true) + ); + assert_eq!( + interpret_state_output("State: STOPPED\n"), + Ok(false) + ); + } + + #[test] + fn stripping_drops_mxc_mounts_and_keeps_baseline_ones() { + let config = concat!( + "lxc.uts.name = box\n", + "lxc.mount.entry = /srv /srv none bind 0 0\n", + "# mxc-managed-mount\n", + "lxc.mount.entry = /tmp/a a none bind,create=dir 0 0\n", + "lxc.rootfs.path = /var/lib/lxc/box/rootfs\n", + ); + let out = LxcContainer::strip_mxc_mount_entries(config); + assert!( + out.contains("/srv /srv none bind 0 0"), + "a baseline mount the operator placed must survive, got {out:?}" + ); + assert!( + !out.contains("/tmp/a"), + "MXC's own mount must go, got {out:?}" + ); + assert!(!out.contains("mxc-managed-mount"), "got {out:?}"); + assert!(out.contains("lxc.rootfs.path"), "got {out:?}"); + } + + #[test] + fn stripping_drops_a_marker_left_without_its_entry() { + // Configs written before the rewrite became atomic can hold a marker + // whose entry never landed. Left behind, those accumulate one stray + // comment per interrupted start. + let out = LxcContainer::strip_mxc_mount_entries("# mxc-managed-mount\nlxc.uts.name = b\n"); + assert_eq!(out, "lxc.uts.name = b\n"); + } + + #[test] + fn a_mount_set_replaces_the_previous_one_in_a_single_config() { + let dir = std::env::temp_dir().join(format!("mxc-mountset-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(dir.join("box")).expect("test temp dir"); + let config = dir.join("box").join("config"); + std::fs::write( + &config, + "lxc.uts.name = box\n# mxc-managed-mount\nlxc.mount.entry = /old old none bind 0 0\n", + ) + .expect("seed config"); + + let container = LxcContainer::new("box", Some(&dir.to_string_lossy())); + container + .replace_mxc_mount_entries(&[ + "/new new none bind,create=dir 0 0".to_string(), + "/two two none bind,ro,create=dir 0 0".to_string(), + ]) + .expect("rewrite must succeed"); + + let after = std::fs::read_to_string(&config).expect("read back"); + assert!( + !after.contains("/old"), + "the previous run's mounts must not accumulate, got {after:?}" + ); + assert!(after.contains("/new new"), "got {after:?}"); + assert!(after.contains("/two two"), "got {after:?}"); + assert!(after.contains("lxc.uts.name = box"), "got {after:?}"); + assert_eq!( + after.matches("# mxc-managed-mount").count(), + 2, + "one marker per entry, got {after:?}" + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn an_empty_mount_set_clears_the_previous_one() { + let dir = std::env::temp_dir().join(format!("mxc-mountclear-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(dir.join("box")).expect("test temp dir"); + let config = dir.join("box").join("config"); + std::fs::write( + &config, + "lxc.uts.name = box\n# mxc-managed-mount\nlxc.mount.entry = /old old none bind 0 0\n", + ) + .expect("seed config"); + + LxcContainer::new("box", Some(&dir.to_string_lossy())) + .replace_mxc_mount_entries(&[]) + .expect("an empty set is a valid policy"); + + let after = std::fs::read_to_string(&config).expect("read back"); + assert!(!after.contains("/old"), "got {after:?}"); + assert!(after.contains("lxc.uts.name = box"), "got {after:?}"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_successful_probe_answers_defined_without_touching_the_filesystem() { + // The common path: `lxc-info` knows the container, so the config file + // never needs consulting. + assert_eq!( + interpret_defined_probe( + true, + "", + std::path::Path::new("/nonexistent/box/config"), + "b" + ), + Ok(true) + ); + } + + #[test] + fn a_failed_probe_with_no_config_file_means_absent() { + let dir = std::env::temp_dir().join(format!("mxc-defined-{}", std::process::id())); + let _ = std::fs::create_dir_all(&dir); + let config = dir.join("no-such-container").join("config"); + assert_eq!( + interpret_defined_probe(false, "container not found", &config, "no-such-container"), + Ok(false) + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_failed_probe_with_a_config_file_present_is_unknown_rather_than_absent() { + // The fail-open this closes: a permission or transient error made + // deprovision skip `destroy()` and then authoritatively strip the + // firewall from a container that was still running. The config file + // proves the container is defined, so the honest answer is "unknown". + let dir = std::env::temp_dir().join(format!("mxc-defined-live-{}", std::process::id())); + let container = dir.join("box"); + std::fs::create_dir_all(&container).expect("temp dir"); + let config = container.join("config"); + std::fs::write(&config, "lxc.uts.name = box\n").expect("config file"); + + let answer = interpret_defined_probe(false, "permission denied", &config, "box"); + assert!( + answer.is_err(), + "a present config file must not read as absent, got {answer:?}" + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_frozen_container_is_live_not_stopped() { + // A frozen container still has processes; thawing resumes them. Reading + // it as stopped is what lets `stop` skip `lxc-stop`, tear the firewall + // down as authoritative, and leave a thawable container unfiltered. + assert_eq!(interpret_state_output("State: FROZEN\n"), Ok(true)); + assert_eq!( + interpret_state_output("State: FREEZING\n"), + Ok(true) + ); + assert_eq!(interpret_state_output("State: THAWED\n"), Ok(true)); + } + + #[test] + fn a_transitional_state_is_live_not_stopped() { + // STOPPED is the only state with nothing left to unfilter. Stopping a + // container that is already going down costs an idempotent `lxc-stop`; + // unfiltering one that is coming up costs the isolation. + assert_eq!( + interpret_state_output("State: STARTING\n"), + Ok(true) + ); + assert_eq!( + interpret_state_output("State: STOPPING\n"), + Ok(true) + ); + assert_eq!( + interpret_state_output("State: ABORTING\n"), + Ok(true) + ); + } + + #[test] + fn a_state_name_we_do_not_know_is_unknown_rather_than_stopped() { + // A state added by a future liblxc must not default into the answer + // that unfilters a container. + assert!(interpret_state_output("State: MARVELLOUS\n").is_err()); + } + + #[test] + fn output_that_names_no_state_is_unknown_rather_than_stopped() { + // The whole point of the three-way answer. Callers read "stopped" as + // safe to unfilter and safe to skip stopping, so inferring it from + // output we could not read is what turns a broken probe into an + // unfiltered running container. + assert!(interpret_state_output("").is_err()); + assert!(interpret_state_output("Name: box\n").is_err()); + } + + #[test] + fn an_unlabelled_running_is_still_read_as_running() { + // The negative control for the test above: unknown must not swallow a + // state we can plainly see. If the label ever changes, the failure has + // to land on the safe side -- a refused operation, never an unfiltered + // container. + assert_eq!(interpret_state_output("RUNNING\n"), Ok(true)); + } + + #[test] + fn every_interface_is_counted_so_a_caller_can_refuse_to_half_filter() { + // liblxc prints the first value inline and the rest bare. The count is + // what decides whether one FORWARD hook covers the container, so a + // second interface has to survive the parse -- including one that only + // an lxc.include declared, which is why the question goes to liblxc. + assert_eq!( + interpret_config_values("lxc.net", "lxc.net = veth\nveth\n\n").len(), + 2 + ); + } + + #[test] + fn a_container_with_no_network_reports_no_interfaces() { + assert_eq!(interpret_config_values("lxc.net", "lxc.net =\n").len(), 0); + } + + #[test] + fn a_netdev_of_type_empty_still_counts_as_an_interface() { + // A config that declares lxc.net.0 properties without a type does not + // leave the type undeclared -- liblxc supplies `empty`, a real netdev + // type that gives the container only a loopback. The absence signal is + // an empty value, so reading the word as absence would report no + // interface for a container that has one and refuse it for the wrong + // reason. + assert_eq!( + interpret_config_values("lxc.net", "lxc.net = empty\n").len(), + 1 + ); + } + + #[test] + fn trailing_blank_lines_are_not_counted_as_interfaces() { + // Counting them would report a second interface that does not exist and + // refuse a container MXC can fully filter. + assert_eq!( + interpret_config_values("lxc.net", "lxc.net = veth\n\n\n\n").len(), + 1 + ); + } + + #[test] + fn a_single_value_is_returned_intact() { + assert_eq!( + interpret_config_values("lxc.net", "lxc.net = veth\n"), + vec!["veth".to_string()] + ); + } + + #[test] + fn multiple_values_span_the_inline_and_continuation_lines() { + // liblxc prints the first value on the key line and the rest bare, so + // every value has to be recovered regardless of which line carries it. + assert_eq!( + interpret_config_values( + "lxc.hook.start-host", + "lxc.hook.start-host = /a/one.sh\n/a/two.sh\n" + ), + vec!["/a/one.sh".to_string(), "/a/two.sh".to_string()] + ); + } + + #[test] + fn an_empty_value_yields_no_values() { + assert!(interpret_config_values("lxc.net", "lxc.net =\n").is_empty()); + } + + #[test] + fn a_value_containing_an_equals_sign_is_returned_intact() { + // A hook command can carry `=` in an argument. Splitting on the first + // `=` anywhere in the line would truncate it; keying the strip on the + // config key preserves the whole value -- both on the inline line and + // on a bare continuation line. + assert_eq!( + interpret_config_values( + "lxc.hook.start-host", + "lxc.hook.start-host = /a/pin.sh --name=veth0\n/a/other.sh k=v\n" + ), + vec![ + "/a/pin.sh --name=veth0".to_string(), + "/a/other.sh k=v".to_string() + ] + ); + } + + #[test] + fn leading_and_trailing_blank_lines_are_ignored() { + assert_eq!( + interpret_config_values("lxc.net", "\n\nlxc.net = veth\n\n"), + vec!["veth".to_string()] + ); + } + + #[test] + fn the_pin_hook_command_passes_the_target_as_a_separate_argument() { + // The script reads the target from $1, so the value has to be the + // script path and the target separated by whitespace, not concatenated. + let cmd = veth_pin_hook_command("/var/lib/lxc/box/mxc-veth-pin.sh", "mxcveth-box"); + assert!(cmd.contains("/var/lib/lxc/box/mxc-veth-pin.sh"), "{cmd}"); + assert!(cmd.contains("mxcveth-box"), "{cmd}"); + let mut parts = cmd.split_whitespace(); + assert_eq!(parts.next(), Some("/var/lib/lxc/box/mxc-veth-pin.sh")); + assert_eq!(parts.next(), Some("mxcveth-box")); + } + + #[test] + fn an_unreadable_container_is_an_error_rather_than_an_empty_answer() { + // No evidence of an interface is not evidence of no interface. Answering + // "none" would send the caller down the zero-interface path and report a + // refusal reason that was never established. + let c = LxcContainer::new("definitely-not-provisioned", Some("/nonexistent-lxcpath")); + assert!(c.configured_net_interfaces().is_err()); + } + #[test] fn lxc_container_honors_explicit_lxc_path() { let c = LxcContainer::new("my-box", Some("/opt/lxc")); @@ -635,14 +1849,71 @@ mod tests { ); } - // ---- build_attach_args ---------------------------------------------- + #[test] + fn clear_config_item_removes_only_matching_key_lines() { + // Set up a real config file with two `lxc.mount.entry` lines (the + // list-type key that accumulates across restarts), a similarly-named + // key that must be preserved, and unrelated keys. + let base = std::env::temp_dir().join(format!( + "mxc-clear-cfg-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0) + )); + let name = "box"; + std::fs::create_dir_all(base.join(name)).unwrap(); + let container = LxcContainer::new(name, Some(base.to_str().unwrap())); + + let original = "lxc.arch = amd64\n\ + lxc.mount.entry = /host/a a none bind,create=dir 0 0\n\ + lxc.mount = /some/fstab\n\ + lxc.mount.entry = /host/b b none bind,ro,create=dir 0 0\n\ + lxc.uts.name = box\n"; + std::fs::write(container.config_file_path(), original).unwrap(); + + container.clear_config_item("lxc.mount.entry").unwrap(); + + let after = std::fs::read_to_string(container.config_file_path()).unwrap(); + assert!( + !after.contains("lxc.mount.entry"), + "all lxc.mount.entry lines must be removed, got:\n{after}" + ); + // The prefix-sharing `lxc.mount` key and unrelated keys survive. + assert!(after.contains("lxc.mount = /some/fstab")); + assert!(after.contains("lxc.arch = amd64")); + assert!(after.contains("lxc.uts.name = box")); + + // Re-clearing is idempotent. + container.clear_config_item("lxc.mount.entry").unwrap(); + let after2 = std::fs::read_to_string(container.config_file_path()).unwrap(); + assert_eq!(after, after2); + + let _ = std::fs::remove_dir_all(&base); + } + + #[test] + fn clear_config_item_missing_file_is_ok() { + // A container whose config file does not exist is already "clear". + let bogus_base = std::env::temp_dir().join(format!( + "mxc-clear-missing-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0) + )); + let container = LxcContainer::new("ghost", Some(bogus_base.to_str().unwrap())); + assert!(container.clear_config_item("lxc.mount.entry").is_ok()); + } #[test] fn build_attach_args_no_env_no_cwd_is_unchanged_legacy_shape() { // Empty env + empty cwd must reproduce the original argv shape: // `-- /bin/sh -c ` so we don't perturb existing call sites // when neither cwd nor env is set. - let args = build_attach_args(&[], "", "echo hi"); + let args = build_attach_args(&[], "", "echo hi", None); assert_eq!(args, vec!["--", "/bin/sh", "-c", "echo hi"]); } @@ -653,7 +1924,7 @@ mod tests { "EMPTY=".to_string(), "HAS_EQ_IN_VAL=a=b=c".to_string(), ]; - let args = build_attach_args(&env, "", "cmd"); + let args = build_attach_args(&env, "", "cmd", None); assert_eq!( args, vec![ @@ -673,7 +1944,7 @@ mod tests { fn build_attach_args_env_entries_without_equals_are_skipped() { // Malformed entry can't poison the whole attach call. let env = vec!["BADENTRY".to_string(), "OK=val".to_string()]; - let args = build_attach_args(&env, "", "cmd"); + let args = build_attach_args(&env, "", "cmd", None); assert_eq!( args, vec![ @@ -698,7 +1969,7 @@ mod tests { "=val=more".to_string(), "OK=val".to_string(), ]; - let args = build_attach_args(&env, "", "cmd"); + let args = build_attach_args(&env, "", "cmd", None); assert_eq!( args, vec![ @@ -714,7 +1985,7 @@ mod tests { #[test] fn build_attach_args_cwd_wraps_command_with_cd_prelude() { - let args = build_attach_args(&[], "/opt/work", "echo hi"); + let args = build_attach_args(&[], "/opt/work", "echo hi", None); assert_eq!( args, vec![ @@ -736,7 +2007,7 @@ mod tests { // pass through sh as `$1` verbatim — no escaping needed here. let cwd = "/tmp/has spaces & 'quotes' $vars `cmd`"; let cmd = "printf '%s' \"$PWD\""; - let args = build_attach_args(&[], cwd, cmd); + let args = build_attach_args(&[], cwd, cmd, None); // cwd and command must appear verbatim as the last two argv entries. assert_eq!(args[args.len() - 2], cwd); @@ -750,7 +2021,7 @@ mod tests { #[test] fn build_attach_args_combines_env_and_cwd() { let env = vec!["FOO=bar".to_string()]; - let args = build_attach_args(&env, "/work", "cmd"); + let args = build_attach_args(&env, "/work", "cmd", None); assert_eq!( args, vec![ @@ -774,7 +2045,7 @@ mod tests { // also has to land BEFORE the `--set-var` entries so lxc-attach // clears first, then applies user vars on top. let env = vec!["FOO=bar".to_string()]; - let args = build_attach_args(&env, "", "cmd"); + let args = build_attach_args(&env, "", "cmd", None); let clear_idx = args .iter() .position(|a| a == "--clear-env") @@ -795,7 +2066,7 @@ mod tests { // Backward-compat guarantee: empty env preserves the legacy // keep-env shape so existing call sites with no explicit env are // undisturbed. - let args = build_attach_args(&[], "", "echo hi"); + let args = build_attach_args(&[], "", "echo hi", None); assert!( !args.iter().any(|a| a == "--clear-env"), "--clear-env must not appear when env is empty, got {:?}", @@ -805,7 +2076,7 @@ mod tests { #[test] fn build_attach_args_can_force_clear_env_when_env_empty() { - let args = build_attach_args_with_env_control(&[], "", "cmd", true); + let args = build_attach_args_with_env_control(&[], "", "cmd", true, None); assert_eq!(args, vec!["--clear-env", "--", "/bin/sh", "-c", "cmd"]); } @@ -816,7 +2087,7 @@ mod tests { // host env doesn't leak in through a back door. lxc-attach's own // baseline (HOME, PATH, USER, ...) keeps the child runnable. let env = vec!["BADENTRY".to_string(), "=alsobad".to_string()]; - let args = build_attach_args(&env, "", "cmd"); + let args = build_attach_args(&env, "", "cmd", None); assert_eq!(args, vec!["--clear-env", "--", "/bin/sh", "-c", "cmd"]); } @@ -831,7 +2102,7 @@ mod tests { // `MXC_TEST_FOO=HOST_LEAK_SHOULD_NOT_APPEAR` and asserts the // child sees the config's `MXC_TEST_FOO=bar baz`. let env = vec!["MXC_TEST_FOO=bar baz".to_string()]; - let args = build_attach_args(&env, "", "cmd"); + let args = build_attach_args(&env, "", "cmd", None); let clear_idx = args.iter().position(|a| a == "--clear-env").unwrap(); let set_idx = args .iter() @@ -844,6 +2115,18 @@ mod tests { ); } + #[test] + fn a_timed_exec_is_stamped_so_its_descendants_can_be_found() { + // The marker is the only handle a timeout has on work that outlived + // the attach. Without it the reap has nothing to match. + let args = build_attach_args(&[], "", "sleep 99", Some("tok123")); + assert!( + args.iter().any(|a| a == "--set-var=MXC_EXEC_ID=tok123"), + "timed exec must carry the marker, got {:?}", + args + ); + } + // ── End-to-end: proxy policy → env → attach args ───────────────────────── // These tests drive apply_proxy_env then build_attach_args_with_env_control // together so the observable output (the lxc-attach argv) is what is @@ -860,7 +2143,7 @@ mod tests { "PATH=/usr/bin".to_string(), ]; apply_proxy_env(&mut env, &ProxyConfig::default()); - let args = build_attach_args_with_env_control(&env, "", "cmd", true); + let args = build_attach_args_with_env_control(&env, "", "cmd", true, None); assert!( args.iter().any(|a| a == "--clear-env"), "the host environment must still be cleared; got {args:?}" @@ -876,6 +2159,181 @@ mod tests { ); } + #[test] + fn stamping_a_marker_does_not_clear_an_untouched_environment() { + // `--clear-env` is the caller's choice, keyed on the caller's env. + // Reaping must not smuggle in a wipe of the container's own + // environment as a side effect. + let args = build_attach_args(&[], "", "sleep 99", Some("tok123")); + assert!( + !args.iter().any(|a| a == "--clear-env"), + "marker must not pull in --clear-env, got {:?}", + args + ); + } + + #[test] + fn the_marker_is_applied_after_the_environment_is_cleared() { + // Same ordering rule as the caller's own vars: a marker set before + // `--clear-env` would be wiped, leaving a timed exec unreapable. + let env = vec!["FOO=bar".to_string()]; + let args = build_attach_args(&env, "", "cmd", Some("tok123")); + let clear_idx = args.iter().position(|a| a == "--clear-env").unwrap(); + let marker_idx = args + .iter() + .position(|a| a == "--set-var=MXC_EXEC_ID=tok123") + .expect("marker should be present"); + assert!( + clear_idx < marker_idx, + "--clear-env must precede the marker, got {:?}", + args + ); + } + + #[test] + fn a_caller_cannot_supply_its_own_marker() { + // A caller that sets the reserved name could otherwise be reaped by + // whichever exec owns that token, so the entry is dropped whether or + // not this exec carries a marker of its own. + let env = vec!["MXC_EXEC_ID=stolen".to_string(), "KEEP=yes".to_string()]; + + let timed = build_attach_args(&env, "", "cmd", Some("mine")); + assert!( + timed.iter().any(|a| a == "--set-var=KEEP=yes"), + "unrelated caller vars must survive, got {:?}", + timed + ); + assert!( + !timed.iter().any(|a| a == "--set-var=MXC_EXEC_ID=stolen"), + "the caller's marker must not reach the guest, got {:?}", + timed + ); + assert_eq!( + timed + .iter() + .filter(|a| a.starts_with("--set-var=MXC_EXEC_ID=")) + .count(), + 1, + "exactly one marker must survive, got {:?}", + timed + ); + + let untimed = build_attach_args(&env, "", "cmd", None); + assert!( + !untimed + .iter() + .any(|a| a.starts_with("--set-var=MXC_EXEC_ID=")), + "an untimed exec must carry no marker at all, got {:?}", + untimed + ); + } + + #[test] + fn an_untimed_exec_carries_no_marker() { + // Nothing can time out, so nothing needs reaping, and the argv stays + // exactly what it was before reaping existed. + let args = build_attach_args(&[], "", "echo hi", None); + assert!( + !args.iter().any(|a| a.contains("MXC_EXEC_ID")), + "untimed exec must not be stamped, got {:?}", + args + ); + } + + #[test] + fn the_reaper_matches_the_marker_of_exactly_one_exec() { + // Two concurrent execs in one container must not reap each other, so + // the argv has to carry the specific token and not just the var name. + let mine = build_reap_args("tok123"); + let theirs = build_reap_args("tok456"); + assert_eq!(mine.last().unwrap(), "MXC_EXEC_ID=tok123"); + assert_eq!(theirs.last().unwrap(), "MXC_EXEC_ID=tok456"); + assert_ne!(mine, theirs); + } + + #[test] + fn the_reaper_never_splices_the_marker_into_its_script() { + // The token reaches the shell as `$1`. Spliced in, a token bearing + // shell syntax would run as code inside the container. + let args = build_reap_args("t'; rm -rf /; #"); + let script = args + .iter() + .find(|a| a.contains("/proc/")) + .expect("reap script should be present"); + assert!( + !script.contains("rm -rf"), + "marker must not appear in the script body, got {:?}", + script + ); + assert!( + script.contains("\"$1\""), + "script must read the marker positionally, got {:?}", + script + ); + } + + #[test] + fn the_reaper_reports_success_when_nothing_matched() { + // `kill` finding no targets is the normal case for a script that had + // already finished; a nonzero exit there would be read as a failed + // reap and reported to the caller as possibly-still-running work. + let args = build_reap_args("tok123"); + let script = args.iter().find(|a| a.contains("/proc/")).unwrap(); + assert!( + script.trim_end().ends_with("exit 0"), + "reap script must end with an unconditional success, got {:?}", + script + ); + } + + #[test] + fn the_reaper_stops_a_process_before_it_kills_anything() { + // The `/proc` glob expands once per pass, so a workload that forks + // after the expansion has a child that pass never sees. Stopping first + // means a process that has been found cannot fork again, so the set + // that can still grow only shrinks. + let script = build_reap_args("tok123") + .into_iter() + .find(|a| a.contains("/proc/")) + .expect("reap script should be present"); + let stop = script.find("kill -STOP").expect("must stop what it finds"); + let kill = script.find("kill -KILL").expect("must then kill it"); + assert!( + stop < kill, + "the stop pass has to precede the kill pass, got {script:?}" + ); + } + + #[test] + fn the_reaper_rescans_until_it_finds_nothing_new() { + // One pass cannot see a process forked after its glob expanded, so a + // single pass leaves that child alive to outlive the exec. + let script = build_reap_args("tok123") + .into_iter() + .find(|a| a.contains("/proc/")) + .expect("reap script should be present"); + assert!( + script.contains("while ["), + "the scan must repeat, got {script:?}" + ); + assert!( + script.contains("break"), + "the scan must stop once a pass finds nothing new, got {script:?}" + ); + } + + #[test] + fn the_reaper_names_its_signals_rather_than_numbering_them() { + // SIGSTOP is 19 on x86 and ARM but 17 on Alpha and SPARC and 23 on + // MIPS. A number here would stop the wrong thing on those hosts. + let script = build_reap_args("tok123") + .into_iter() + .find(|a| a.contains("/proc/")) + .expect("reap script should be present"); + assert!(!script.contains("kill -9"), "got {script:?}"); + assert!(!script.contains("kill -19"), "got {script:?}"); + } + #[test] fn proxy_enabled_emits_clear_env_and_proxy_keys_in_attach_args() { use wxc_common::{ @@ -888,7 +2346,7 @@ mod tests { }; let mut env = vec!["PATH=/usr/bin".to_string()]; apply_proxy_env(&mut env, &proxy); - let args = build_attach_args_with_env_control(&env, "", "cmd", true); + let args = build_attach_args_with_env_control(&env, "", "cmd", true, None); assert!( args.iter().any(|a| a == "--clear-env"), "proxy enabled must emit --clear-env; got {args:?}" diff --git a/src/backends/lxc/common/src/lxc_runner.rs b/src/backends/lxc/common/src/lxc_runner.rs index e292c5652..29d507714 100644 --- a/src/backends/lxc/common/src/lxc_runner.rs +++ b/src/backends/lxc/common/src/lxc_runner.rs @@ -10,14 +10,12 @@ use std::thread; use std::time::{Duration, Instant}; use wxc_common::logger::Logger; -use wxc_common::models::{ - ExecutionRequest, LifecycleConfig, LxcConfig, NetworkEnforcementMode, ScriptResponse, -}; +use wxc_common::models::{ExecutionRequest, LifecycleConfig, LxcConfig, ScriptResponse}; use wxc_common::script_runner::ScriptRunner; use wxc_common::validator::{validate_network_policy_support, NetworkPolicySupport}; use crate::filesystem_mounts; -use crate::lxc_bindings::LxcContainer; +use crate::lxc_bindings::{LxcContainer, NetInterfaceConfig}; use crate::network_ingress::IngressManager; use crate::network_iptables::NetworkIptablesManager; use crate::signal_cleanup; @@ -58,6 +56,36 @@ impl LxcScriptRunner { } } + /// Whether the pin hook can rename this container's interface before start. + /// + /// A nonzero exit from the hook aborts the start, so a shape it would + /// reject keeps the post-start path rather than becoming a new refusal. + fn pinnable_before_start(net: &NetInterfaceConfig) -> bool { + net.count == 1 && net.sole_kind.as_deref() == Some("veth") + } + + /// Halt a container after a failure that happened once it was already + /// running, retaining its filtering if it could not be halted. + /// + /// A run that will not destroy the container -- `destroy_on_exit` off on + /// one it did not create -- otherwise returns while the container keeps + /// running, and the managers then drop their rules on the way out. The + /// result is a live container with its egress filtering removed, reported + /// to the caller as a failure. Preserving the policy when the stop fails + /// keeps the rules in place for whatever is still transmitting. + fn halt_after_failed_start( + &self, + container: &LxcContainer, + container_created: bool, + fw_manager: &mut NetworkIptablesManager, + ) { + if self.destroy_on_exit || container_created { + let _ = container.destroy(); + } else if container.stop().is_err() { + fw_manager.set_preserve_policy(true); + } + } + /// Wait for the container's network stack to initialize. /// Polls `lxc-info` until the container has an IP address or the timeout is reached. fn wait_for_network(container_name: &str, timeout: Duration, logger: &mut Logger) -> bool { @@ -100,11 +128,7 @@ impl LxcScriptRunner { /// Core execution logic. fn run_internal(&self, request: &ExecutionRequest, logger: &mut Logger) -> ScriptResponse { - // Object-based FS-policy normalization (D6): tighten aliases of the same - // host object to the strictest intent (deny > ro > rw) before building - // mounts. See `wxc_common::filesystem_object`. Only clone the request - // when an aliasing conflict actually needs tightening; an unresolvable - // path with deniedPaths present fails closed. + // tighten aliases of the same host object to the strictest intent (deny > ro > rw) let normalized; let request = match wxc_common::filesystem_object::normalize_object_conflicts( &request.policy, @@ -120,10 +144,7 @@ impl LxcScriptRunner { Ok(None) => request, Err(msg) => return ScriptResponse::error(&msg), }; - // Delegation check (D3): reject any policy path the invoking user cannot - // access, so the sandbox never gains access the caller lacks. Runs AFTER - // object normalization so it is evaluated against the already-tightened - // intents. + // reject any policy path the invoking user cannot access if let Err(msg) = wxc_common::filesystem_access::check_delegation(&request.policy) { return ScriptResponse::error(&msg); } @@ -137,18 +158,7 @@ impl LxcScriptRunner { } let container_name = self.resolve_container_name(); - // Refuse a credential-bearing proxy URL here as well as at parse time. - // The parser guard only covers requests it built; `ExecutionRequest` - // and `ProxyAddress::from_url` are public, so a caller can hand this - // runner a policy the parser never saw. Below, `apply_proxy_env` sets - // HTTP(S)_PROXY to `to_url()`, which returns the original URL verbatim, - // and `build_attach_args_with_env_control` turns every environment - // entry into a `--set-var=KEY=VALUE` argument of the `lxc-attach` - // process this backend spawns (lxc_bindings.rs). A process's argv is - // readable through /proc//cmdline by any local user for the - // lifetime of the command. The check sits ahead of container creation - // and firewall programming so a rejected request leaves no state - // behind. + // Credentials here would be visible to other local users. if let Some(url) = request .policy .network_proxy @@ -157,24 +167,14 @@ impl LxcScriptRunner { .map(|address| address.to_url()) { if wxc_common::proxy_env::proxy_url_has_credentials(&url) { - // Built from the redacted form so the rejection cannot become - // the leak it is rejecting. return ScriptResponse::error(&format!( - "LXC: network.proxy.url must not carry credentials ('{}'). LXC passes the \ - proxy URL to lxc-attach as a --set-var command-line argument, and process \ - arguments are world-readable through /proc//cmdline, so the password \ - would be visible to every local user while the command runs. Use a proxy \ + "LXC: network.proxy.url must not carry credentials ('{}'). Use a proxy \ that does not require inline credentials, or supply them to the proxy \ itself rather than through the URL.", wxc_common::proxy_env::redact_proxy_url(&url) )); } } - // Make the name visible to the signal-cleanup watchdog so a fatal - // signal during create/start/attach still tears the container down — - // but only when the caller actually wants the container destroyed at - // exit. With `destroyOnExit = false` the normal completion path - // preserves the container, so the signal path must too. if self.destroy_on_exit { signal_cleanup::set_active(&container_name); } @@ -200,8 +200,19 @@ impl LxcScriptRunner { let container = LxcContainer::new(&container_name, None); let mut container_created = false; - // Create the container if it doesn't exist - if !container.is_defined() { + // Create the container if it doesn't exist. A probe that could not run + // is not evidence of absence, so it aborts rather than creating a + // second container over a first one we simply failed to see. + let defined = match container.is_defined() { + Ok(defined) => defined, + Err(e) => { + return ScriptResponse::error(&format!( + "Failed to determine whether the container exists: {}", + e + )); + } + }; + if !defined { let _ = writeln!(logger, "Creating LXC container..."); if let Err(e) = container.create(&self.config.distribution, &self.config.release) { return ScriptResponse::error(&format!("Failed to create container: {}", e)); @@ -222,8 +233,104 @@ impl LxcScriptRunner { return ScriptResponse::error(&format!("Failed to configure filesystem: {}", e)); } - // Ensure the container is running so that the veth interface exists - if !container.is_running() { + // Ensure the container is running so that the veth interface exists. An + // unreadable probe aborts: starting a container that is already running + // is not harmless here, and neither is proceeding as though it were up. + let running = match container.is_running() { + Ok(running) => running, + Err(e) => { + if self.destroy_on_exit || container_created { + let _ = container.destroy(); + } + return ScriptResponse::error(&format!( + "Failed to determine whether the container is running: {}", + e + )); + } + }; + let mut fw_manager = NetworkIptablesManager::new(&container_name); + + // Read once, so the refusal below and the pin decision cannot disagree + // about the same container. + let net_config = if request.policy.requires_firewall() { + match container.configured_net_interfaces() { + Ok(net) => Some(net), + Err(e) => { + let _ = writeln!(logger, "Could not read the network config: {}", e); + None + } + } + } else { + None + }; + + // The FORWARD hook matches a single veth, so traffic on any other + // interface never reaches the chain and the policy goes unenforced. + // A container with no usable veth already fails when the rules are + // applied, because nothing calls `allow_missing_veth_interface` on this + // path; one with several interfaces instead resolves the first and + // silently bypasses the rest, so it is refused here. A read that failed + // keeps the pre-existing behavior rather than turning an unreadable + // config into a new way to fail. + if let Some(net) = net_config.as_ref() { + if net.count > 1 { + if self.destroy_on_exit || container_created { + let _ = container.destroy(); + } + return ScriptResponse::error(&format!( + "Container {:?} has {} configured network interfaces; a firewall-enforced \ + network policy can only be applied to a container with a single interface, \ + because traffic on the others would bypass it", + container_name, net.count, + )); + } + } + + // Install egress before the container starts, so nothing inside it + // transmits during an interval MXC already reports as deny-all. + // iptables accepts a rule naming an interface that does not exist yet, + // so the name is pinned here and the interface catches up at start. + let mut egress_installed = false; + if !running && request.policy.requires_firewall() { + let pinnable = net_config + .as_ref() + .map(Self::pinnable_before_start) + .unwrap_or(false); + if pinnable { + let veth = NetworkIptablesManager::deterministic_veth_name(&container_name); + if let Err(e) = container.ensure_veth_pin_hook(&veth) { + if self.destroy_on_exit || container_created { + let _ = container.destroy(); + } + return ScriptResponse::error(&format!( + "Failed to pin the container's network interface: {}", + e + )); + } + let _ = writeln!(logger, "Pinned veth interface before start: {}", veth); + fw_manager.set_veth_interface(&veth); + if self.destroy_on_exit { + signal_cleanup::set_active_veth(&veth); + } + match fw_manager.apply_firewall_rules(&request.policy, logger) { + Ok(true) => egress_installed = true, + Ok(false) => { + if self.destroy_on_exit || container_created { + let _ = container.destroy(); + } + return ScriptResponse::error("Failed to apply network firewall rules."); + } + Err(e) => { + if self.destroy_on_exit || container_created { + let _ = container.destroy(); + } + return ScriptResponse::error(&format!("Network policy error: {}", e)); + } + } + } + } + + if !running { let _ = writeln!(logger, "Starting LXC container..."); if let Err(e) = container.start() { if self.destroy_on_exit || container_created { @@ -236,92 +343,94 @@ impl LxcScriptRunner { let _ = writeln!(logger, "Container already running."); } - // Wait for network only when the config uses network features (firewall rules - // or allowed/blocked hosts), or when the container must reach a proxy. - let needs_network = matches!( - request.policy.network_enforcement_mode, - NetworkEnforcementMode::Firewall | NetworkEnforcementMode::Both - ) || !request.policy.allowed_hosts.is_empty() - || !request.policy.blocked_hosts.is_empty() - || request.policy.network_proxy.is_enabled(); - - if needs_network { + if request.policy.requires_firewall() { Self::wait_for_network(&container_name, Duration::from_secs(10), logger); } - // Configure network rules - let mut fw_manager = NetworkIptablesManager::new(&container_name); - fw_manager.set_preserve_policy(!self.cleanup_policy); - - // Try to discover the container's veth interface for scoped rules - if let Some(veth) = NetworkIptablesManager::discover_veth_interface(&container_name) { - let _ = writeln!(logger, "Discovered veth interface: {}", veth); - fw_manager.set_veth_interface(&veth); - if self.destroy_on_exit { - // Tell the watchdog about the veth so signal-time cleanup - // can also remove the FORWARD hook, not just the chain. - signal_cleanup::set_active_veth(&veth); + // Shapes the pre-start path could not pin: an already running + // container, several interfaces, or one the hook cannot rename. + if !egress_installed { + // Resolve the container's veth interface for scoped rules. The pin hook + // persists in a container's config, so a container this runner did not + // pin may still have been renamed by an earlier state-aware start; the + // container itself is the only thing that knows which. + let pinned = NetworkIptablesManager::deterministic_veth_name(&container_name); + let pin_hook_present = match container.has_veth_pin_hook(&pinned) { + Ok(present) => present, + Err(e) => { + // Guessing here picks between two names, one of which filters + // nothing. Report it and resolve nothing rather than scope the + // rules to a name that may already be stale. + let _ = writeln!(logger, "Could not read the veth pin hook: {}", e); + self.halt_after_failed_start(&container, container_created, &mut fw_manager); + return ScriptResponse::error( + "Failed to resolve the container's network interface.", + ); + } + }; + if let Some(veth) = + NetworkIptablesManager::live_veth_interface(&container_name, pin_hook_present) + { + let _ = writeln!(logger, "Resolved veth interface: {}", veth); + fw_manager.set_veth_interface(&veth); + if self.destroy_on_exit { + // Tell the watchdog about the veth so signal-time cleanup + // can also remove the FORWARD hook, not just the chain. + signal_cleanup::set_active_veth(&veth); + } } - } - match fw_manager.apply_firewall_rules(&request.policy, logger) { - Ok(true) => {} - Ok(false) => { - if self.destroy_on_exit || container_created { - let _ = container.destroy(); + match fw_manager.apply_firewall_rules(&request.policy, logger) { + Ok(true) => {} + Ok(false) => { + self.halt_after_failed_start(&container, container_created, &mut fw_manager); + return ScriptResponse::error("Failed to apply network firewall rules."); } - return ScriptResponse::error("Failed to apply network firewall rules."); - } - Err(e) => { - if self.destroy_on_exit || container_created { - let _ = container.destroy(); + Err(e) => { + self.halt_after_failed_start(&container, container_created, &mut fw_manager); + return ScriptResponse::error(&format!("Network policy error: {}", e)); } - return ScriptResponse::error(&format!("Network policy error: {}", e)); } } + // Until the container is up, a failure above must still tear down what + // the pre-start install put in place. + fw_manager.set_preserve_policy(!self.cleanup_policy); + // Configure inbound (ingress) network rules inside the container's own // netns. This is a separate, orthogonal chain from the egress rules // above: it enforces `allowLocalNetwork` (inbound default-deny) via the // container's own iptables INPUT chain, reached with `nsenter`. // - // A firewall enforcement mode means the caller asked for the inbound - // deny chain. LXC enforces it inside the container's own netns, so it - // is useless without the init PID that lets us enter that netns — and - // the ingress manager cannot even be constructed without one. - let use_firewall = matches!( - request.policy.network_enforcement_mode, - NetworkEnforcementMode::Firewall | NetworkEnforcementMode::Both - ); - - // Kept in scope for post-execution cleanup; `None` when there is no - // netns PID and no firewall was requested (nothing to enforce). - let mut ingress_manager: Option = None; - - match container.init_pid() { + // Ingress installs unconditionally, so the init PID is mandatory for + // every start: LXC enters the container's netns through it, and the + // ingress manager cannot even be constructed without one. + + // Kept in scope for post-execution cleanup. A missing init PID returns + // below, so the match always yields the manager for a successful + // install on the path that continues. + let mut ingress_manager: Option = match container.init_pid() { Some(pid) => { let _ = writeln!(logger, "Container init PID: {}", pid); - if self.destroy_on_exit { - // Tell the watchdog about the netns PID so signal-time - // cleanup can remove the container's INPUT rules before it's - // destroyed. - signal_cleanup::set_active_pid(pid); - } let mut mgr = IngressManager::new(&container_name, pid); match mgr.apply_firewall_rules(&request.policy, logger) { Ok(true) => {} Ok(false) => { - if self.destroy_on_exit || container_created { - let _ = container.destroy(); - } + self.halt_after_failed_start( + &container, + container_created, + &mut fw_manager, + ); return ScriptResponse::error( "Failed to apply inbound network firewall rules.", ); } Err(e) => { - if self.destroy_on_exit || container_created { - let _ = container.destroy(); - } + self.halt_after_failed_start( + &container, + container_created, + &mut fw_manager, + ); return ScriptResponse::error(&format!( "Inbound network policy error: {}", e @@ -329,39 +438,30 @@ impl LxcScriptRunner { } } // Every non-success arm above returns, so the apply call - // succeeded. Only now may the policy be marked for - // preservation: the flag also suppresses `Drop`, and a partial - // chain from a failed install must still be torn down. Success - // does not imply a chain exists — a non-firewall enforcement - // mode succeeds without installing one — but in that case no - // ownership flag is set and `Drop` has nothing to do either way. + // succeeded and the inbound chain now exists. Only now may the + // policy be marked for preservation: the flag also suppresses + // `Drop`, and a partial chain from a failed install must still be + // torn down. mgr.set_preserve_policy(!self.cleanup_policy); - ingress_manager = Some(mgr); + Some(mgr) } - None if use_firewall => { - // The run asked for a firewall but we could not find the - // container netns to enforce it in. There is no legitimate - // ingress-without-a-netns case: enforcing inbound requires - // entering the container's namespace, so running anyway would - // silently disable the requested inbound deny (a fail-open). - // Abort instead. This guard is specific to the LXC ingress - // path, which addresses its namespace by init PID; other - // backends reach their firewall handling through their own - // runners and never construct an `IngressManager`. - if self.destroy_on_exit || container_created { - let _ = container.destroy(); - } + None => { + // Ingress installs unconditionally now, so a missing netns is + // always fatal. Enforcing inbound requires entering the + // container's namespace, so running anyway would silently + // disable the inbound deny (a fail-open). Abort instead. This + // guard is specific to the LXC ingress path, which addresses its + // namespace by init PID; other backends reach their firewall + // handling through their own runners and never construct an + // `IngressManager`. + self.halt_after_failed_start(&container, container_created, &mut fw_manager); return ScriptResponse::error( "Failed to discover the container init PID; cannot enter the container \ - network namespace to enforce the requested inbound firewall. Aborting \ - rather than running with inbound enforcement silently disabled.", + network namespace to enforce the inbound firewall. Aborting rather than \ + running with inbound enforcement silently disabled.", ); } - None => { - // No firewall requested and no netns PID: nothing to enforce - // inbound, so no ingress chain is installed. - } - } + }; let mut pinned = false; @@ -376,7 +476,7 @@ impl LxcScriptRunner { pin.ip() ); let pin_outcome = - container.attach_run(&command, "/", &[], true, Some(HOSTS_COMMAND_TIMEOUT)); + container.attach_run(&command, "/", &[], true, Some(HOSTS_COMMAND_TIMEOUT), None); let pin_error = match pin_outcome { Ok((0, _, _)) => None, @@ -409,12 +509,18 @@ impl LxcScriptRunner { // in place it keeps resolving a hostname to an address that only // some earlier policy authorized. let unpin = Self::build_hosts_unpin_command(); - let stale_pin_error = - match container.attach_run(&unpin, "/", &[], true, Some(HOSTS_COMMAND_TIMEOUT)) { - Ok((0, _, _)) => None, - Ok((code, _, _)) => Some(Self::hosts_command_failure("clearing", code)), - Err(e) => Some(e.to_string()), - }; + let stale_pin_error = match container.attach_run( + &unpin, + "/", + &[], + true, + Some(HOSTS_COMMAND_TIMEOUT), + None, + ) { + Ok((0, _, _)) => None, + Ok((code, _, _)) => Some(Self::hosts_command_failure("clearing", code)), + Err(e) => Some(e.to_string()), + }; // Unlike the post-run removal, this one runs *before* the script, so // a failure still changes what the script would resolve. Refuse the @@ -442,6 +548,12 @@ impl LxcScriptRunner { Some(Duration::from_millis(u64::from(request.script_timeout))) }; let _ = writeln!(logger, "Executing script inside container..."); + // Stamped so a timeout can reap what the script started. The one-shot + // container is normally destroyed on exit, which reaps everything, but + // a run configured to leave it behind has the same persistence problem + // the state-aware path does. + let marker = crate::lxc_bindings::mint_exec_marker(); + let mut exec_env = request.env.clone(); // Scrub every inherited proxy variable and, when the policy carries a // proxy, point HTTP(S)_PROXY at it. @@ -456,6 +568,7 @@ impl LxcScriptRunner { &exec_env, true, timeout, + Some(&marker), ); let response = match result { @@ -473,12 +586,18 @@ impl LxcScriptRunner { // must not outlive that chain. if pinned && self.cleanup_policy { let unpin = Self::build_hosts_unpin_command(); - let unpin_error = - match container.attach_run(&unpin, "/", &[], true, Some(HOSTS_COMMAND_TIMEOUT)) { - Ok((0, _, _)) => None, - Ok((code, _, _)) => Some(Self::hosts_command_failure("clearing", code)), - Err(e) => Some(e.to_string()), - }; + let unpin_error = match container.attach_run( + &unpin, + "/", + &[], + true, + Some(HOSTS_COMMAND_TIMEOUT), + None, + ) { + Ok((0, _, _)) => None, + Ok((code, _, _)) => Some(Self::hosts_command_failure("clearing", code)), + Err(e) => Some(e.to_string()), + }; // The script has already run, so a failure here cannot change its // result and must not replace it. @@ -681,6 +800,42 @@ fn uuid_simple() -> String { mod tests { use super::*; + #[test] + fn a_single_veth_can_be_pinned_before_start() { + let net = NetInterfaceConfig { + count: 1, + sole_kind: Some("veth".to_string()), + }; + assert!(LxcScriptRunner::pinnable_before_start(&net)); + } + + #[test] + fn a_sole_interface_that_is_not_veth_is_left_to_the_post_start_path() { + let net = NetInterfaceConfig { + count: 1, + sole_kind: Some("macvlan".to_string()), + }; + assert!(!LxcScriptRunner::pinnable_before_start(&net)); + } + + #[test] + fn a_container_with_no_interfaces_is_left_to_the_post_start_path() { + let net = NetInterfaceConfig { + count: 0, + sole_kind: None, + }; + assert!(!LxcScriptRunner::pinnable_before_start(&net)); + } + + #[test] + fn a_container_with_several_interfaces_is_left_to_the_post_start_path() { + let net = NetInterfaceConfig { + count: 2, + sole_kind: None, + }; + assert!(!LxcScriptRunner::pinnable_before_start(&net)); + } + #[test] fn uuid_simple_is_8_chars() { let id = uuid_simple(); diff --git a/src/backends/lxc/common/src/network_ingress.rs b/src/backends/lxc/common/src/network_ingress.rs index eabc45715..986fbf351 100644 --- a/src/backends/lxc/common/src/network_ingress.rs +++ b/src/backends/lxc/common/src/network_ingress.rs @@ -57,7 +57,7 @@ use std::process::Command; use wxc_common::logger::Logger; -use wxc_common::models::{ContainerPolicy, NetworkEnforcementMode}; +use wxc_common::models::ContainerPolicy; use crate::network_iptables::{ ingress_chain_name_for, HostIpv6State, Ip6tablesStatus, NetworkIptablesManager, @@ -158,7 +158,7 @@ pub struct IngressManager { /// `/proc//net/if_inet6`, which names the same namespace by PID /// without entering it. A caller that cannot supply a PID must not /// construct an `IngressManager` at all. See `lxc_runner`, which aborts the - /// run when a firewall mode is requested but no init PID can be found. + /// run when no init PID can be found. netns_pid: u32, /// Per-resource ownership. What we actually created or hooked, tracked /// separately per family, so teardown attempts only the operations this run @@ -505,17 +505,8 @@ impl IngressManager { policy: &ContainerPolicy, logger: &mut Logger, ) -> Result { - // Skip if network enforcement doesn't use a firewall. - let use_firewall = matches!( - policy.network_enforcement_mode, - NetworkEnforcementMode::Firewall | NetworkEnforcementMode::Both - ); - if !use_firewall { - logger.log_line( - "Network enforcement mode does not use firewall, skipping ingress chain.", - ); - return Ok(true); - } + // Gating this on the egress policy would stop installing an inbound + // deny for a permissive-egress config that gets one today. // Permissive host-loopback inbound (`allowLocalNetwork: true`) is not // yet implementable safely. Scoping it to host loopback needs a schema @@ -675,9 +666,9 @@ impl IngressManager { /// Instead we *force* a known state: remove every INPUT reference (there /// may be more than one), then flush and delete any existing chain. This /// reuses the same over-approximating teardown planner and executor as - /// `force_cleanup` ("assume everything might exist"), so install-time reset - /// and ownership teardown share one shape. A genuinely absent rule or chain - /// is a no-op; any real failure aborts install fail-closed. + /// ownership teardown ("assume everything might exist"), so install-time + /// reset and ownership teardown share one shape. A genuinely absent rule or + /// chain is a no-op; any real failure aborts install fail-closed. fn reset_family( &mut self, family: IpFamily, @@ -1008,47 +999,6 @@ impl IngressManager { } } - /// Best-effort cleanup of iptables state when the owning [`IngressManager`] - /// instance isn't reachable (e.g. signal-time cleanup from the watchdog - /// thread). We do not know which resources a prior run installed, so this - /// assumes all of them and lets teardown over-approximate: a genuinely - /// absent chain or hook is treated as an already-removed no-op. The caller - /// supplies the container's init PID, which it only has while the netns - /// still exists — once the container is gone there is nothing to remove, so - /// the caller simply does not call this. The result is ignored: this path is - /// best-effort by nature. - pub fn force_cleanup(container_name: &str, netns_pid: u32, logger: &mut Logger) { - let mut runner = NsenterRunner; - Self::force_cleanup_with(container_name, netns_pid, &mut runner, logger); - } - - /// The body of [`Self::force_cleanup`] against an injectable - /// [`CommandRunner`] — the real path passes [`NsenterRunner`]; tests pass a - /// runner that captures the planned argv without spawning. Constructs the - /// over-approximating manager and runs teardown; the result is ignored - /// because this path is best-effort by nature. - fn force_cleanup_with( - container_name: &str, - netns_pid: u32, - runner: &mut dyn CommandRunner, - logger: &mut Logger, - ) { - let mut mgr = Self::for_full_reset(container_name, netns_pid); - let _ = mgr.remove_firewall_rules_with(runner, logger); - } - - /// A manager that assumes every resource might exist, for over-approximating - /// cleanup where we do not know what a dead run installed. Used by - /// [`Self::force_cleanup`]. - fn for_full_reset(container_name: &str, netns_pid: u32) -> Self { - let mut mgr = Self::new(container_name, netns_pid); - mgr.v4_chain_created = true; - mgr.v6_chain_created = true; - mgr.v4_hooked = true; - mgr.v6_hooked = true; - mgr - } - /// Build the full argv for running `binary args...` inside this container's /// network namespace: `["nsenter", "-t", , "-n", binary, args...]`. /// @@ -1879,12 +1829,12 @@ mod tests { ); } - /// A firewall-mode policy with `allowLocalNetwork: true`. - fn permissive_firewall_policy() -> ContainerPolicy { + /// A policy with `allowLocalNetwork: true`. Ingress ignores + /// `enforcementMode`, so the mode is left at its default. + fn permissive_policy() -> ContainerPolicy { ContainerPolicy { allow_local_network: true, default_network_policy: NetworkPolicy::Block, - network_enforcement_mode: NetworkEnforcementMode::Firewall, ..Default::default() } } @@ -1900,7 +1850,7 @@ mod tests { // The PID value is irrelevant: the refusal precedes every use of it. for pid in [1u32, 42u32, 999_999u32] { let mut mgr = IngressManager::new("permissive-container", pid); - let result = mgr.apply_firewall_rules(&permissive_firewall_policy(), &mut logger); + let result = mgr.apply_firewall_rules(&permissive_policy(), &mut logger); assert!( result.is_err(), "allowLocalNetwork: true must be refused (pid={pid})" @@ -1962,11 +1912,10 @@ mod tests { assert_eq!(tail, want, "the wrapped command args must be preserved"); } - /// The teardown path (used by `remove_firewall_rules`, `Drop`, and - /// `force_cleanup`) must also route every command through `nsenter`, and - /// only against the manager's own chain. Assert the actual planned argv - /// rather than just the pure helper: a command that skips `nsenter` would - /// execute against the host's tables. + /// The teardown path (used by `remove_firewall_rules` and `Drop`) must also + /// route every command through `nsenter`, and only against the manager's own + /// chain. Assert the actual planned argv rather than just the pure helper: a + /// command that skips `nsenter` would execute against the host's tables. #[test] fn teardown_commands_are_nsenter_prefixed_and_chain_scoped() { let pid = 4242u32; @@ -2059,82 +2008,6 @@ mod tests { mgr.v4_hooked = false; } - /// `force_cleanup` must actually run — over the injectable runner — so a - /// regression inside it fails this test, and it must plan to remove *all* - /// resources (it cannot know what a dead run installed). Assert the real - /// commands `force_cleanup_with` issues: every one nsenter-scoped to the - /// container netns, naming only our chain, unhook before flush before - /// delete, for both families. A no-leftover script (everything reports - /// already-absent) lets cleanup complete without error. - #[test] - fn force_cleanup_removes_all_resources_via_nsenter() { - let pid = 9001u32; - let container = "force-cleanup-container"; - let chain = IngressManager::new(container, pid).chain_name().to_string(); - let mut logger = Logger::new(wxc_common::logger::Mode::Buffer); - - let mut runner = FakeRunner { - calls: Vec::new(), - respond: |argv: &[String]| match verb(argv) { - "-D" => Err(absent_rule()), - _ => Err(absent_chain()), - }, - }; - - IngressManager::force_cleanup_with(container, pid, &mut runner, &mut logger); - - // Six commands: unhook + flush + delete, per family. - assert_eq!( - runner.calls.len(), - 6, - "force_cleanup must issue all six teardown commands, got {:?}", - runner.calls - ); - - // Every command is nsenter-scoped and names only our chain. - for argv in &runner.calls { - assert_eq!( - &argv[..4], - &[ - "nsenter".to_string(), - "-t".to_string(), - pid.to_string(), - "-n".to_string(), - ], - "force_cleanup command must be nsenter-scoped to the netns: {argv:?}" - ); - assert!( - argv[4] == "iptables" || argv[4] == "ip6tables", - "force_cleanup command must target a packet-filter binary: {argv:?}" - ); - assert!( - argv.iter().any(|a| a == &chain), - "force_cleanup command must name our chain '{chain}': {argv:?}" - ); - } - - // Both families are covered. - for binary in ["iptables", "ip6tables"] { - assert!( - runner.calls.iter().any(|a| a[4] == binary), - "force_cleanup must cover {binary}" - ); - } - - // Per family: unhook (-D) before flush (-F) before delete (-X). - for binary in ["iptables", "ip6tables"] { - let idx = |v: &str| { - runner - .calls - .iter() - .position(|a| a[4] == binary && verb(a) == v) - .unwrap_or_else(|| panic!("{binary} {v} command missing")) - }; - assert!(idx("-D") < idx("-F"), "{binary}: unhook must precede flush"); - assert!(idx("-F") < idx("-X"), "{binary}: flush must precede delete"); - } - } - /// The pure builder must return the chain *body* and the `INPUT` *hook* as /// separate values, so install cannot confuse one for the other and needs no /// `-I`-filtering. The body carries neither the `-N` creation nor the `-I` diff --git a/src/backends/lxc/common/src/network_ingress_permissive_spec_tests.rs b/src/backends/lxc/common/src/network_ingress_permissive_spec_tests.rs index ed6911a3f..df6d31e72 100644 --- a/src/backends/lxc/common/src/network_ingress_permissive_spec_tests.rs +++ b/src/backends/lxc/common/src/network_ingress_permissive_spec_tests.rs @@ -7,17 +7,14 @@ //! # Decision table //! //! The netns PID is mandatory: [`IngressManager`] cannot be constructed without -//! one, so there is no "no-PID" row. All cells assume -//! `NetworkEnforcementMode::Firewall` unless stated otherwise, because -//! `apply_firewall_rules` returns early with `Ok(true)` for `Capabilities` mode -//! before reaching the permissive guard. +//! one, so there is no "no-PID" row. Ingress installs unconditionally, so +//! `enforcementMode` no longer changes the outcome -- only `allowLocalNetwork` +//! does. //! -//! | allow_local_network | enforcement mode | Required outcome | Source | -//! |---------------------|------------------|------------------|--------| -//! | false | Firewall | NOT refused | guard is permissive-path only | -//! | true | Firewall | REFUSED (Err) | "apply_firewall_rules returns a clear not-yet-implemented error" | -//! | true | Both | REFUSED (Err) | Both ∈ firewall-using modes | -//! | true | Capabilities | NOT refused | early-return before guard; no firewall path | +//! | allow_local_network | Required outcome | Source | +//! |---------------------|------------------|--------| +//! | false | NOT refused (installs the inbound deny) | guard is permissive-path only | +//! | true | REFUSED (Err), whatever the mode | "apply_firewall_rules returns a clear not-yet-implemented error" | use super::*; use wxc_common::logger::Mode; @@ -64,14 +61,11 @@ fn make_logger() -> Logger { /// Build a policy that reaches the permissive guard. /// -/// The guard is only reachable when `network_enforcement_mode` is `Firewall` -/// or `Both`; `Capabilities` (the default) causes an early return before the -/// guard. Every fixture that must exercise the guard must set one of the -/// firewall-using modes explicitly. +/// The guard fires on `allow_local_network` alone; ingress installs +/// regardless of `enforcementMode`, so the mode is left at its default here. fn firewall_policy(allow_local: bool) -> ContainerPolicy { ContainerPolicy { allow_local_network: allow_local, - network_enforcement_mode: NetworkEnforcementMode::Firewall, ..Default::default() } } @@ -116,41 +110,51 @@ fn permissive_inbound_in_a_container_netns_is_refused_not_installed() { ); } -/// NetworkEnforcementMode::Both also uses the firewall path. The permissive -/// guard must refuse for Both too. +/// The refusal must not depend on `enforcementMode`: ingress installs +/// unconditionally, so `allowLocalNetwork` is refused whatever the mode. This +/// is the observable form of "the mode is ignored" -- an explicit +/// `capabilities`, which before this change returned early and skipped the +/// guard entirely, must now refuse exactly as `firewall` does. #[test] -fn permissive_inbound_in_both_mode_is_refused_not_installed() { - let policy = ContainerPolicy { - allow_local_network: true, - network_enforcement_mode: NetworkEnforcementMode::Both, - ..Default::default() - }; - let mut mgr = IngressManager::new("test-container-refused-both", UNOCCUPIABLE_NETNS_PID); - let mut logger = make_logger(); - - let result = mgr.apply_firewall_rules(&policy, &mut logger); +fn permissive_inbound_is_refused_whatever_the_enforcement_mode() { + for mode in [ + NetworkEnforcementMode::Capabilities, + NetworkEnforcementMode::Firewall, + NetworkEnforcementMode::Both, + ] { + let mode_label = format!("{mode:?}"); + let policy = ContainerPolicy { + allow_local_network: true, + network_enforcement_mode: mode, + ..Default::default() + }; + let mut mgr = IngressManager::new("test-container-refused-modes", UNOCCUPIABLE_NETNS_PID); + let mut logger = make_logger(); + + let result = mgr.apply_firewall_rules(&policy, &mut logger); - assert!( - result.is_err(), - "allow_local_network=true, mode=Both: expected Err, got {:?}", - result - ); - let msg = result.unwrap_err(); - assert!( - msg.contains("not yet implemented"), - "mode=Both: message must contain \"not yet implemented\", got: {:?}", - msg - ); - assert!( - msg.contains("allowLocalNetwork"), - "mode=Both: message must contain \"allowLocalNetwork\", got: {:?}", - msg - ); - assert!( - msg.contains("over-broad accept"), - "mode=Both: message must contain \"over-broad accept\", got: {:?}", - msg - ); + assert!( + result.is_err(), + "allow_local_network=true, mode={mode_label}: expected Err, got {:?}", + result + ); + let msg = result.unwrap_err(); + assert!( + msg.contains("not yet implemented"), + "mode={mode_label}: message must contain \"not yet implemented\", got: {:?}", + msg + ); + assert!( + msg.contains("allowLocalNetwork"), + "mode={mode_label}: message must contain \"allowLocalNetwork\", got: {:?}", + msg + ); + assert!( + msg.contains("over-broad accept"), + "mode={mode_label}: message must contain \"over-broad accept\", got: {:?}", + msg + ); + } } /// A refusal must not mark the manager as having applied rules, because @@ -212,29 +216,3 @@ fn default_deny_with_netns_is_not_the_permissive_refusal() { ); } } - -/// allow_local_network=true + NetworkEnforcementMode::Capabilities. -/// -/// Capabilities mode returns early before the firewall path is entered; the -/// permissive guard is never reached. This catches anyone who hoists the guard -/// above the enforcement-mode gate, which would break every capabilities-mode -/// config. -#[test] -fn permissive_inbound_capabilities_mode_is_not_refused() { - let policy = ContainerPolicy { - allow_local_network: true, - network_enforcement_mode: NetworkEnforcementMode::Capabilities, - ..Default::default() - }; - let mut mgr = IngressManager::new("test-container-perm-caps", UNOCCUPIABLE_NETNS_PID); - let mut logger = make_logger(); - - let result = mgr.apply_firewall_rules(&policy, &mut logger); - - // Capabilities mode returns Ok(true) before the firewall path. - assert!( - result.is_ok(), - "allow_local_network=true, mode=Capabilities: expected early Ok, got {:?}", - result - ); -} diff --git a/src/backends/lxc/common/src/network_iptables.rs b/src/backends/lxc/common/src/network_iptables.rs index 496fc6845..6c6e4edb2 100644 --- a/src/backends/lxc/common/src/network_iptables.rs +++ b/src/backends/lxc/common/src/network_iptables.rs @@ -13,9 +13,7 @@ use std::process::Command; use sha2::{Digest, Sha256}; use wxc_common::logger::Logger; -use wxc_common::models::{ - ContainerPolicy, NetworkEnforcementMode, NetworkPolicy, ProxyAddress, ProxyHostPin, -}; +use wxc_common::models::{ContainerPolicy, NetworkPolicy, ProxyAddress, ProxyHostPin}; /// One destination the container is allowed to reach when the policy routes /// egress through a cooperative proxy: an address the proxy host resolved to, @@ -86,6 +84,26 @@ impl FirewallRuleArgs { /// container name, so a chain already present under our name belongs to an /// earlier or concurrent run and is not ours to remove. /// +/// What ` -S FORWARD` was able to tell us. +/// +/// The two failure cases are not the same question. A tool that is not +/// installed cannot be holding a hook, and answering "present" for it invents a +/// residual that teardown can never remove: every attempt fails, ownership is +/// retained, and stop and deprovision report an error after doing everything +/// right. A tool that is installed but whose ruleset would not read is +/// genuinely unknown, and that one has to fail closed. +/// +/// The distinction matters on any IPv6-disabled host, where setup deliberately +/// permits a missing `ip6tables` and installs only v4 state. +enum ForwardProbe { + /// `-S FORWARD` succeeded; this is its output. + Dump(String), + /// The tool could not be spawned, so it holds no rules at all. + ToolAbsent, + /// The tool ran and refused, so its ruleset is unknown. + Unreadable, +} + /// Visible to the crate (with private fields) purely so `signal_cleanup` can /// carry the value from the runner thread to the watchdog thread. The watchdog /// never inspects it; it only hands it back to [`NetworkIptablesManager::force_cleanup`]. @@ -155,7 +173,7 @@ impl CreatedResources { /// Only reachable from the signal path, which is Linux-only; kept /// compiled on every target so Windows and macOS CI still type-check it. #[cfg_attr(not(target_os = "linux"), allow(dead_code))] - fn is_empty(&self) -> bool { + pub(crate) fn is_empty(&self) -> bool { self.v4.is_empty() && self.v6.is_empty() } @@ -464,6 +482,31 @@ impl NetworkIptablesManager { None } + /// The container's host-side veth as it exists on the host right now. + /// + /// [`discover_veth_interface`](Self::discover_veth_interface) reports the + /// name liblxc recorded when it created the interface, which stops being + /// the live name once MXC's pin hook has renamed it. That hook is written + /// into the container's own config and outlives the process that added it, + /// so any later start of that container renames the interface again -- + /// including a start by a caller that never installed the hook and would + /// otherwise scope its rules to a name no interface answers to, leaving the + /// container unfiltered. + /// + /// `pin_hook_present` is that container's own answer to whether it carries + /// the hook, from + /// [`has_veth_pin_hook`](crate::lxc_bindings::LxcContainer::has_veth_pin_hook). + /// The question has to be asked of the container: an interface answering to + /// the pinned name proves only that the name is taken, not that it is taken + /// by this container, and a host that cannot be asked would answer "no" and + /// send teardown back to the stale name. + pub fn live_veth_interface(container_name: &str, pin_hook_present: bool) -> Option { + if pin_hook_present { + return Some(Self::deterministic_veth_name(container_name)); + } + Self::discover_veth_interface(container_name) + } + /// Set the veth interface name for the container. pub fn set_veth_interface(&mut self, iface: &str) { self.veth_interface = Some(iface.to_string()); @@ -1362,11 +1405,371 @@ impl NetworkIptablesManager { Self::run_firewall_command("iptables", args, logger) } + /// FNV-1a over the container name, used to derive a fixed-width token for + /// names with a tight length budget. + /// + /// Not a security primitive: FNV-1a is invertible. It exists to remove the + /// accidental collisions that truncation produces, not to make the token + /// unforgeable. + fn name_hash(name: &str) -> u64 { + const FNV_OFFSET: u64 = 0xcbf2_9ce4_8422_2325; + const FNV_PRIME: u64 = 0x0000_0100_0000_01b3; + + let mut hash = FNV_OFFSET; + for byte in name.bytes() { + hash ^= u64::from(byte); + hash = hash.wrapping_mul(FNV_PRIME); + } + hash + } + + /// Encode the full 64-bit name hash as a fixed 11-character base36 token + /// (`0-9a-z`). 36^11 ≈ 2^56.9, so the hash is reduced modulo 36^11 rather + /// than truncated to 32 bits: this preserves ~56.9 bits of the hash while + /// fitting the tight `IFNAMSIZ` budget of the veth interface name. Base36 + /// is valid in Linux interface names. Zero-padded so the token is always + /// exactly 11 characters, keeping the derived name fixed-width. + fn hash_token(name: &str) -> String { + const ALPHABET: &[u8; 36] = b"0123456789abcdefghijklmnopqrstuvwxyz"; + // 36^11 = 131_621_703_842_267_136 ≈ 2^56.9, fits in u64. + const MODULUS: u64 = 36u64.pow(11); + + let mut value = Self::name_hash(name) % MODULUS; + let mut buf = [b'0'; 11]; + for slot in buf.iter_mut().rev() { + *slot = ALPHABET[(value % 36) as usize]; + value /= 36; + } + // `value` is now 0: 11 base36 digits cover the full [0, 36^11) range. + String::from_utf8(buf.to_vec()).expect("base36 alphabet is valid ASCII") + } + + /// Derive the deterministic host-side veth interface name for a container. + /// + /// The firewall must be installed *before* the container starts, but the + /// veth pair liblxc creates by default has a random name that is only known + /// once the container is running. A `lxc.hook.start-host` hook renames the + /// host end to this name after liblxc creates it but before the container's + /// init runs, which lets the FORWARD hook reference the interface by name + /// ahead of time — iptables accepts a not-yet-existing interface — so there + /// is no window in which a started container has unfiltered network. That + /// hook key carries no interface index, so enforcement does not depend on + /// which `lxc.net.` the container numbers its interface. + /// + /// The name must fit the kernel `IFNAMSIZ` limit of 15 characters and be + /// unique per container, so a `mxcv` prefix (4) is followed by an + /// 11-character base36 hash token (15 chars total). + pub fn deterministic_veth_name(container_name: &str) -> String { + format!("mxcv{}", Self::hash_token(container_name)) + } + + /// Whether this manager currently owns any chain or FORWARD hook. + /// + /// The state-aware start path uses this to decide whether a failed apply + /// left anything of ours behind that must be torn down through *this* + /// manager rather than a fresh one. + pub fn owns_resources(&self) -> bool { + !self.created.is_empty() + } + + /// The set of chains and hooks this manager created, for handing to + /// signal-time cleanup. + pub(crate) fn created(&self) -> CreatedResources { + self.created + } + /// Run an ip6tables command and return success/failure. fn run_ip6tables(args: &[&str], logger: &mut Logger) -> Result { Self::run_firewall_command("ip6tables", args, logger) } + /// Read ` -S FORWARD`, distinguishing a missing tool from a ruleset + /// that would not read. + /// + /// Only [`NotFound`](std::io::ErrorKind::NotFound) means the tool is absent. + /// A spawn can also fail because the executable is there but unusable -- + /// `PermissionDenied`, an exhausted descriptor table, or a transient + /// resource shortage -- and those say nothing about what is installed. + /// Reading them as "absent" would report an empty FORWARD chain on a host + /// whose hooks are still in place, so they fail closed as `Unreadable`. + fn probe_forward_chain(tool: &str) -> ForwardProbe { + match Command::new(tool).args(["-S", "FORWARD"]).output() { + Ok(o) if o.status.success() => { + ForwardProbe::Dump(String::from_utf8_lossy(&o.stdout).into_owned()) + } + Ok(_) => ForwardProbe::Unreadable, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => ForwardProbe::ToolAbsent, + Err(_) => ForwardProbe::Unreadable, + } + } + + /// Whether `chain` currently exists in `tool`'s filter table. + /// + /// `-S ` fails the same way for a chain that is absent and for a + /// ruleset that cannot be read at all, so the exit status alone does not + /// answer the question. FORWARD always exists, so it separates the two: if + /// FORWARD reads, the tool works and this chain is genuinely gone. If it + /// does not, the answer is unknown, and reporting "absent" there would let + /// authoritative teardown declare a clean host while the chain survives. + /// + /// A host without the tool at all reports absent, which is the right answer + /// for teardown: a chain that cannot be addressed cannot be removed, and + /// there is nothing to remove. A tool that exists but could not be run is a + /// different case entirely -- nothing was learned, so it is an error rather + /// than a clean bill of health. + fn chain_exists(tool: &str, chain: &str) -> Result { + match Command::new(tool).args(["-S", chain]).output() { + Ok(o) if o.status.success() => Ok(true), + Ok(o) => match Self::probe_forward_chain(tool) { + ForwardProbe::Dump(_) | ForwardProbe::ToolAbsent => Ok(false), + ForwardProbe::Unreadable => Err(format!( + "{tool} could not read chain {chain} and could not read FORWARD either, so \ + whether the chain exists is unknown: {}", + String::from_utf8_lossy(&o.stderr).trim() + )), + }, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(false), + Err(e) => Err(format!( + "{tool} is installed but could not be run, so whether chain {chain} exists is \ + unknown: {e}" + )), + } + } + + /// Build an ownership record from what is actually installed right now, + /// rather than from what this process created. + /// + /// Teardown is gated on a [`CreatedResources`] record so that a process + /// removes only what it installed. `stop` and `deprovision` cannot satisfy + /// that gate: they run in a different process from the `start` that created + /// the chain, so their record is empty and a record-gated teardown would do + /// nothing. They are nonetheless the documented remedy for a stranded chain + /// -- `apply_network_policy` tells the caller to "stop or deprovision the + /// sandbox to clear it", and the start path `mem::forget`s its manager + /// precisely because stop and deprovision are what remove the chain later. + /// A no-op teardown would strand the chain and leave the sandbox unable to + /// start again, since the next start fails on the chain it finds. + /// + /// Observing live state keeps that authoritative teardown honest. An + /// all-true record would assume every resource exists, then fail `-X` + /// against chains that never did, log those failures, and retain residual + /// ownership that schedules a pointless retry in `Drop`. Probing reports + /// exactly what is present, so an already-clean container issues no + /// commands at all and a half-installed one removes only its own half. + /// Classify the FORWARD rules this backend installs, from a live dump. + /// + /// Every delete in [`teardown_family_forward`](Self::teardown_family_forward) + /// is gated on the matching ownership bit, so a bit left false on the + /// authoritative path leaves a rule installed that nothing will ever + /// remove. Rules are recognized by the matches they carry rather than by + /// their text, because `iptables -S` does not echo back what was submitted: + /// a rule built as `--state ESTABLISHED,RELATED` prints as + /// `RELATED,ESTABLISHED`, so comparing against a rebuilt spec would miss + /// every return rule. + /// + /// The two hook forms are told apart by `--physdev-in` alone, so they are + /// still recognized when the interface is unknown. The return rules jump to + /// ACCEPT rather than to the chain, so the interface is the only thing that + /// identifies them as ours; without it they stay unclaimed. + fn observe_forward_rules(dump: &str, chain: &str, iface: Option<&str>) -> FamilyResources { + let mut found = FamilyResources::default(); + for line in dump.lines() { + let tokens: Vec<&str> = line.split_whitespace().collect(); + if tokens.first() != Some(&"-A") || tokens.get(1) != Some(&"FORWARD") { + continue; + } + let targets = |name: &str| tokens.windows(2).any(|w| w[0] == "-j" && w[1] == name); + let matches_iface = + |opt: &str, val: &str| tokens.windows(2).any(|w| w[0] == opt && w[1] == val); + // The return rules this backend installs are the only ones it may + // claim. An interface-scoped ACCEPT that carries no connection-state + // match belongs to someone else, and claiming it would have teardown + // submit our fuller specification against it -- a delete that cannot + // match, reported forever as a residual nothing can clear. + let accepts_established = || { + tokens.windows(2).any(|w| { + w[0] == "--state" && { + let mut states: Vec<&str> = w[1].split(',').collect(); + states.sort_unstable(); + states == ["ESTABLISHED", "RELATED"] + } + }) + }; + // The same trap one qualifier further out. A foreign rule can carry + // the interface, the state match, and a narrowing this backend never + // writes -- `-p tcp`, an address, a port, a negation -- and the + // delete rebuilt from our own specification would not name it, so it + // could not match. Anything outside the vocabulary these rules are + // built from is therefore somebody else's rule. + let only_our_vocabulary = || { + tokens.iter().all(|t| { + if *t == "!" { + return false; + } + if !t.starts_with('-') { + return true; + } + matches!(*t, "-A" | "-o" | "-m" | "--physdev-out" | "--state" | "-j") + }) + }; + + if targets(chain) { + if tokens.contains(&"--physdev-in") { + found.physdev_hook = true; + } else { + found.hook = true; + } + } + if let Some(i) = iface { + if targets("ACCEPT") && accepts_established() && only_our_vocabulary() { + if matches_iface("--physdev-out", i) { + found.physdev_return = true; + } else if matches_iface("-o", i) { + found.return_rule = true; + } + } + } + } + found + } + + fn observe_existing( + chain_name: &str, + veth_interface: Option<&str>, + ) -> Result { + let family = |tool: &str| -> Result { + let mut observed = match Self::probe_forward_chain(tool) { + ForwardProbe::Dump(dump) => { + Self::observe_forward_rules(&dump, chain_name, veth_interface) + } + // A tool that is not installed holds no rules, so it holds no + // hook. See `hook_present` for why that is an answer and not a + // guess. + ForwardProbe::ToolAbsent => FamilyResources::default(), + // Fail closed on an unreadable FORWARD, for the reason + // `hook_present` gives: claiming no hook would let the chain be + // flushed while a jump into it still stands, and an emptied + // chain that is still hooked returns to its caller instead of + // reaching its own closing DROP, which unfilters a live + // container. Both hook forms are claimed, not just the plain + // one, because either alone gates the flush: clearing the plain + // hook would otherwise open that gate while an unobserved + // physdev hook still referenced the chain. A delete for a rule + // that never existed fails and holds the resource as residual, + // which reports the stop as failed and retries -- the losing + // side of a trade whose other side is an unfiltered container. + ForwardProbe::Unreadable => FamilyResources { + hook: true, + physdev_hook: true, + ..FamilyResources::default() + }, + }; + observed.chain = Self::chain_exists(tool, chain_name)?; + Ok(observed) + }; + Ok(CreatedResources { + v4: family("iptables")?, + v6: family("ip6tables")?, + }) + } + + /// Whether a FORWARD jump to `chain` should be treated as installed, given + /// what [`probe_forward_chain`](Self::probe_forward_chain) could learn. + /// + /// An unreadable FORWARD answers `true`, the same verdict + /// [`remove_forward_hooks`](Self::remove_forward_hooks) gives it. Answering + /// `false` would clear the `created.*_hook` gate in + /// [`teardown_created`](Self::teardown_created), skip hook removal + /// entirely, and let `teardown_chain` flush a chain whose jump is still + /// installed -- and an emptied chain that is still hooked returns to its + /// caller instead of reaching its own closing DROP, which unfilters a live + /// container. Being wrong this way strands a chain for a later pass to + /// reclaim; being wrong the other way removes the filtering. + /// + /// A tool that is not installed is the one case where `false` is not a + /// guess but the answer: it holds no rules, so it holds no hook. Reporting + /// `true` there would record a residual on every IPv6-disabled host, and + /// teardown would then fail forever against an `ip6tables` that does not + /// exist -- turning a clean stop into a reported error. + /// + /// Pure so that fail-open-versus-fail-closed decision can be unit-tested + /// without iptables or a privileged host. + fn hook_present(forward: &ForwardProbe, chain: &str) -> bool { + match forward { + ForwardProbe::Dump(dump) => !Self::forward_hook_deletions(dump, chain).is_empty(), + ForwardProbe::ToolAbsent => false, + ForwardProbe::Unreadable => true, + } + } + + /// Parse ` -S FORWARD` output into a `-D` argument list for every + /// rule that jumps to `chain`. + /// + /// Each `-A FORWARD ... -j ` line becomes the same rule spec with the + /// leading `-A` swapped for `-D`, so the delete matches the exact rule that + /// was appended regardless of the `-i`/`-o` interface qualifiers it carries. + /// Split out from process execution so the matching is testable without + /// iptables, and family-agnostic because both tables print the same syntax. + fn forward_hook_deletions(forward_dump: &str, chain: &str) -> Vec> { + let mut deletions = Vec::new(); + for line in forward_dump.lines() { + let tokens: Vec<&str> = line.split_whitespace().collect(); + if tokens.first() != Some(&"-A") || tokens.get(1) != Some(&"FORWARD") { + continue; + } + let jumps_to_chain = tokens.windows(2).any(|w| w[0] == "-j" && w[1] == chain); + if !jumps_to_chain { + continue; + } + let mut deletion: Vec = tokens.iter().map(|t| t.to_string()).collect(); + deletion[0] = "-D".to_string(); + deletions.push(deletion); + } + deletions + } + + /// Delete every rule in `tool`'s FORWARD chain that jumps to `chain_name`, + /// whatever interface each was scoped to. + /// + /// Reads the live ruleset with ` -S FORWARD` and issues a matching + /// `-D` for each jump, so the hook is removed even when the veth is unknown + /// or was never discovered. Deleting by the remembered `-i ` instead + /// leaves the jump behind on any teardown that never learned the veth -- + /// signal-time `force_cleanup` is exactly that case. Because the chain then + /// stays referenced, the `-X` fails too and the whole chain leaks, in a + /// state nothing later can reclaim. + /// + /// Returns whether FORWARD is now free of jumps to this chain. That verdict + /// gates the `-F` that follows, because flushing is the dangerous half of + /// teardown: an emptied user chain returns to its caller instead of reaching + /// its own closing DROP, so a chain that is still hooked but no longer + /// filtering is a fail-open. Deleting is the safe half -- `-X` on a still + /// referenced chain simply fails and the chain stays, owned and filtering, + /// for a later teardown to retry. + /// + /// An unreadable FORWARD answers `false`: the question was not answered, so + /// it is read as still hooked. Being wrong that way costs a retry; being + /// wrong the other way unfilters a live container. A tool that is not + /// installed answers `true`, for the same reason + /// [`hook_present`](Self::hook_present) reports no hook for it -- it holds + /// no rules, so there is no hook left to remove, and answering `false` + /// would schedule a retry that can never succeed. + fn remove_forward_hooks(tool: &str, chain_name: &str, logger: &mut Logger) -> bool { + let dump = match Self::probe_forward_chain(tool) { + ForwardProbe::Dump(dump) => dump, + ForwardProbe::ToolAbsent => return true, + ForwardProbe::Unreadable => return false, + }; + for deletion in Self::forward_hook_deletions(&dump, chain_name) { + let args: Vec<&str> = deletion.iter().map(String::as_str).collect(); + let _ = Self::run_firewall_command(tool, &args, logger); + } + // Re-read rather than trust the deletes. A `-D` can fail for a rule + // another process is also tearing down, and the exit code does not say + // whether the rule is gone -- only whether this call removed it. + !Self::hook_present(&Self::probe_forward_chain(tool), chain_name) + } + /// Classify whether `ip6tables` is usable, given whether the read-only /// probe succeeded and whether the host currently has active IPv6. Pure so /// the fail-open-vs-fail-closed decision can be unit-tested without a @@ -1625,16 +2028,6 @@ impl NetworkIptablesManager { Ok(()) } - /// Whether the given enforcement mode is served by the iptables firewall - /// backend. Pure and side-effect-free so the gate can be exercised without - /// invoking the host firewall. - fn enforcement_mode_uses_firewall(mode: &NetworkEnforcementMode) -> bool { - matches!( - mode, - NetworkEnforcementMode::Firewall | NetworkEnforcementMode::Both - ) - } - /// Apply network firewall rules based on the container policy. /// /// On any failure after resources are created, the inner call rolls back @@ -1653,35 +2046,13 @@ impl NetworkIptablesManager { policy: &ContainerPolicy, logger: &mut Logger, ) -> Result { - // Skip if network enforcement doesn't use firewall. - if !Self::enforcement_mode_uses_firewall(&policy.network_enforcement_mode) { - // ...unless the policy also carries a proxy, in which case skipping - // is the dangerous outcome rather than the safe one. The runner - // injects HTTP(S)_PROXY from the same policy regardless of what - // happens here, so returning `Ok(true)` with no rules installed - // yields a container that advertises a proxy and restricts nothing: - // any client ignoring the environment reaches the network directly. - // - // The JSON parser rejects this combination, but the parser is not - // the only door. `LxcScriptRunner::execute` and `mxc_engine::run` - // take an already-built `ExecutionRequest`, and - // `NetworkEnforcementMode` derives `Default` as `Capabilities` -- so - // a policy constructed in code gets the unenforced mode without - // anyone choosing it. Restating the invariant here puts it in the - // layer that can actually observe whether rules were installed, - // which is the only layer every caller passes through. - if policy.network_proxy.is_enabled() { - return Err( - "network.proxy requires network.enforcementMode='firewall' or 'both'. \ - This policy enables a proxy under 'capabilities', where no iptables \ - rules are installed, so the proxy environment would be injected while \ - direct egress stayed unrestricted -- any client that ignores HTTP_PROXY \ - would bypass the proxy entirely. Refusing to apply rather than reporting \ - success for an enforcement that did not happen." - .to_string(), - ); - } - logger.log_line("Network enforcement mode does not use firewall, skipping iptables."); + // `enforcementMode` is deliberately not consulted here; a caller may get + // more enforcement than the mode asked for, never less. + if !policy.requires_firewall() { + logger.log_line( + "Network policy requires no egress firewall (permissive default, no host \ + lists, no proxy); skipping iptables.", + ); return Ok(true); } @@ -1784,8 +2155,8 @@ impl NetworkIptablesManager { Err(e) => { let residual = Self::teardown_created( &self.chain_name, - self.veth_interface.as_deref(), &created, + self.veth_interface.as_deref(), logger, ); Err((e, residual)) @@ -2218,35 +2589,82 @@ impl NetworkIptablesManager { /// returning, so signal-time cleanup retries exactly the leftovers. fn teardown_created( chain_name: &str, - veth_interface: Option<&str>, created: &CreatedResources, + veth_interface: Option<&str>, logger: &mut Logger, ) -> CreatedResources { let mut residual = *created; - // Remove from FORWARD only for families this attempt hooked, and only - // the hook forms it actually installed. Both specs come from the same - // builders used at insertion, because iptables deletes by full rule - // specification: a spec that differs by even one match -- `-o` instead - // of `-i`, or the interface rule standing in for the physdev one -- - // finds nothing and leaks the hook. - if let Some(iface) = veth_interface { - Self::teardown_family_forward( - Self::run_iptables_rule_args, - chain_name, - iface, - &created.v4, - &mut residual.v4, - logger, - ); - Self::teardown_family_forward( - Self::run_ip6tables_rule_args, - chain_name, - iface, - &created.v6, - &mut residual.v6, - logger, - ); + // Remove from FORWARD only for families this attempt hooked. + // + // Which route gets there depends on whether the veth is known, because + // iptables deletes a rule by its full specification and every rule this + // backend installs into FORWARD is scoped to the interface. + // + // With the veth the remembered specs are replayed. That is the exact + // delete, and it is the only one that reaches the return-path rules: + // they jump to ACCEPT rather than to our chain, so nothing that + // searches FORWARD for jumps into the chain can see them. + // + // Without it -- signal-time `force_cleanup`, or a veth that was never + // discovered -- there is nothing to replay, and the interface-scoped + // delete removed nothing at all. The hooks survived, `teardown_chain` + // below then correctly refused to flush or delete a still-referenced + // chain, and the result was a leaked chain plus hooks that no later + // pass could reclaim, because every later pass hit the same missing + // veth. Enumerating the live FORWARD chain for jumps to this chain + // needs no interface and clears both hook forms at once. + // + // It cannot reach the return rules. They are marked residual, and + // whether a later pass can reclaim them depends on the interface being + // known by then: `observe_existing` claims a return rule only when it + // can match the interface the rule names, because nothing else marks + // that rule as ours. This path is the one that has no interface, so on + // it the two return rules are a known strand rather than a deferred + // retry. Enumerating is still the better of the two, because it + // reclaims the chain and both hooks rather than leaking all four. + match veth_interface { + Some(iface) => { + Self::teardown_family_forward( + Self::run_iptables_rule_args, + chain_name, + iface, + &created.v4, + &mut residual.v4, + logger, + ); + Self::teardown_family_forward( + Self::run_ip6tables_rule_args, + chain_name, + iface, + &created.v6, + &mut residual.v6, + logger, + ); + // A delete that reported success removed one matching rule. + // iptables holds duplicates, and a jump may carry qualifiers + // this backend never writes, so that exit code is evidence + // about one rule rather than about the chain -- and the flush + // below is gated on the chain. Read FORWARD back and let a + // surviving jump hold the gate shut. It can only hold it, not + // open it, so a failed delete still counts. + Self::hold_flush_if_hooks_survive("iptables", chain_name, iface, &mut residual.v4); + Self::hold_flush_if_hooks_survive("ip6tables", chain_name, iface, &mut residual.v6); + } + None => { + if created.v4.hooks_remain() + && Self::remove_forward_hooks("iptables", chain_name, logger) + { + residual.v4.hook = false; + residual.v4.physdev_hook = false; + } + if created.v6.hooks_remain() + && Self::remove_forward_hooks("ip6tables", chain_name, logger) + { + residual.v6.hook = false; + residual.v6.physdev_hook = false; + } + } } // Flush and delete only the chains this attempt created, and only once @@ -2280,6 +2698,10 @@ impl NetworkIptablesManager { /// Remove one family's FORWARD rules, clearing ownership only where the /// removal succeeded. + /// + /// Requires the veth, because iptables deletes by full rule specification. + /// A teardown that never learned it takes the enumerating path in + /// [`teardown_created`](Self::teardown_created) instead. fn teardown_family_forward( run: fn(&[Vec], &mut Logger) -> Result<(), String>, chain_name: &str, @@ -2336,6 +2758,50 @@ impl NetworkIptablesManager { } } + /// Re-read FORWARD and let a jump that is still there hold the flush gate + /// shut, whatever the deletes reported. + /// + /// Strictly inhibitory: it can only set a hook bit, never clear one. A + /// delete that reported success removed one matching rule, and iptables + /// holds duplicates -- the recovery path in + /// [`force_cleanup_authoritative`](Self::force_cleanup_authoritative) + /// reconstructs ownership for a container whose state was lost, which is + /// where a second jump is likeliest to have accumulated. A jump may also + /// carry qualifiers this backend never writes and still reach the chain. + /// So the exit code is evidence about one rule, and the flush is gated on + /// the chain. + /// + /// Letting it clear a bit instead would make it authoritative, and it is + /// not: it would then overrule a delete that genuinely failed on the word + /// of a probe that can be a moment stale, opening the gate this exists to + /// hold shut. + /// + /// Only the hook bits are touched. They are the ones that gate the flush. + /// The return-path rules jump to `ACCEPT`, reference no chain, and gate + /// nothing. + fn hold_flush_if_hooks_survive( + tool: &str, + chain_name: &str, + iface: &str, + residual: &mut FamilyResources, + ) { + match Self::probe_forward_chain(tool) { + ForwardProbe::Dump(dump) => { + let live = Self::observe_forward_rules(&dump, chain_name, Some(iface)); + residual.hook |= live.hook; + residual.physdev_hook |= live.physdev_hook; + } + // A tool that is not installed holds no FORWARD chain, so nothing + // in it jumps here and there is nothing to add. + ForwardProbe::ToolAbsent => {} + // Unreadable establishes nothing, so it may not open the gate. + ForwardProbe::Unreadable => { + residual.hook = true; + residual.physdev_hook = true; + } + } + } + /// Remove all iptables/ip6tables rules created by this manager. pub fn remove_firewall_rules(&mut self, logger: &mut Logger) -> Result<(), String> { if !self.rules_applied { @@ -2349,15 +2815,26 @@ impl NetworkIptablesManager { let residual = Self::teardown_created( &self.chain_name, - self.veth_interface.as_deref(), &self.created, + self.veth_interface.as_deref(), logger, ); // A removal command can fail, and what survived is still ours. Clearing // the gate here regardless would strand it: Drop would then skip the // retry that is the last chance to remove it. - self.retain_residual_ownership(residual); + if self.retain_residual_ownership(residual) { + // Something survived. Answering `Ok` here told every caller the + // chain was gone while it was still filtering traffic -- stop and + // deprovision both reported success over a chain that outlived + // them and that blocks every later start. Ownership stays set, so + // Drop still gets its retry. + return Err(format!( + "failed to remove every iptables resource for chain {}; it is still installed, \ + and ownership is retained so the drop path can retry", + self.chain_name + )); + } Ok(()) } @@ -2398,6 +2875,73 @@ impl NetworkIptablesManager { mgr.created = created; let _ = mgr.remove_firewall_rules(logger); } + + /// Remove whatever firewall state currently exists for `container_name`, + /// regardless of which process installed it. + /// + /// This is the counterpart to [`force_cleanup`](Self::force_cleanup), and + /// the difference between them is authority, not thoroughness. + /// `force_cleanup` removes only what its caller created, because its + /// callers -- signal rollback, and a start that lost the chain to a + /// concurrent start -- may be looking at a chain somebody else owns and + /// still needs. Deleting it there would leave a live container unfiltered, + /// which is why the empty-record guard exists. + /// + /// `stop` and `deprovision` are in the opposite position. Each has already + /// stopped or destroyed the container before calling this, and propagates a + /// failure rather than continuing, so by the time teardown runs no process + /// is filtered by this chain. Keeping it would protect nothing and would + /// block the next start, which fails on a chain it did not create. They are + /// also what `apply_network_policy` names as the remedy when it finds a + /// stranded chain, so they must actually clear one. + /// + /// The residual race is a `start` of the same sandbox interleaved with a + /// `stop` or `deprovision` of it, which can delete the chain that start + /// just installed. That is not a new exposure: the same interleaving has + /// the stop or destroy tearing the container itself out from under that + /// start. Concurrent lifecycle calls on one sandbox are a caller error, and + /// the alternative -- never clearing a chain whose creator has exited -- + /// strands every state-aware sandbox that ever installed one. + /// + /// Linux-only in practice, like `force_cleanup`; kept compiled everywhere + /// so Windows and macOS CI still type-check it. + #[cfg_attr(not(target_os = "linux"), allow(dead_code))] + pub(crate) fn force_cleanup_authoritative( + container_name: &str, + veth_interface: Option<&str>, + logger: &mut Logger, + ) -> Result<(), String> { + let mut mgr = Self::new(container_name); + // Ask the host what is installed instead of asserting it, so a + // container with nothing left issues no commands and logs no failures. + let observed = Self::observe_existing(&mgr.chain_name, veth_interface)?; + if observed.is_empty() { + return Ok(()); + } + if let Some(v) = veth_interface { + mgr.set_veth_interface(v); + } + // Bypass the rules_applied gate: the manager that set it belonged to + // the start process and is long gone. + mgr.rules_applied = true; + mgr.created = observed; + let attempt = mgr.remove_firewall_rules(logger); + + // The manager's `Drop` retries whatever survived the first pass, so the + // honest answer is what the host reports once that has run -- not what + // the first attempt returned. Reporting the first attempt would raise a + // false alarm every time the retry succeeded. + let chain_name = mgr.chain_name.clone(); + drop(mgr); + match Self::observe_existing(&chain_name, veth_interface) { + Ok(o) if o.is_empty() => Ok(()), + Ok(_) => Err(attempt.err().unwrap_or_else(|| { + format!("iptables state for chain {chain_name} survived authoritative cleanup") + })), + // A probe that could not answer is not evidence of a clean host. + Err(e) => Err(attempt.err().unwrap_or(e)), + } + } } impl Drop for NetworkIptablesManager { @@ -2597,7 +3141,7 @@ mod tests { use super::*; use std::io::{Error, ErrorKind}; use wxc_common::logger::{Logger, Mode}; - use wxc_common::models::{ContainerPolicy, NetworkEnforcementMode, ProxyAddress, ProxyConfig}; + use wxc_common::models::{ContainerPolicy, NetworkEnforcementMode, NetworkPolicy}; /// Build a policy requesting the given enforcement mode, leaving every /// other field at its default. @@ -2902,6 +3446,323 @@ mod tests { ); } + /// A verbatim `iptables -S FORWARD` dump taken from a host running a + /// default-block sandbox. Written down rather than rebuilt from the + /// builders because the point of the exercise is that the kernel does not + /// echo back what was submitted -- note `RELATED,ESTABLISHED` here against + /// the `ESTABLISHED,RELATED` the rules were installed with. + const LIVE_FORWARD_DUMP: &str = "\ +-P FORWARD ACCEPT +-A FORWARD -m physdev --physdev-out mxcvjdmbp6vwk6m -m state --state RELATED,ESTABLISHED -j ACCEPT +-A FORWARD -o mxcvjdmbp6vwk6m -m state --state RELATED,ESTABLISHED -j ACCEPT +-A FORWARD -m physdev --physdev-in mxcvjdmbp6vwk6m -j MXC-mxc-d0e-oozdlgpmvay3kobe +-A FORWARD -i mxcvjdmbp6vwk6m -j MXC-mxc-d0e-oozdlgpmvay3kobe"; + + #[test] + fn every_installed_forward_rule_is_claimed() { + // Teardown deletes only what ownership claims, so a rule this misses is + // a rule that survives stop, deprovision, and every later retry. + let observed = NetworkIptablesManager::observe_forward_rules( + LIVE_FORWARD_DUMP, + "MXC-mxc-d0e-oozdlgpmvay3kobe", + Some("mxcvjdmbp6vwk6m"), + ); + + assert!(observed.hook, "the -i hook must be claimed"); + assert!( + observed.physdev_hook, + "the --physdev-in hook must be claimed" + ); + assert!(observed.return_rule, "the -o return rule must be claimed"); + assert!( + observed.physdev_return, + "the --physdev-out return rule must be claimed" + ); + } + + #[test] + fn a_rule_belonging_to_another_container_is_not_claimed() { + // Claiming a neighbour's rule would delete another sandbox's filtering. + let observed = NetworkIptablesManager::observe_forward_rules( + LIVE_FORWARD_DUMP, + "MXC-someone-else", + Some("mxcvsomeoneelse"), + ); + + assert!( + observed.is_empty(), + "no rule naming another chain or interface may be claimed, got: {observed:?}" + ); + } + + #[test] + fn return_rules_are_left_unclaimed_when_the_interface_is_unknown() { + // Nothing but the interface name marks a return rule as ours -- it + // jumps to ACCEPT, not to our chain. Claiming one on a guess would + // issue a delete for a rule this backend never installed. + let observed = NetworkIptablesManager::observe_forward_rules( + LIVE_FORWARD_DUMP, + "MXC-mxc-d0e-oozdlgpmvay3kobe", + None, + ); + + assert!( + observed.hook && observed.physdev_hook, + "both hook forms are identifiable without the interface" + ); + assert!( + !observed.return_rule && !observed.physdev_return, + "a return rule must not be claimed without the interface that names it" + ); + } + + #[test] + fn a_hook_in_another_table_is_not_read_as_a_forward_hook() { + // Only FORWARD is being torn down here; an identical jump installed in + // INPUT or OUTPUT belongs to the ingress path and is not ours to delete. + let dump = "\ +-A INPUT -i mxcvjdmbp6vwk6m -j MXC-mxc-d0e-oozdlgpmvay3kobe +-A OUTPUT -o mxcvjdmbp6vwk6m -m state --state RELATED,ESTABLISHED -j ACCEPT"; + + let observed = NetworkIptablesManager::observe_forward_rules( + dump, + "MXC-mxc-d0e-oozdlgpmvay3kobe", + Some("mxcvjdmbp6vwk6m"), + ); + + assert!( + observed.is_empty(), + "only FORWARD rules may be claimed, got: {observed:?}" + ); + } + + #[test] + fn the_two_hook_forms_are_claimed_independently() { + // Installing the physdev hook is attempted on either topology and can + // fail without failing the apply, so a chain may carry one form and not + // the other. Claiming a form that was never installed makes teardown + // issue a delete that fails, and a failed delete is held as a residual + // that reports the stop as failed. + let dump = "-A FORWARD -i mxcvjdmbp6vwk6m -j MXC-solo"; + + let observed = NetworkIptablesManager::observe_forward_rules( + dump, + "MXC-solo", + Some("mxcvjdmbp6vwk6m"), + ); + + assert!(observed.hook, "the hook that is present must be claimed"); + assert!( + !observed.physdev_hook, + "a hook form that is absent must not be claimed" + ); + + let physdev_only = "-A FORWARD -m physdev --physdev-in mxcvjdmbp6vwk6m -j MXC-solo"; + + let observed = NetworkIptablesManager::observe_forward_rules( + physdev_only, + "MXC-solo", + Some("mxcvjdmbp6vwk6m"), + ); + + assert!( + observed.physdev_hook, + "the physdev hook that is present must be claimed" + ); + assert!( + !observed.hook, + "a physdev hook must not also be claimed as a plain hook" + ); + } + + #[test] + fn the_two_return_forms_are_claimed_independently() { + // Same trade as the hooks: one form present must not be read as both, + // or teardown deletes a rule that was never installed. + let plain_only = + "-A FORWARD -o mxcvjdmbp6vwk6m -m state --state RELATED,ESTABLISHED -j ACCEPT"; + + let observed = NetworkIptablesManager::observe_forward_rules( + plain_only, + "MXC-solo", + Some("mxcvjdmbp6vwk6m"), + ); + + assert!(observed.return_rule, "the -o return rule must be claimed"); + assert!( + !observed.physdev_return, + "a plain return rule must not also be claimed as a physdev one" + ); + + let physdev_only = "-A FORWARD -m physdev --physdev-out mxcvjdmbp6vwk6m \ + -m state --state RELATED,ESTABLISHED -j ACCEPT"; + + let observed = NetworkIptablesManager::observe_forward_rules( + physdev_only, + "MXC-solo", + Some("mxcvjdmbp6vwk6m"), + ); + + assert!( + observed.physdev_return, + "the --physdev-out return rule must be claimed" + ); + assert!( + !observed.return_rule, + "a physdev return rule must not also be claimed as a plain one" + ); + } + + #[test] + fn a_stateful_accept_carrying_a_qualifier_we_never_write_is_not_ours() { + // The interface and the connection-state match together are still not + // enough. A foreign rule can carry both and narrow further, and the + // delete this backend rebuilds names no such narrowing -- so it cannot + // match, and the rule is held as a residual no later pass can clear. + for foreign in [ + "-A FORWARD -o mxcvjdmbp6vwk6m -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT", + "-A FORWARD -o mxcvjdmbp6vwk6m -d 10.0.0.0/8 \ + -m state --state RELATED,ESTABLISHED -j ACCEPT", + "-A FORWARD -m physdev --physdev-out mxcvjdmbp6vwk6m -p udp \ + -m state --state RELATED,ESTABLISHED -j ACCEPT", + ] { + let observed = NetworkIptablesManager::observe_forward_rules( + foreign, + "MXC-solo", + Some("mxcvjdmbp6vwk6m"), + ); + + assert!( + !observed.return_rule && !observed.physdev_return, + "a rule narrowed by something this backend never writes must not be claimed: \ + {foreign}" + ); + } + } + + #[test] + fn an_accept_without_the_connection_state_match_is_not_ours() { + // The interface name alone does not make a rule ours. A host rule + // accepting everything on the same interface would be claimed on the + // name and then deleted with our fuller specification -- a delete that + // cannot match, held as a residual no later pass can ever clear. + let dump = "-A FORWARD -o mxcvjdmbp6vwk6m -j ACCEPT"; + + let observed = NetworkIptablesManager::observe_forward_rules( + dump, + "MXC-solo", + Some("mxcvjdmbp6vwk6m"), + ); + + assert!( + !observed.return_rule && !observed.physdev_return, + "an ACCEPT carrying no connection-state match must not be claimed, got: {observed:?}" + ); + } + + #[test] + fn an_unreadable_forward_is_read_as_still_hooked() { + // A tool that ran and refused leaves the question unanswered -- another + // process holding the xtables lock, or no privilege to read. Reading + // that silence as "no hook" clears the gate in teardown_created, skips + // hook removal, and sends teardown on to flush a chain FORWARD may + // still jump to, unfiltering a live container. + assert!( + NetworkIptablesManager::hook_present(&ForwardProbe::Unreadable, "MXC-abc123"), + "an unanswered probe must not be read as an absent hook" + ); + + // Negative control: a FORWARD that really is free of our jump must + // still report not-hooked, so the assertion above cannot be satisfied + // by always answering true and stranding every chain. + assert!( + !NetworkIptablesManager::hook_present( + &ForwardProbe::Dump( + "-P FORWARD ACCEPT\n-A FORWARD -i veth0 -j SOMEONE-ELSE\n".to_string() + ), + "MXC-abc123" + ), + "a readable FORWARD without our jump must report not hooked" + ); + } + + #[test] + fn a_tool_that_is_not_installed_holds_no_hook() { + // Distinct from the unreadable case above, and the distinction is the + // whole point. On an IPv6-disabled host setup deliberately permits a + // missing ip6tables and installs only v4 state. Calling that absence + // "hooked" records a v6 residual that no teardown can ever clear: every + // removal attempt fails against a binary that is not there, ownership is + // retained, and a stop that did everything right reports an error. + assert!( + !NetworkIptablesManager::hook_present(&ForwardProbe::ToolAbsent, "MXC-abc123"), + "a tool that cannot be spawned holds no rules, so it holds no hook" + ); + } + + #[test] + fn a_missing_binary_is_absent_but_an_unrunnable_one_is_unknown() { + // `Command::output()` fails for more reasons than a missing executable, + // and only one of them means "there are no rules here". Permission + // denied, an exhausted descriptor table, or a transient resource + // shortage all leave the ruleset exactly as it was, so reading them as + // absent would let an authoritative teardown certify a host whose hooks + // are untouched. + assert!( + matches!( + NetworkIptablesManager::probe_forward_chain("mxc-no-such-firewall-tool"), + ForwardProbe::ToolAbsent + ), + "a binary that is not installed is genuinely absent" + ); + assert_eq!( + NetworkIptablesManager::chain_exists("mxc-no-such-firewall-tool", "MXC-abc123"), + Ok(false), + "a chain that cannot be addressed at all cannot exist" + ); + } + + #[cfg(unix)] + #[test] + fn a_binary_that_will_not_execute_is_never_read_as_absent() { + let dir = std::env::temp_dir().join(format!("mxc-unrunnable-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("test temp dir"); + // Present on disk, but not executable: the spawn fails with + // PermissionDenied rather than NotFound, which says nothing about what + // is installed in the filter table. + let tool = dir.join("iptables"); + std::fs::write(&tool, "#!/bin/sh\nexit 0\n").expect("seed a non-executable tool"); + let tool = tool.to_string_lossy().to_string(); + + assert!( + matches!( + NetworkIptablesManager::probe_forward_chain(&tool), + ForwardProbe::Unreadable + ), + "a tool that exists but will not run leaves FORWARD unknown, not empty" + ); + assert!( + NetworkIptablesManager::chain_exists(&tool, "MXC-abc123").is_err(), + "an unrunnable tool must not answer the chain question at all" + ); + + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_readable_forward_carrying_our_jump_is_read_as_hooked() { + assert!( + NetworkIptablesManager::hook_present( + &ForwardProbe::Dump( + "-P FORWARD ACCEPT\n-A FORWARD -i veth0 -j MXC-abc123\n".to_string() + ), + "MXC-abc123" + ), + "a FORWARD dump containing our jump must report hooked" + ); + } + #[test] fn a_flush_is_withheld_while_the_chain_is_still_hooked() { // -F succeeds no matter who references the chain, and an emptied user @@ -3020,6 +3881,43 @@ mod tests { ); } + #[test] + fn a_removal_that_left_the_chain_installed_says_so() { + // The residual was retained but never reported. remove_firewall_rules + // answered Ok whether or not anything survived, so stop and + // deprovision told their callers the filtering was gone while the + // chain was still installed -- and still blocking the next start of + // that container name, which fails on a chain it did not create. + let fake = test_firewall::install(); + fake.fail_every_command("iptables: permission denied"); + + let mut manager = NetworkIptablesManager::new("stranded"); + manager.retain_residual_ownership(CreatedResources::for_test(true, false, false, false)); + + let mut logger = Logger::new(Mode::Buffer); + assert!( + manager.remove_firewall_rules(&mut logger).is_err(), + "a teardown that left the chain installed must not report success" + ); + } + + #[test] + fn a_removal_that_removed_everything_reports_success() { + // Negative control for the assertion above: the error has to come from + // the residual, not from every teardown. Without this, returning Err + // unconditionally would satisfy the test above. + let _fake = test_firewall::install(); + + let mut manager = NetworkIptablesManager::new("departed"); + manager.retain_residual_ownership(CreatedResources::for_test(true, false, false, false)); + + let mut logger = Logger::new(Mode::Buffer); + assert!( + manager.remove_firewall_rules(&mut logger).is_ok(), + "a teardown whose commands all succeeded must report success" + ); + } + #[test] fn a_second_apply_is_refused_while_the_first_still_owns_resources() { // Both arms of apply_firewall_rules replace self.created with the new @@ -4132,10 +5030,18 @@ mod tests { } #[test] - fn a_non_firewall_policy_is_a_successful_no_op() { + fn a_policy_that_requires_no_firewall_is_a_successful_no_op() { + // Default-allow, no host lists, no proxy: the one policy shape that asks + // for no filtering at all. The gate must report it as a clean no-op and + // install nothing -- observed through the fake, which records every + // command an apply would have issued. + let fake = test_firewall::install(); let mut manager = NetworkIptablesManager::new("skip-noop"); manager.set_veth_interface("veth-skip"); - let policy = policy_with_enforcement_mode(NetworkEnforcementMode::Capabilities); + let policy = ContainerPolicy { + default_network_policy: NetworkPolicy::Allow, + ..Default::default() + }; let mut logger = Logger::new(Mode::Buffer); let result = manager.apply_firewall_rules(&policy, &mut logger); @@ -4143,92 +5049,83 @@ mod tests { assert_eq!( result, Ok(true), - "a policy that does not use firewall enforcement must be reported as a successful no-op" + "a policy that requires no firewall must be reported as a successful no-op" ); assert!( !manager.rules_applied(), "a no-op firewall skip must leave no rules marked as applied" ); + assert!( + fake.issued().is_empty(), + "a no-op must not issue a single command, got: {:?}", + fake.issued() + ); } #[test] - fn every_enforcement_mode_takes_the_contractual_firewall_gate() { - for (mode, uses_firewall) in enforcement_modes_with_firewall_contract() { - assert_eq!( - NetworkIptablesManager::enforcement_mode_uses_firewall(&mode), - uses_firewall, - "{mode:?} firewall-gate predicate mismatch" - ); - } - } - - // The JSON parser rejects proxy-under-capabilities, but it is not the only - // way in: `LxcScriptRunner::execute` and `mxc_engine::run` take an - // already-built `ExecutionRequest`. Skipping here would report success for - // an enforcement that never happened, while the runner still injects the - // proxy environment -- a container that advertises a proxy and restricts - // nothing. - #[test] - fn a_proxy_under_a_non_firewall_mode_is_refused_rather_than_skipped() { - // `Capabilities` is the only mode the firewall gate rejects, and it is - // also `NetworkEnforcementMode`'s `Default` -- so this is what a policy - // built in code gets when nobody sets the field at all. - let mut policy = policy_with_enforcement_mode(NetworkEnforcementMode::Capabilities); - policy.network_proxy = ProxyConfig { - address: Some(ProxyAddress::new("10.0.0.5".to_string(), 3128)), - builtin_test_server: false, - }; - let mut manager = NetworkIptablesManager::new("proxy-gate"); + fn a_default_block_policy_installs_rules_without_an_enforcement_mode() { + // The core behavior change, seen through the commands: the struct + // default is `block`, and with the mode gone that alone must drive an + // install. Before the change this policy took the no-op path and the + // container reached the whole internet. + let fake = test_firewall::install(); + let mut manager = NetworkIptablesManager::new("default-block"); + manager.set_veth_interface("veth-block0"); + let policy = ContainerPolicy::default(); let mut logger = Logger::new(Mode::Buffer); let result = manager.apply_firewall_rules(&policy, &mut logger); - let error = result.expect_err( - "a proxy under an enforcement mode that installs no rules must not report success", - ); assert!( - error.contains("enforcementMode"), - "the error must name the setting that has to change; got: {error}" + result.is_ok(), + "a default-block policy must apply, got: {:?}", + result ); assert!( - !manager.rules_applied(), - "a refused apply must leave no rules marked as applied" + fake.issued().contains(&strings(&[ + "iptables", + "-N", + &chain_name_for("default-block") + ])), + "a default-block policy must create the filtering chain, got: {:?}", + fake.issued() ); } - // `builtin_test_server` enables the proxy without an address, and it takes - // the same injection path, so the gate cannot key on the address alone. #[test] - fn the_builtin_test_server_proxy_is_gated_the_same_way() { - let mut policy = policy_with_enforcement_mode(NetworkEnforcementMode::Capabilities); - policy.network_proxy = ProxyConfig { - address: None, - builtin_test_server: true, + fn an_explicit_capabilities_mode_installs_the_same_rules_as_omitting_it() { + // `enforcementMode` is now parsed and ignored: an explicit + // `capabilities` must produce exactly the commands an omitted mode does, + // never fewer. A caller may get more enforcement than asked for, never + // less. + let restriction = ContainerPolicy { + blocked_hosts: vec!["203.0.113.9".to_string()], + ..Default::default() + }; + let explicit = ContainerPolicy { + network_enforcement_mode: NetworkEnforcementMode::Capabilities, + ..restriction.clone() }; - let mut manager = NetworkIptablesManager::new("builtin-gate"); - let mut logger = Logger::new(Mode::Buffer); - - assert!( - manager.apply_firewall_rules(&policy, &mut logger).is_err(), - "an address-free proxy is still a proxy and must not be silently unenforced" - ); - } - // The refusal must be narrow: without a proxy there is nothing to leave - // unenforced, so `capabilities` remains an ordinary supported mode. - #[test] - fn a_proxy_free_policy_still_skips_cleanly_under_capabilities() { - let policy = policy_with_enforcement_mode(NetworkEnforcementMode::Capabilities); - let mut manager = NetworkIptablesManager::new("no-proxy-skip"); - let mut logger = Logger::new(Mode::Buffer); + let issued_for = |policy: &ContainerPolicy| -> Vec> { + let fake = test_firewall::install(); + let mut manager = NetworkIptablesManager::new("mode-parity"); + manager.set_veth_interface("veth-parity0"); + let mut logger = Logger::new(Mode::Buffer); + let _ = manager.apply_firewall_rules(policy, &mut logger); + fake.issued() + }; assert_eq!( - manager.apply_firewall_rules(&policy, &mut logger), - Ok(true), - "capabilities mode without a proxy must stay a successful no-op" + issued_for(&explicit), + issued_for(&restriction), + "an explicit capabilities mode must issue the same commands as omitting it" ); } + /// Build a policy carrying the given enforcement mode over the struct + /// defaults. The default policy is `block`, so every mode this produces + /// requires the firewall -- the tests below use it to drive a real install. fn policy_with_enforcement_mode( network_enforcement_mode: NetworkEnforcementMode, ) -> ContainerPolicy { @@ -4237,16 +5134,6 @@ mod tests { ..Default::default() } } - - /// The expected answers are written out as literals rather than derived from - /// a second copy of the predicate. A test that recomputes the contract it is - /// checking passes even when both copies are wrong in the same way. - fn enforcement_modes_with_firewall_contract() -> [(NetworkEnforcementMode, bool); 3] { - use NetworkEnforcementMode::{Both, Capabilities, Firewall}; - - [(Capabilities, false), (Firewall, true), (Both, true)] - } - // ----------------------------------------------------------------------- // Spec-derived tests: ip6tables status // ----------------------------------------------------------------------- diff --git a/src/backends/lxc/common/src/network_iptables_veth_spec.rs b/src/backends/lxc/common/src/network_iptables_veth_spec.rs index 2b14eac83..8e13066e1 100644 --- a/src/backends/lxc/common/src/network_iptables_veth_spec.rs +++ b/src/backends/lxc/common/src/network_iptables_veth_spec.rs @@ -165,16 +165,18 @@ fn apply_tears_down_the_chain_it_created_when_it_fails_closed() { ); } -// A container that never asked for a firewall (`Capabilities` is the default -// enforcement mode) must not be punished for an interface the caller was -// never required to set. Any firewall command touching the host here would -// be an unrequested side effect on a container that opted out of firewalling -// entirely. +// A policy that requires no firewall at all -- default-allow, no host lists, +// no proxy -- must not be punished for an interface the caller was never +// required to set. Any firewall command touching the host here would be an +// unrequested side effect on a container that opted out of filtering. #[test] -fn capabilities_only_container_is_unaffected_by_a_missing_veth_interface() { +fn a_policy_requiring_no_firewall_is_unaffected_by_a_missing_veth_interface() { let fake = super::test_firewall::install(); let mut manager = NetworkIptablesManager::new("ctrl-capsonly"); - let policy = policy_requesting(NetworkEnforcementMode::Capabilities); + let policy = ContainerPolicy { + default_network_policy: wxc_common::models::NetworkPolicy::Allow, + ..Default::default() + }; let mut logger = Logger::new(Mode::Buffer); let _ = fake.forget_issued(); @@ -182,12 +184,12 @@ fn capabilities_only_container_is_unaffected_by_a_missing_veth_interface() { assert!( result.is_ok(), - "Capabilities mode must not fail just because the veth interface is unknown, got {:?}", + "a policy requiring no firewall must not fail just because the veth interface is unknown, got {:?}", result ); assert!( fake.issued().is_empty(), - "Capabilities-only enforcement must not issue any iptables commands, issued: {:?}", + "a policy requiring no firewall must not issue any iptables commands, issued: {:?}", fake.issued() ); } diff --git a/src/backends/lxc/common/src/signal_cleanup.rs b/src/backends/lxc/common/src/signal_cleanup.rs index a58f989ea..ffa647122 100644 --- a/src/backends/lxc/common/src/signal_cleanup.rs +++ b/src/backends/lxc/common/src/signal_cleanup.rs @@ -24,14 +24,154 @@ use nix::sys::signal::{SigSet, Signal}; #[cfg(target_os = "linux")] use crate::lxc_bindings::LxcContainer; -#[cfg(target_os = "linux")] -use crate::network_ingress::IngressManager; use crate::network_iptables::CreatedResources; #[cfg(target_os = "linux")] use crate::network_iptables::NetworkIptablesManager; #[cfg(target_os = "linux")] use wxc_common::logger::{Logger, Mode}; +/// How much of the active sandbox a fatal signal should roll back. +/// +/// The one-shot runner and the state-aware lifecycle both install a firewall +/// chain, but they own the container very differently, and rolling back the +/// wrong amount is damaging in both directions. Destroying a provisioned +/// state-aware container would discard a resource its owner expects to survive +/// the process; leaving a one-shot container behind would leak it forever. +#[derive(Default, Clone, Copy, PartialEq, Eq, Debug)] +enum SignalRollback { + /// One-shot: this process created the container to run a single script, so + /// a signal takes the firewall and the container with it. + #[default] + DestroyContainer, + /// State-aware: the container is provisioned and deliberately outlives this + /// process, so a signal removes only the firewall this process installed. + /// The container is stopped, not destroyed, and left for a later `stop` or + /// `deprovision` to reclaim. + NetworkOnly, + /// State-aware `exec`: the container, its firewall, and its ingress rules + /// are all meant to survive. Only the processes this exec started inside + /// the container are rolled back, because killing this process kills the + /// host-side attach and nothing else -- the container is persistent, so its + /// descendants would otherwise run on into the next exec. + ReapExec, +} + +/// One action the watchdog takes on a fatal signal. +/// +/// The watchdog is Linux-only, so this is dead code elsewhere. It stays +/// compiled on every target rather than being `cfg`-gated so Windows and macOS +/// CI still type-check and test the ordering. +#[cfg_attr(not(target_os = "linux"), allow(dead_code))] +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +enum RollbackStep { + /// Halt the container without discarding it. + StopContainer, + /// Remove the firewall chain and hooks this process created. + RemoveFirewall, + /// Discard the container entirely. + DestroyContainer, + /// Kill the processes one exec started inside the container, found by the + /// marker that exec was stamped with. + ReapExec, +} + +/// The steps a signal rollback runs, in the order they must run. +/// +/// Ordering is the whole content of this function, and it differs by rollback +/// kind for a reason. +/// +/// A state-aware start installs the chain *before* `lxc-start` so the container +/// is never up without it. The rollback has to preserve that invariant in +/// reverse: `lxc-start` may already have succeeded when the signal lands, so +/// removing the firewall first would leave a running container unfiltered with +/// no process left to notice. Stopping first cannot produce that state. +/// +/// The one-shot path runs the same invariant rather than the mirror of it. Its +/// container is going away entirely and `destroy` subsumes stopping it, which +/// once argued for removing the firewall first, while the name is still +/// unambiguous. But `lxc-destroy` can fail, and that order would then have +/// stripped the egress chain off a container that is still up. Destroy runs +/// first and the firewall is removed only once it has succeeded, which is what +/// the ordinary deprovision path already does. +/// +/// The inbound rules are different in kind. They live inside the container's +/// own network namespace, reachable only by entering it through the init PID, +/// so they cease to exist when the container does. That is why only the +/// one-shot path removes them, and why it does so before `destroy` -- after +/// that there is no namespace left to enter. The stop path deliberately omits +/// them: the only moment it could remove them is *before* the stop, which is +/// exactly the unfiltered-and-still-running state this ordering exists to +/// prevent, and stopping discards them anyway. +/// +/// A process that never created the chain must not remove one: the chain name +/// depends only on the container name, so the name may by now answer for a +/// different, live container. +/// +/// The exec rollback shares none of that. Nothing it touches was installed by +/// start, so it removes no firewall and stops no container -- it kills only the +/// processes carrying this exec's marker, which no other owner can be holding. +#[cfg_attr(not(target_os = "linux"), allow(dead_code))] +fn rollback_plan(rollback: SignalRollback, owns_firewall: bool) -> Vec { + let mut plan = Vec::new(); + match rollback { + SignalRollback::ReapExec => { + plan.push(RollbackStep::ReapExec); + } + SignalRollback::NetworkOnly => { + plan.push(RollbackStep::StopContainer); + if owns_firewall { + plan.push(RollbackStep::RemoveFirewall); + } + } + SignalRollback::DestroyContainer => { + // No ingress step: those rules live inside the container's own + // netns, so the destroy takes them with it. Running one first would + // buy nothing when the destroy succeeds and cost real exposure when + // it fails -- a container left running with its inbound deny + // already stripped off. The firewall goes last, after the destroy + // has actually succeeded -- see `execute_rollback`. + plan.push(RollbackStep::DestroyContainer); + if owns_firewall { + plan.push(RollbackStep::RemoveFirewall); + } + } + } + plan +} + +/// Runs `plan`, asking `run_step` to perform each step and report whether it +/// succeeded. +/// +/// A failed `StopContainer` abandons the rest of the plan. The steps after it +/// exist to clean up a container that is no longer running, and the only one +/// that follows it is `RemoveFirewall` -- so continuing would strip the egress +/// chain off a container that is still up, which is precisely the state the +/// ordering above exists to prevent. Ordering alone does not achieve that; +/// `lxc-stop` can fail, and then the order it ran in no longer matters. +/// +/// A failed `DestroyContainer` abandons the rest for the same reason. The only +/// step that follows it is `RemoveFirewall`, and a destroy that failed may well +/// have left the container running, so continuing would unfilter it. +/// +/// Bailing out leaks the chain rather than exposing the container, which is the +/// same trade the ordinary stop path already makes deliberately: it propagates +/// the stop error and leaves the rules in place rather than unfilter a +/// still-running container (`state_aware.rs`, `stop`). +#[cfg_attr(not(target_os = "linux"), allow(dead_code))] +fn execute_rollback(plan: &[RollbackStep], run_step: &mut impl FnMut(RollbackStep) -> bool) { + for &step in plan { + let ok = run_step(step); + if !ok + && matches!( + step, + RollbackStep::StopContainer | RollbackStep::DestroyContainer + ) + { + return; + } + } +} + /// What the watchdog needs to roll back on a fatal signal: the container /// name (so we can `lxc-destroy` it), the host-side veth interface when /// known (so we can also remove the iptables FORWARD hook the runner @@ -48,8 +188,26 @@ use wxc_common::logger::{Logger, Mode}; struct ActiveSandbox { name: Option, veth: Option, + /// Read only by the Linux watchdog, but written on every target. + #[cfg_attr(not(target_os = "linux"), allow(dead_code))] + rollback: SignalRollback, + /// Which per-family chains and FORWARD hooks this process created for + /// `name`, so the watchdog removes only what this process installed. + /// + /// The chain name is derived from the container name, so two processes + /// starting the same sandbox target the same chain and only one of them + /// creates it. Without this the watchdog would tear the chain down on a + /// signal no matter which process it interrupted, and interrupting the + /// loser would strip the firewall off the winner's running container. The + /// record is published the moment a creating command succeeds, so the + /// whole window in which the resource exists is covered and no wider. + #[cfg_attr(not(target_os = "linux"), allow(dead_code))] created: CreatedResources, - netns_pid: Option, + /// The marker stamped on the exec currently running in `name`, when one is + /// running, so a signal can reap the processes it started inside the + /// container. + #[cfg_attr(not(target_os = "linux"), allow(dead_code))] + exec_marker: Option, } static ACTIVE_CONTAINER: OnceLock> = OnceLock::new(); @@ -72,38 +230,73 @@ pub fn set_active(name: &str) { let mut slot = lock_slot(); slot.name = Some(name.to_owned()); slot.veth = None; + slot.rollback = SignalRollback::DestroyContainer; slot.created = CreatedResources::default(); - slot.netns_pid = None; + slot.exec_marker = None; } -/// Records the host-side veth interface for the active container so the -/// watchdog can also remove the iptables FORWARD hook on a fatal signal. -/// No-op if no container is currently registered. -pub fn set_active_veth(veth: &str) { +/// Records `name` as the currently active container for the *state-aware* +/// lifecycle, where a fatal signal must remove the firewall this process +/// installed but must not destroy the container. +/// +/// The state-aware `start` phase installs the chain before the container runs, +/// so a signal in that window would otherwise leave the chain behind with no +/// one to remove it — the watchdog only acts on a registered name, and until +/// now only the one-shot runner ever registered one. Registering with +/// [`set_active`] instead would be worse than leaking: it would destroy a +/// container that was provisioned to outlive this process. +/// +/// Call [`clear_active`] once the start has succeeded. Leaving the +/// registration in place past that point would let a later signal strip the +/// firewall off a container that is up and running. +pub fn set_active_network_only(name: &str) { let mut slot = lock_slot(); - if slot.name.is_some() { - slot.veth = Some(veth.to_owned()); - } + slot.name = Some(name.to_owned()); + slot.veth = None; + slot.rollback = SignalRollback::NetworkOnly; + slot.created = CreatedResources::default(); + slot.exec_marker = None; } -/// Records the container's init PID for the active container so the watchdog -/// can remove the container-netns iptables INPUT (inbound) rules on a fatal -/// signal, before the container is destroyed. No-op if no container is -/// currently registered. -pub fn set_active_pid(pid: u32) { +/// Records the exec running in `name` under `marker`, so a fatal signal reaps +/// the processes it started inside the container. +/// +/// The exec phase registered nothing before this. `start` clears the +/// registration once it succeeds -- correctly, since a later signal must not +/// strip the firewall off a running container -- which left the whole exec +/// window uncovered. A signal there killed the host-side `lxc-attach` and +/// returned, while the script's descendants carried on inside a container that +/// is persistent by design and would hand them to the next exec. +/// +/// This does not resurrect the start-time registration. Nothing here stops the +/// container or touches its rules; they are all meant to survive the exec, and +/// the marker reaches only what this exec started. +/// +/// Call [`clear_active`] as soon as the attach returns. +pub fn set_active_exec(name: &str, marker: &str) { let mut slot = lock_slot(); - if slot.name.is_some() { - slot.netns_pid = Some(pid); - } + slot.name = Some(name.to_owned()); + slot.veth = None; + slot.rollback = SignalRollback::ReapExec; + slot.created = CreatedResources::default(); + slot.exec_marker = Some(marker.to_owned()); } -/// Records which iptables chains and FORWARD hooks the runner has created so -/// far, so signal-time cleanup removes exactly those and nothing else. +/// Records which iptables chains and FORWARD hooks this process created for +/// the active container, so the watchdog removes exactly those on a fatal +/// signal. +/// +/// Called by [`crate::network_iptables::NetworkIptablesManager`] the moment a +/// creating command succeeds — the instant the resource starts existing, and +/// not before. Registering a name is deliberately not enough on its own: a +/// process whose `-N` lost the race to a concurrent start of the same sandbox +/// owns nothing, and a signal must not make it delete the winner's chain and +/// leave the winner's container running unfiltered. /// -/// No-op when no container is registered. Backends that never call -/// [`set_active`] — Bubblewrap builds the same firewall manager but installs -/// no watchdog — therefore publish nothing, which keeps the watchdog from -/// acting on a lifecycle it does not manage. +/// Publication is whole-record rather than incremental: each call supersedes +/// the previous one, so callers pass the complete set they own. +/// +/// No-op if no container is currently registered. pub(crate) fn set_active_created(created: CreatedResources) { let mut slot = lock_slot(); if slot.name.is_some() { @@ -111,28 +304,37 @@ pub(crate) fn set_active_created(created: CreatedResources) { } } +/// Unregisters the active sandbox, so a later signal rolls nothing back. +/// +/// Used at the end of a successful state-aware start: the chain and the +/// container are both meant to persist from there on, and rolling either back +/// would be the bug rather than the fix. +/// +/// Also the mirror of [`set_active_created`] after a successful teardown. +/// Chain names depend only on the container name, so an ownership record left +/// published after a successful teardown would let a later signal run cleanup +/// against a name that by then may answer for a different, live container — +/// stripping its firewall while it runs. +pub fn clear_active() { + *lock_slot() = ActiveSandbox::default(); +} + +/// Records the host-side veth interface for the active container so the +/// watchdog can also remove the iptables FORWARD hook on a fatal signal. +/// No-op if no container is currently registered. +pub fn set_active_veth(veth: &str) { + let mut slot = lock_slot(); + if slot.name.is_some() { + slot.veth = Some(veth.to_owned()); + } +} + /// Reads back what the watchdog would act on. Test-only: production code has /// exactly one reader, and it is the watchdog itself. #[cfg(test)] -fn active_snapshot() -> ( - Option, - Option, - CreatedResources, - Option, -) { +fn active_snapshot() -> (Option, Option, CreatedResources) { let slot = lock_slot(); - ( - slot.name.clone(), - slot.veth.clone(), - slot.created, - slot.netns_pid, - ) -} - -/// Returns the slot to its process-start state so a test leaves nothing behind. -#[cfg(test)] -fn clear_active() { - *lock_slot() = ActiveSandbox::default(); + (slot.name.clone(), slot.veth.clone(), slot.created) } /// Block SIGHUP/SIGTERM/SIGINT in the calling thread and spawn a watchdog @@ -186,6 +388,20 @@ pub fn install() -> Result<(), String> { Ok(()) } +/// Whether a stop attempt left the container safe to unfilter. +/// +/// `lxc-stop -k` exits non-zero on a container that is not running, so a kill +/// that reported failure does not on its own mean the container survived. A +/// state that could not be read stays a failure, keeping the filtering in +/// place for a container that might still be transmitting. +/// +/// Only the Linux watchdog calls this, but it stays compiled on every target +/// so Windows and macOS CI still type-check and test the decision. +#[cfg_attr(not(target_os = "linux"), allow(dead_code))] +fn stop_left_container_down(killed: bool, running_after: Option) -> bool { + killed || running_after == Some(false) +} + #[cfg(target_os = "linux")] fn run_watchdog(mask: SigSet) -> ! { loop { @@ -193,28 +409,41 @@ fn run_watchdog(mask: SigSet) -> ! { let Ok(sig) = mask.wait() else { continue }; let active = std::mem::take(&mut *lock_slot()); if let Some(name) = active.name { - // Remove iptables rules first so the FORWARD hook and chain - // don't outlive the container. The veth disappears once the - // container is destroyed below; cleaning up first avoids a - // dangling reference. Best-effort with a buffered logger so - // signal-time output doesn't interleave with whatever else - // might still be writing to the host's stdio. + // Best-effort, with a buffered logger so signal-time output doesn't + // interleave with whatever else might still be writing to the + // host's stdio. The order comes from `rollback_plan`, which is + // where the reasoning about it lives. let mut buf_logger = Logger::new(Mode::Buffer); - NetworkIptablesManager::force_cleanup( - &name, - active.veth.as_deref(), - active.created, - &mut buf_logger, - ); - // Also remove the container-netns inbound INPUT rules while the - // netns still exists (it vanishes when the container is destroyed - // below). Only possible when we know the init PID; without it the - // netns is unaddressable, so there is nothing we can enter to clean. - // Best-effort; a no-op if nothing was installed. - if let Some(pid) = active.netns_pid { - IngressManager::force_cleanup(&name, pid, &mut buf_logger); - } - let _ = LxcContainer::new(&name, None).destroy(); + let plan = rollback_plan(active.rollback, !active.created.is_empty()); + execute_rollback(&plan, &mut |step| match step { + RollbackStep::StopContainer => { + let container = LxcContainer::new(&name, None); + // A signal can land after the firewall is installed and + // before the start, where there is nothing to kill. + // Reading that as a failed stop strands the chain, which + // then blocks every later start of this name. + let killed = container.kill().is_ok(); + stop_left_container_down(killed, container.is_running().ok()) + } + RollbackStep::RemoveFirewall => { + NetworkIptablesManager::force_cleanup( + &name, + active.veth.as_deref(), + active.created, + &mut buf_logger, + ); + true + } + RollbackStep::DestroyContainer => LxcContainer::new(&name, None).destroy().is_ok(), + // Without a marker there is nothing to match on, and matching + // too widely would kill processes this exec never started. + RollbackStep::ReapExec => { + if let Some(marker) = active.exec_marker.as_deref() { + let _ = LxcContainer::new(&name, None).reap_marked_processes(marker); + } + true + } + }); } std::process::exit(128 + sig as i32); } @@ -224,6 +453,295 @@ fn run_watchdog(mask: SigSet) -> ! { mod tests { use super::*; + /// Records the steps `execute_rollback` actually ran, failing whichever + /// steps `failing` names. + fn run_plan(plan: &[RollbackStep], failing: &[RollbackStep]) -> Vec { + let mut ran = Vec::new(); + execute_rollback(plan, &mut |step| { + ran.push(step); + !failing.contains(&step) + }); + ran + } + + #[test] + fn an_exec_rollback_touches_nothing_but_the_exec() { + // The container, its firewall, and its ingress rules are all meant to + // survive an exec. A rollback that stopped the container or removed + // its rules would turn a cancelled command into a destroyed sandbox. + assert_eq!( + rollback_plan(SignalRollback::ReapExec, true), + vec![RollbackStep::ReapExec], + "cancelling an exec must not stop the container or remove its rules" + ); + } + + #[test] + fn an_exec_rollback_still_reaps_when_this_process_owns_the_firewall() { + // Ownership of the firewall is a start-phase fact and says nothing + // about an exec. The plan must not vary with it. + assert_eq!( + rollback_plan(SignalRollback::ReapExec, false), + rollback_plan(SignalRollback::ReapExec, true), + "firewall ownership must not change what an exec rollback does" + ); + } + + #[test] + fn a_stop_that_failed_does_not_unfilter_the_container_anyway() { + // Ordering the stop first is not enough on its own: `lxc-stop` can + // fail, and then removing the firewall next leaves a running container + // with no egress policy -- the exact state the ordering exists to + // prevent, reached by a different route. The rollback must abandon the + // rest of the plan instead. + // + // Leaking the chain is the correct trade here, and the ordinary stop + // path already makes it deliberately: it propagates the stop error and + // leaves the rules in place rather than unfilter a still-running + // container (`state_aware.rs`, `stop`). + let plan = rollback_plan(SignalRollback::NetworkOnly, true); + assert_eq!( + run_plan(&plan, &[RollbackStep::StopContainer]), + vec![RollbackStep::StopContainer], + "a failed stop must not be followed by removing the firewall" + ); + } + + #[test] + fn a_kill_that_failed_on_a_container_already_down_still_clears_the_way() { + // A signal can land between installing the firewall and starting the + // container. `lxc-stop -k` exits non-zero with nothing to kill, and + // treating that as a failed stop strands the chain -- which then blocks + // every later start of this name with "its chain already exists". + assert!( + stop_left_container_down(false, Some(false)), + "a container that is already down must not strand its chain" + ); + } + + #[test] + fn a_kill_that_failed_on_a_running_container_retains_the_firewall() { + // This is the case the ordering exists to protect: the container is + // still transmitting, so its egress rules must stay. + assert!( + !stop_left_container_down(false, Some(true)), + "a still-running container must keep its filtering" + ); + } + + #[test] + fn a_container_state_that_could_not_be_read_retains_the_firewall() { + // An unreadable state is indistinguishable from a running one, and + // guessing wrong here unfilters a live container. + assert!( + !stop_left_container_down(false, None), + "an unreadable container state must be treated as still running" + ); + } + + #[test] + fn a_stop_that_succeeded_still_removes_the_firewall() { + // The negative control for the test above. Bailing out is only correct + // when the stop actually failed; a rollback that never removed the + // firewall would leak the chain on every signal, so "always bail" must + // not pass. + let plan = rollback_plan(SignalRollback::NetworkOnly, true); + assert_eq!( + run_plan(&plan, &[]), + vec![RollbackStep::StopContainer, RollbackStep::RemoveFirewall], + "a successful stop must still be followed by removing the firewall" + ); + } + + #[test] + fn a_destroy_that_succeeded_still_removes_the_firewall() { + // The negative control for the test below. The gate is specific to a + // step that actually failed; a rollback that never removed the firewall + // would leak the chain on every signal, so "always bail" must not pass. + let plan = rollback_plan(SignalRollback::DestroyContainer, true); + assert_eq!( + run_plan(&plan, &[]), + vec![RollbackStep::DestroyContainer, RollbackStep::RemoveFirewall], + "a destroy rollback must still remove the firewall it created" + ); + } + + #[test] + fn a_destroy_that_failed_does_not_unfilter_the_container_anyway() { + // A failed `lxc-destroy` may well have left the container running, so + // removing the firewall afterwards would strip egress filtering off a + // live container with no process left to notice -- the same fail-open + // the stop path already guards against. Ordering alone does not achieve + // this: destroy runs first precisely so that its failure is observable + // before anything unfilters the container. + let plan = rollback_plan(SignalRollback::DestroyContainer, true); + assert_eq!( + run_plan(&plan, &[RollbackStep::DestroyContainer]), + vec![RollbackStep::DestroyContainer], + "a failed destroy must not be followed by removing the firewall" + ); + } + + #[test] + fn a_state_aware_rollback_stops_the_container_before_unfiltering_it() { + // The start phase installs the chain before lxc-start so the container + // is never up without it. A signal can land after lxc-start has already + // succeeded, so a rollback that removed the firewall first would create + // exactly the state the ordering exists to prevent -- a running, + // unfiltered container with no process left to notice. Nothing later + // repairs it: the watchdog exits the process immediately afterward. + let plan = rollback_plan(SignalRollback::NetworkOnly, true); + assert_eq!( + plan, + vec![RollbackStep::StopContainer, RollbackStep::RemoveFirewall], + "a state-aware rollback must stop the container before removing its firewall" + ); + + // The container is provisioned to outlive this process, so the rollback + // stops it and never destroys it. + assert!( + !plan.contains(&RollbackStep::DestroyContainer), + "a provisioned container must survive a signal" + ); + + // The inbound rules live inside the container netns and are reachable + // only through its init PID, so the sole moment this plan could remove + // them is before the stop -- which is the unfiltered-and-still-running + // state the ordering above exists to prevent. Stopping discards them + // anyway, so there is nothing to gain for the exposure. No plan removes + // them for that reason, which is why there is no step left to name. + assert!( + !plan.contains(&RollbackStep::StopContainer) + || plan + .iter() + .all(|s| !matches!(s, RollbackStep::DestroyContainer)), + "a stop rollback must not also destroy the container" + ); + + // A process that created no chain must remove none: the name depends + // only on the container name, so the chain may by now belong to a + // different live container. The container this process was starting is + // still stopped, because that start is what is being rolled back. + assert_eq!( + rollback_plan(SignalRollback::NetworkOnly, false), + vec![RollbackStep::StopContainer], + "a rollback that owns no chain must not remove one" + ); + } + + #[test] + fn a_one_shot_rollback_destroys_the_container_before_unfiltering_it() { + // This container is going away entirely and destroy subsumes stopping + // it, which once argued for removing the chain first, while the name + // still unambiguously referred to this container. But `lxc-destroy` can + // fail, and that order would then have unfiltered a container that is + // still running. + // + // Inbound is not a step at all. Those rules live inside the container's + // own netns, so the destroy takes them with it: removing them first + // would change nothing on success and would strip the inbound deny off + // a still-running container on failure. + assert_eq!( + rollback_plan(SignalRollback::DestroyContainer, true), + vec![RollbackStep::DestroyContainer, RollbackStep::RemoveFirewall], + ); + + // Same ownership rule, and the destroy is unconditional: a one-shot + // container is this process's to reclaim whether or not a chain was + // ever created. + assert_eq!( + rollback_plan(SignalRollback::DestroyContainer, false), + vec![RollbackStep::DestroyContainer], + ); + } + + /// The registration slot is process-global, so these assertions cannot be + /// split across test functions without racing each other. + #[test] + fn registration_records_who_owns_the_container_not_just_its_name() { + // One-shot: this process made the container, so a signal takes it. + set_active("box"); + { + let slot = lock_slot(); + assert_eq!(slot.name.as_deref(), Some("box")); + assert_eq!(slot.rollback, SignalRollback::DestroyContainer); + } + + // State-aware: the container is provisioned and must survive a signal, + // so only the firewall this process installed is rolled back. + set_active_network_only("provisioned-box"); + { + let slot = lock_slot(); + assert_eq!(slot.name.as_deref(), Some("provisioned-box")); + assert_eq!(slot.rollback, SignalRollback::NetworkOnly); + } + + // A veth discovered later attaches to whichever registration is live. + set_active_veth("mxcv-abc"); + assert_eq!(lock_slot().veth.as_deref(), Some("mxcv-abc")); + + // Registering a name is not by itself a claim on the firewall chain. + // The watchdog gates its ownership-blind force_cleanup on this record, + // so a process whose `iptables -N` lost the race to a concurrent start + // must not be holding one — otherwise a signal would make the loser + // delete the winner's chain and leave the winner unfiltered. + assert!(lock_slot().created.is_empty()); + + // The manager publishes it the moment its own creating command succeeds. + let v4_only = CreatedResources::for_test(true, false, true, false); + set_active_created(v4_only); + assert_eq!(lock_slot().created, v4_only); + + // A successful teardown gives the claim back. Without this the record + // outlives the chain it describes, and since the chain name depends + // only on the container name, a signal arriving afterwards would run + // cleanup against a name that may by then answer for a different, live + // container. + set_active_network_only("torn-down-box"); + set_active_created(v4_only); + assert!(!lock_slot().created.is_empty()); + set_active_created(CreatedResources::default()); + assert!(lock_slot().created.is_empty()); + + // Giving the claim back is not the same as unregistering: the container + // is still the active one, so a signal must still roll back whatever + // else the registration covers. + assert_eq!(lock_slot().name.as_deref(), Some("torn-down-box")); + + // Re-registering a different container drops the claim with it. + set_active_network_only("yet-another-box"); + assert!(lock_slot().created.is_empty()); + set_active_created(v4_only); + assert!(!lock_slot().created.is_empty()); + set_active("one-shot-box"); + assert!(lock_slot().created.is_empty()); + + // Clearing leaves nothing to roll back. Without this, a signal after a + // successful start would strip the firewall off a running container. + clear_active(); + { + let slot = lock_slot(); + assert!(slot.name.is_none()); + assert!(slot.veth.is_none()); + assert!(slot.created.is_empty()); + } + + // Neither late registration may resurrect a cleared slot. + set_active_veth("mxcv-def"); + set_active_created(v4_only); + { + let slot = lock_slot(); + assert!(slot.veth.is_none()); + assert!(slot.created.is_empty()); + } + + // Re-registering resets the veth, since the new container has not had + // one discovered yet. + set_active_network_only("another-box"); + assert!(lock_slot().veth.is_none()); + clear_active(); + } + /// `ACTIVE_CONTAINER` is process-global and the test binary runs tests in /// parallel, so the whole publication contract is asserted in one test. /// Splitting it would let two tests race on the same slot. @@ -235,8 +753,7 @@ mod tests { // Bubblewrap builds the same firewall manager but never registers, so // its resources must not become something the watchdog would remove. set_active_created(CreatedResources::for_test(true, true, true, true)); - set_active_pid(4242); - let (name, veth, created, netns_pid) = active_snapshot(); + let (name, veth, created) = active_snapshot(); assert_eq!(name, None, "no container should be registered yet"); assert_eq!( created, @@ -244,15 +761,10 @@ mod tests { "ownership published with no registered container must be discarded" ); assert_eq!(veth, None); - assert_eq!( - netns_pid, None, - "a netns PID published with no registered container must be discarded" - ); // Registering a container opens the slot. set_active("ctr-a"); set_active_veth("veth-a"); - set_active_pid(1234); let v4_chain_only = CreatedResources::for_test(true, false, false, false); set_active_created(v4_chain_only); assert_eq!( @@ -261,7 +773,6 @@ mod tests { Some("ctr-a".to_owned()), Some("veth-a".to_owned()), v4_chain_only, - Some(1234) ), "a registered container must see its own identity and ownership" ); @@ -283,13 +794,8 @@ mod tests { set_active("ctr-b"); assert_eq!( active_snapshot(), - ( - Some("ctr-b".to_owned()), - None, - CreatedResources::default(), - None - ), - "registering a new container must reset veth, ownership, and netns PID" + (Some("ctr-b".to_owned()), None, CreatedResources::default()), + "registering a new container must reset both veth and ownership" ); clear_active(); diff --git a/src/backends/lxc/common/src/state_aware.rs b/src/backends/lxc/common/src/state_aware.rs new file mode 100644 index 000000000..6ebf8d91a --- /dev/null +++ b/src/backends/lxc/common/src/state_aware.rs @@ -0,0 +1,1878 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +//! State-aware lifecycle implementation for the LXC backend. +//! +//! LXC keeps the durable sandbox state in the named container. Provision creates +//! the container, start applies mount/network policy and starts it, exec reuses +//! the one-shot `lxc-attach` PTY path, stop stops the container, and +//! deprovision destroys it plus any remaining iptables state. + +use std::time::Duration; + +use serde::Serialize; + +use wxc_common::id::mint_random_token; +use wxc_common::logger::{Logger, Mode}; +use wxc_common::models::{ContainerPolicy, ExecutionRequest, LxcConfig}; +use wxc_common::mxc_error::MxcError; +use wxc_common::state_aware_backend::{ + null_pipe_handle, DeprovisionResult, ExecConsumer, ExecHandle, ExecOutcome, ProvisionResult, + StartResult, StatefulSandboxBackend, StopResult, +}; + +use crate::filesystem_mounts; +use crate::lxc_bindings::{mint_exec_marker, LxcContainer}; +use crate::network_ingress::IngressManager; +use crate::network_iptables::{CreatedResources, NetworkIptablesManager}; +use crate::signal_cleanup; + +/// Stateless state-aware LXC runner. +pub struct LxcStateAwareRunner; + +impl LxcStateAwareRunner { + pub fn new() -> Self { + Self + } +} + +impl Default for LxcStateAwareRunner { + fn default() -> Self { + Self::new() + } +} + +/// Provision-phase metadata for diagnostics and caller cleanup visibility. +#[derive(Debug, Clone, Serialize)] +pub struct LxcProvisionMetadata { + #[serde(rename = "containerName")] + pub container_name: String, + pub created: bool, +} + +/// Parses the `lxc:` sandbox_id form and returns the container +/// name segment. +fn extract_container_name(sandbox_id: &str) -> Result<&str, MxcError> { + let prefix = ::ID_PREFIX; + match sandbox_id.split_once(':') { + Some((p, rest)) if p == prefix && is_valid_container_name(rest) => Ok(rest), + _ => Err(MxcError::malformed_id(format!( + "expected {}:, got {:?}", + prefix, sandbox_id + ))), + } +} + +/// Maximum LXC sandbox container-name length. +/// +/// Bounds a sandbox container name to a sane length and character set so the +/// name is well-formed for LXC and for the derived iptables chain name. The +/// bound is input hygiene, and on the state-aware path it is also the only +/// thing narrowing the set of names that reach chain derivation. +/// +/// It does not make chain names unique. `NetworkIptablesManager` folds a +/// deterministic hash of the full container name into the chain name, which +/// breaks the systematic collapse of shared prefixes, but that derivation is +/// non-cryptographic and not injective — distinct names can still map to one +/// chain, and a caller that chooses `containerId` can construct such a pair. +/// See `NetworkIptablesManager::chain_name_for` for the work factor. A +/// collision lets one container's stop/deprovision tear down another +/// container's rules, leaving the incumbent running with no firewall. The +/// durable fix is persisted chain ownership verified before any flush or +/// delete, not a wider hash; tracked in AB#62953349. +const MAX_CONTAINER_NAME_LEN: usize = 20; + +/// Returns whether `name` is a valid LXC sandbox container name: non-empty, at +/// most [`MAX_CONTAINER_NAME_LEN`] characters, and restricted to ASCII +/// alphanumerics, `-`, and `_`. +/// +/// The character set keeps the name well-formed for LXC and for the derived +/// iptables chain name. `'.'` is intentionally excluded because it is stripped +/// by the chain-name sanitizer; the chain hash guards against collisions +/// regardless, but rejecting it up front keeps the sandbox id readable. +fn is_valid_container_name(name: &str) -> bool { + // Valid characters are ASCII (one byte each), so the byte length reported + // by `str::len` equals the character count for any otherwise-valid name. + !name.is_empty() + && name.len() <= MAX_CONTAINER_NAME_LEN + && name + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_')) +} + +/// Where a container name came from. +/// +/// Provision has to tell the two apart. A caller-supplied name that already +/// exists is adopted on purpose, so provisioning the same named sandbox twice +/// is idempotent. A name MXC minted and that already exists is a collision: +/// the token is 32 bits, so adopting it would silently hand the caller a +/// container somebody else is already using. +enum ContainerName { + Supplied(String), + Minted(String), +} + +fn resolve_container_name(request: &ExecutionRequest) -> Result { + if request.container_id.is_empty() { + return Ok(ContainerName::Minted(format!( + "mxc-{}", + mint_random_token() + ))); + } + if is_valid_container_name(&request.container_id) { + Ok(ContainerName::Supplied(request.container_id.clone())) + } else { + Err(MxcError::malformed_request(format!( + "containerId contains characters that are not valid for an LXC sandbox id: {:?}", + request.container_id + ))) + } +} + +/// Number of names `mint_unused_container_name` will try before giving up. +/// +/// The token is 32 bits, so a single collision is already unlikely and eight +/// consecutive ones are not something a healthy host produces. The bound +/// exists so a host that answers "taken" for every name -- a stuck probe, or a +/// `lxc-info` that reports every query as defined -- fails instead of looping. +const NAME_MINT_ATTEMPTS: usize = 8; + +/// Re-mint until the name is free, rather than adopting whatever is there. +/// +/// `claim` is injected so the decision can be tested without an LXC host. A +/// probe that cannot answer propagates: "unknown" must not be read as "free", +/// because that is how a collision becomes an adoption. +/// +/// The claim closes the collision window rather than merely narrowing it. An +/// earlier version probed for a free name and returned it, leaving the caller +/// to create it afterwards; two provisions that minted the same candidate could +/// both see it free, and the loser would then adopt the winner's container -- +/// handing two callers who each asked for a fresh sandbox the same one. Taking +/// the lifecycle lock inside the claim makes the probe and the create one +/// critical section, and a candidate found defined under the lock is re-minted +/// rather than adopted. Adoption remains correct for a name the caller supplied, +/// which is a request for *that* container; it is never correct for a name MXC +/// invented precisely because nobody else was using it. +fn mint_unused_container_name( + first: String, + mut claim: impl FnMut(&str) -> Result, MxcError>, +) -> Result<(String, T), MxcError> { + let mut candidate = first; + for _ in 0..NAME_MINT_ATTEMPTS { + if let Some(claimed) = claim(&candidate)? { + return Ok((candidate, claimed)); + } + candidate = format!("mxc-{}", mint_random_token()); + } + Err(MxcError::backend_error(format!( + "Could not mint an unused LXC container name in {NAME_MINT_ATTEMPTS} attempts; \ + the last candidate {candidate:?} was already defined" + ))) +} + +fn validate_lxc_config(config: Option<&LxcConfig>) -> Result<(), MxcError> { + let Some(config) = config else { + return Err(MxcError::malformed_request( + "experimental.lxc.provision with distribution and release is required", + )); + }; + if config.distribution.is_empty() || config.release.is_empty() { + return Err(MxcError::malformed_request( + "LXC distribution and release are required", + )); + } + Ok(()) +} + +/// Map a container state-probe failure onto a backend error. +/// +/// Every phase asks whether the container exists or is running before deciding +/// to create, start, stop, or unfilter it. An unreadable answer is a backend +/// failure, never a licence to assume whichever value is convenient -- assuming +/// "gone" or "stopped" is what turns a broken probe into an unfiltered running +/// container. +fn probe_failed(question: &str, container_name: &str, detail: String) -> MxcError { + MxcError::backend_error(format!( + "Failed to determine whether LXC container {container_name:?} {question}: {detail}" + )) +} + +fn has_filesystem_policy(policy: &ContainerPolicy) -> bool { + !policy.readwrite_paths.is_empty() + || !policy.readonly_paths.is_empty() + || !policy.denied_paths.is_empty() +} + +fn has_network_policy(policy: &ContainerPolicy) -> bool { + // `network_specified` is the outermost bit: it is true for any `network` + // block at all, including an empty one and one that only sets + // `allowLocalNetwork: false`. The narrower bits below cannot see those -- + // both produce a policy indistinguishable from the struct default -- so a + // phase that documents "no network section" would otherwise accept one. + policy.network_specified + || !policy.allowed_hosts.is_empty() + || !policy.blocked_hosts.is_empty() + || policy.allow_local_network + || policy.network_proxy.is_enabled() +} + +fn reject_start_policy_on_other_phase( + phase: &str, + policy: &ContainerPolicy, +) -> Result<(), MxcError> { + if has_filesystem_policy(policy) || has_network_policy(policy) { + return Err(MxcError::policy_validation(format!( + "LXC state-aware {phase} does not accept filesystem or network policy; pass it to start" + ))); + } + Ok(()) +} + +fn normalized_policy( + request: &ExecutionRequest, + logger: &mut Logger, +) -> Result { + let policy = + match wxc_common::filesystem_object::normalize_object_conflicts(&request.policy, logger) { + Ok(Some(policy)) => policy, + Ok(None) => request.policy.clone(), + Err(msg) => return Err(MxcError::policy_validation(msg)), + }; + + wxc_common::filesystem_access::check_delegation(&policy) + .map_err(MxcError::policy_validation)?; + Ok(policy) +} + +fn apply_filesystem_policy( + container: &LxcContainer, + request: &ExecutionRequest, + logger: &mut Logger, +) -> Result<(), MxcError> { + let policy = normalized_policy(request, logger)?; + filesystem_mounts::configure_filesystem_mounts(container, &policy, logger) + .map_err(|e| MxcError::policy_validation(format!("Failed to configure filesystem: {e}"))) +} + +/// The network rejections that depend on the requested policy alone. +/// +/// Split out of `apply_network_policy` so `validate_start` can reach the same +/// verdict without a container. A dry run stops after `validate_start` +/// (`state_aware_dispatch.rs`, `Phase::Start`), so anything checked only inside +/// the apply path is invisible to it -- and a dry run that answers "this start +/// is fine" for a policy the real start refuses is worse than no dry run, since +/// the caller has asked precisely that question and been told the wrong answer. +fn reject_unenforceable_network_policy(policy: &ContainerPolicy) -> Result<(), MxcError> { + if policy.network_proxy.is_enabled() { + return Err(MxcError::policy_validation( + "LXC state-aware start does not support network.proxy", + )); + } + + // A dry run stops before the ingress apply, so without this it would report + // success for a start that `IngressManager` refuses. + if policy.allow_local_network { + return Err(MxcError::policy_validation( + "LXC state-aware start does not support network.allowLocalNetwork: the container's \ + inbound chain can only open a source range, and opening every source is broader \ + than the local-network access requested. See microsoft/mxc AB#63505947.", + )); + } + + Ok(()) +} + +/// Every start rejection that needs only the request, in the order the real +/// start reaches them: filesystem normalization first (`apply_filesystem_policy` +/// runs before `apply_network_policy`), then the network verdicts. Keeping the +/// order means a dry run reports the same error the real start would, not merely +/// some error. +fn validate_start_policy(request: &ExecutionRequest) -> Result<(), MxcError> { + let mut logger = Logger::new(Mode::Buffer); + let policy = normalized_policy(request, &mut logger)?; + reject_unenforceable_network_policy(&policy) +} + +/// Apply the network policy, returning the record of what it installed. +/// +/// The record is what lets the caller undo exactly this attempt if the start +/// then fails, without touching a chain that a concurrent start owns. An +/// enforcement-free policy installs nothing and returns an empty record. +fn apply_network_policy( + container: &LxcContainer, + request: &ExecutionRequest, + logger: &mut Logger, +) -> Result { + reject_unenforceable_network_policy(&request.policy)?; + + let policy = normalized_policy(request, logger)?; + + let mut fw_manager = NetworkIptablesManager::new(container.name()); + + // When the policy requires the egress firewall, MXC installs a + // `lxc.hook.start-host` hook that renames the container's host-side veth to + // a deterministic name. The hook runs after liblxc creates the veth pair and + // attaches it to the bridge, but before the container's init runs, so the + // chain and its FORWARD hook can be built against that name *before* anything + // in the container can transmit. The prior flow discovered the veth only + // after start and applied rules afterward, leaving a window in which a + // container with a deny policy had unrestricted network. iptables accepts an + // interface name that does not exist yet, so hooking the not-yet-created + // veth by its deterministic name closes that window entirely. The hook key + // is container-global, so enforcement no longer depends on which + // `lxc.net.` index the interface uses. + // + // This gate is the same predicate `apply_firewall_rules` uses to decide + // whether to install rules, so the pin hook and the multi-interface guard + // run exactly when a chain will be scoped to the veth, and never when the + // apply is a no-op. + if policy.requires_firewall() { + // Exactly one interface gets a pinned veth, and apply_firewall_rules + // hooks exactly one interface. A container this run created has exactly + // that one interface, but provision also adopts containers it did not + // create, and an adopted one can carry several. Those would keep + // routing while start reported that a deny policy had been applied — + // the policy would be a claim rather than a control. Refuse instead: + // failing closed on a config MXC cannot fully enforce is the only + // honest answer, and hooking every interface is the follow-up. + let net = container + .configured_net_interfaces() + .map_err(|e| MxcError::backend_error(format!("Failed to read network config: {e}")))?; + + if net.count > 1 { + return Err(MxcError::policy_validation(format!( + "Container {:?} has {} configured network interfaces; \ + a firewall-enforced network policy can only be applied to a container \ + with a single interface, because traffic on the others would bypass it", + container.name(), + net.count, + ))); + } + // Zero is refused for the mirror-image reason. There is no interface to + // enforce against, so the pin hook would find nothing to rename and the + // run would either fail later or install a chain nothing routes + // through. Either way the caller asked for enforced networking on a + // container that has no network to enforce. + if net.count == 0 { + return Err(MxcError::policy_validation(format!( + "Container {:?} has no configured network interface, \ + so a firewall-enforced network policy has nothing to attach to", + container.name() + ))); + } + + // Exactly one interface is necessary but not sufficient -- it also has + // to be a veth. Enforcement works by naming the host end of a veth pair + // and hooking that name in FORWARD, so a macvlan or phys interface would + // get a chain built against a name that never exists while its traffic + // ran unfiltered and MXC reported the policy as enforced. + let sole_kind = net.sole_kind.as_deref().unwrap_or_default(); + if sole_kind != "veth" { + return Err(MxcError::policy_validation(format!( + "Container {:?} configures a network interface of type {:?}; a \ + firewall-enforced network policy can only be applied to a veth \ + interface, because enforcement pins the host end of the veth pair and \ + hooks that name in FORWARD. Use a veth interface, or run without \ + firewall enforcement", + container.name(), + sole_kind, + ))); + } + + let veth = NetworkIptablesManager::deterministic_veth_name(container.name()); + container.ensure_veth_pin_hook(&veth).map_err(|e| { + MxcError::backend_error(format!("Failed to install veth pin hook: {e}")) + })?; + fw_manager.set_veth_interface(&veth); + } + + match fw_manager.apply_firewall_rules(&policy, logger) { + Ok(true) => { + if fw_manager.rules_applied() { + // Rules must survive after the start phase returns. stop and + // deprovision call force_cleanup_authoritative to remove this + // persistent state. + // + // Read the ownership record out before forgetting the manager. + // It is the only surviving evidence of what this attempt + // installed, and the start path needs it to tear down exactly + // that much if the container then fails to start. + let created = fw_manager.created(); + std::mem::forget(fw_manager); + return Ok(created); + } + Ok(CreatedResources::default()) + } + Ok(false) => Err(MxcError::policy_validation( + "Failed to apply network firewall rules", + )), + Err(e) => { + // Fail closed: tear down any partially-applied chain so an aborted + // policy application does not leak iptables state, and let the error + // propagate so the caller does not start the container unfiltered. + // + // Only when this run created the chain, though. The chain name is + // derived from the container name, so a concurrent start of the same + // sandbox aims at the same chain and the loser's `iptables -N` fails + // against the winner's. Cleaning up unconditionally would have the + // loser delete the winner's chain and hooks, and the winner would + // then start its container with nothing filtering it — turning a + // recoverable collision into a fail-open. + if fw_manager.owns_resources() { + // Tear down through the manager that created the chain, not a + // fresh one. `Drop` re-runs the teardown whenever + // `chain_created` is still set, and a fresh manager cannot clear + // that flag on the original -- so cleaning up any other way + // leaves this function returning into a second teardown of a + // chain it has already deleted. Between the two, another start + // can create the same deterministic chain name and install its + // rules, and the trailing teardown would then strip the new + // owner's hooks and delete its chain, leaving *its* container + // running unfiltered. + // + // Going through `fw_manager` clears `chain_created` on a + // successful `-X`, so the drop is a no-op; when `-X` fails the + // flag stays set and the drop is the retry it is there to be. + // It also knows the pinned veth, which a fresh manager does not. + let _ = fw_manager.remove_firewall_rules(logger); + return Err(MxcError::policy_validation(format!( + "Network policy error: {e}" + ))); + } + // Nothing was created, so nothing is removed. Say so explicitly: + // the chain existing without this run creating it means either a + // concurrent start holds it or a previous run left it behind, and + // the two are indistinguishable from iptables alone until ownership + // is persisted (AB#62953349). Both are cleared the same way, so name + // the remedy rather than leaving the caller to guess. + Err(MxcError::policy_validation(format!( + "Network policy error: could not create the firewall chain for container {:?}, \ + so no rules were applied and none were removed. Its chain already exists, \ + which means another start of this sandbox is in progress or an earlier run \ + left it behind; stop or deprovision the sandbox to clear it, then start again. \ + Underlying error: {e}", + container.name() + ))) + } + } +} + +/// Best-effort teardown of iptables state this process installed. +/// +/// `veth` is the host-side veth interface name when it is known. Teardown no +/// longer needs it to find the FORWARD hooks — those are located by +/// enumerating the live FORWARD chain and matching the `-j ` target — +/// but it is still passed through for logging and for the callers that +/// discovered it while the container was running. +/// +/// `created` is the ownership record: only the chains and hooks named in it are +/// removed, so a process that created nothing removes nothing. Use this from +/// the start path, which knows what it installed. +/// Install the container's inbound default-deny chain, failing closed. +/// +/// The egress chain is installed *before* the container runs, because iptables +/// accepts a veth name that does not exist yet. Ingress cannot work that way: +/// the chain lives inside the container's **own** network namespace, which does +/// not exist until the container starts, so this necessarily runs afterwards. +/// The one-shot path orders it the same way for the same reason. +/// +/// Without this the state-aware path enforced only half the network policy. +/// `allowLocalNetwork` defaults to false, so a container started here accepted +/// inbound connections from the host and the LAN while MXC reported the policy +/// as enforced -- egress filtered, ingress wide open. `IngressManager` also +/// refuses `allowLocalNetwork: true` outright rather than installing an +/// over-broad accept, so wiring it in is what makes that value honored or +/// refused instead of silently ignored. +fn apply_ingress_policy( + container: &LxcContainer, + container_name: &str, + request: &ExecutionRequest, + logger: &mut Logger, +) -> Result<(), MxcError> { + let policy = normalized_policy(request, logger)?; + + let Some(pid) = container.init_pid() else { + // Ingress installs unconditionally, so a missing netns is always fatal. + // Enforcing inbound means entering the container's netns, and the init + // PID is the only handle on it. Continuing would silently drop the + // inbound deny, so refuse the start instead. + return Err(MxcError::backend_error( + "Failed to discover the container init PID; cannot enter the container \ + network namespace to enforce the inbound network policy", + )); + }; + + let mut manager = IngressManager::new(container_name, pid); + let applied = manager + .apply_firewall_rules(&policy, logger) + .map_err(|e| MxcError::backend_error(format!("Inbound network policy error: {e}")))?; + if !applied { + return Err(MxcError::backend_error( + "Failed to apply inbound network firewall rules", + )); + } + + // The container outlives this call, so its rules must too. `Drop` otherwise + // tears them down on the way out of this function, undoing the install this + // function exists to perform. + manager.set_preserve_policy(true); + Ok(()) +} + +fn cleanup_network_owned( + container_name: &str, + veth: Option<&str>, + created: CreatedResources, + logger: &mut Logger, +) { + NetworkIptablesManager::force_cleanup(container_name, veth, created, logger); +} + +/// Teardown of whatever iptables state exists for a container, whichever +/// process installed it. +/// +/// For `stop` and `deprovision` only. They run in a different process from the +/// `start` that created the chain, so they hold no ownership record and an +/// ownership-gated teardown would silently do nothing — stranding the chain and +/// blocking every later start, which fails on a chain it did not create. Both +/// callers have already stopped or destroyed the container by this point, so +/// nothing is left for the chain to protect. See +/// `NetworkIptablesManager::force_cleanup_authoritative` for the full argument. +/// +/// A failure is reported rather than swallowed. Discarding it made stop and +/// deprovision answer success over a chain that survived them, which is the +/// one outcome the caller has to know about: the stranded chain blocks every +/// later start of that container name. +/// The host-side veth name MXC guarantees for a container it enforces. +/// +/// Teardown deletes FORWARD rules by their full specification, so it needs the +/// name those rules actually carry. Asking liblxc for it is wrong: the pin hook +/// renames the interface *after* liblxc creates it, so `lxc-info` keeps +/// reporting the random name it generated, and deletes replayed against that +/// name match nothing — the hooks survive and the chain is stranded. The name +/// is derived rather than discovered because enforcement is what put it there. +fn enforced_veth_name(container_name: &str) -> Option { + Some(NetworkIptablesManager::deterministic_veth_name( + container_name, + )) +} + +fn cleanup_network_authoritative( + container_name: &str, + veth: Option<&str>, + logger: &mut Logger, +) -> Result<(), MxcError> { + NetworkIptablesManager::force_cleanup_authoritative(container_name, veth, logger).map_err(|e| { + MxcError::backend_error(format!( + "Failed to remove the network filtering state for LXC container {container_name:?}; \ + it is still installed and will block a later start: {e}" + )) + }) +} + +/// Advisory lock that makes each state transition one critical section per +/// sandbox. +/// +/// `start` reads whether the container is running and then, if it is not, +/// writes filesystem policy, installs the firewall, and starts it. Nothing +/// held the container still in between, so two concurrent starts both read +/// "not running" and both take the else arm. Both then write policy: the +/// container that comes up can be running behind the other start's mounts, and +/// the `already_started` refusal — whose whole job is to stop start policy +/// being reapplied to a live container — never fires. The loser gets `Ok`, +/// not the error the guard exists to produce. +/// +/// `stop` and `deprovision` have to take the same lock, because excluding only +/// a second start leaves the worse race open: a concurrent stop reads "not +/// running" in the window after a start installs its FORWARD chain and before +/// `container.start()`, runs the authoritative network teardown, and removes +/// the chain. The start then brings the container up with no egress filter at +/// all. Teardown is only safely ordered against a running container if no +/// start can be part-way through one. +/// +/// `exec` deliberately stays out. It installs and removes no host state, so it +/// cannot produce that fail-open, and an exclusive lock there would serialize +/// concurrent execs into one sandbox, which is a supported thing to do. Every +/// other phase takes it: `provision` because minting probes for an unused name +/// and then creates it, which is the same shape of race. +/// +/// The lock file lives in the LXC root rather than in the container directory +/// because `deprovision` destroys that directory: a lock file inside it would +/// be unlinked while still held, and the next phase would create a fresh file +/// and take a *different* lock, which is no lock at all. Container names are +/// validated for length and character set before they reach this, so the name +/// cannot escape the root. `flock` is released by the kernel when the +/// descriptor closes, so a phase that dies mid-sequence frees it; an `O_EXCL` +/// lock file would instead wedge every later phase of that container name. +struct LifecycleLock { + /// Held for its `Drop`, which is what releases the lock. `None` when the + /// LXC root does not exist; see `acquire`. + #[cfg(target_os = "linux")] + _guard: Option>, +} + +/// How many times `acquire` will re-take a lock whose file was replaced under +/// it before giving up. +/// +/// Each retry costs one open and one `flock`, and the only thing that triggers +/// one is a `deprovision` reclaiming the file, so the bound exists to stop an +/// adversary spinning the loop rather than to absorb ordinary contention. +/// Exhausting it refuses the phase, which is the safe direction. +#[cfg(target_os = "linux")] +const LOCK_ACQUIRE_ATTEMPTS: usize = 8; + +impl LifecycleLock { + /// Where the lock file for `container_name` lives. + /// + /// One function so `acquire` and `release_and_reclaim` cannot drift: a + /// reclaim that computed a different path would unlink a file nobody holds + /// and leave the real one behind. + #[cfg(target_os = "linux")] + fn lock_path(container: &LxcContainer, container_name: &str) -> String { + format!( + "{}/.mxc-lifecycle-{}.lock", + container.lxc_path(), + container_name + ) + } + + /// Take the lock, waiting for any concurrent transition of the same sandbox. + /// + /// Off Linux there is no LXC to serialize, so this is a no-op that exists + /// only so the phases read the same on every target. + fn acquire(container: &LxcContainer, container_name: &str) -> Result { + #[cfg(target_os = "linux")] + { + let path = Self::lock_path(container, container_name); + for _ in 0..LOCK_ACQUIRE_ATTEMPTS { + let file = match std::fs::OpenOptions::new() + .create(true) + .truncate(false) + .write(true) + .open(&path) + { + Ok(file) => file, + // No LXC root means no container directory under it, so + // nothing has passed the `is_defined` gate that every start + // clears before it touches host state — there is no + // transition to be serialized against. `stop` and + // `deprovision` are required to be idempotent, and refusing + // them here would make cleanup on a host that never had LXC + // an error. Any other failure, permission in particular, + // still refuses the phase. + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + return Ok(LifecycleLock { _guard: None }) + } + Err(e) => { + return Err(MxcError::backend_error(format!( + "Failed to open the lifecycle lock for LXC container \ + {container_name:?} at {path}: {e}" + ))) + } + }; + let guard = nix::fcntl::Flock::lock(file, nix::fcntl::FlockArg::LockExclusive) + .map_err(|(_, errno)| { + MxcError::backend_error(format!( + "Failed to take the lifecycle lock for LXC container \ + {container_name:?}: {errno}" + )) + })?; + if Self::path_still_names(&guard, &path) { + return Ok(LifecycleLock { + _guard: Some(guard), + }); + } + // A `deprovision` reclaimed the file between the open and the + // lock, so this lock is on an inode nobody else can reach -- + // which excludes nobody. Drop it and take the current one. + drop(guard); + } + Err(MxcError::backend_error(format!( + "Failed to take the lifecycle lock for LXC container {container_name:?}: the lock \ + file at {path} was replaced on every one of {LOCK_ACQUIRE_ATTEMPTS} attempts" + ))) + } + #[cfg(not(target_os = "linux"))] + { + let _ = (container, container_name); + Ok(LifecycleLock {}) + } + } + + /// Whether `path` still names the inode `guard` holds a lock on. + /// + /// `flock` follows the open descriptor, not the name, so a lock taken on a + /// file that has since been unlinked excludes nothing: the next phase + /// creates a fresh file at the same path and locks that instead. The + /// inode is what makes the two distinguishable. + /// + /// Any error reading either side answers `false`, which costs a retry and + /// eventually refuses the phase. Answering `true` on an unreadable path + /// would hand out a lock that excludes nothing. + #[cfg(target_os = "linux")] + fn path_still_names(guard: &nix::fcntl::Flock, path: &str) -> bool { + use std::os::unix::fs::MetadataExt; + + let (Ok(locked), Ok(named)) = (guard.metadata(), std::fs::metadata(path)) else { + return false; + }; + locked.ino() == named.ino() && locked.dev() == named.dev() + } + + /// Release the lock and remove its file. + /// + /// Only the terminal phase may call this, and only once the container is + /// gone: the file is a permanent inode per sandbox name otherwise, and a + /// host that provisions many short-lived sandboxes accumulates one for + /// every name it ever used. + /// + /// The unlink happens while the lock is still held, so no other phase can + /// be between its own open and its own `flock` and reach a state this did + /// not create. A phase that is already blocked on the lock wakes holding + /// the now-unlinked inode, sees the path no longer names it, and retakes + /// the current one; see `path_still_names`. + /// + /// A failed unlink is not reported. The container is already destroyed by + /// this point, so an error here would fail a `deprovision` that did + /// everything it was asked to, and the only consequence is the zero-byte + /// file this was trying to reclaim. + fn release_and_reclaim(self, container: &LxcContainer, container_name: &str) { + #[cfg(target_os = "linux")] + { + if self._guard.is_some() { + let _ = std::fs::remove_file(Self::lock_path(container, container_name)); + } + } + #[cfg(not(target_os = "linux"))] + { + let _ = (container, container_name); + } + // `self` is consumed, so the lock releases here -- after the unlink, + // which is the order that matters. + } +} + +impl StatefulSandboxBackend for LxcStateAwareRunner { + const ID_PREFIX: &'static str = "lxc"; + const BACKEND_KEY: &'static str = "lxc"; + + type ProvisionConfig = LxcConfig; + type StartConfig = (); + type ExecConfig = (); + type StopConfig = (); + type DeprovisionConfig = (); + type ProvisionMetadata = LxcProvisionMetadata; + type StartMetadata = (); + type StopMetadata = (); + type DeprovisionMetadata = (); + + fn provision( + &mut self, + request: &ExecutionRequest, + config: Option, + ) -> Result, MxcError> { + validate_lxc_config(config.as_ref())?; + reject_start_policy_on_other_phase("provision", &request.policy)?; + + let config = config.expect("validated above"); + // The lock has to be held across the "does this name exist" probe and + // whatever that answer leads to, or the answer is stale before it is + // used: two provisions both see the name free and both create. + // + // What a name found defined under the lock *means* differs by origin. A + // supplied name is a request for that specific container, so finding it + // already there is the adoption the caller asked for. A minted name was + // invented because nobody was using it, so finding it defined means the + // mint lost a race -- adopting there would hand two callers who each + // asked for a fresh sandbox the same container, and the loser would + // report `created: false` and never reclaim it. + let (container_name, container, _lifecycle_lock, created) = + match resolve_container_name(request)? { + ContainerName::Supplied(name) => { + let container = LxcContainer::new(&name, None); + let lock = LifecycleLock::acquire(&container, &name)?; + let created = !container + .is_defined() + .map_err(|e| probe_failed("exists", &name, e))?; + (name, container, lock, created) + } + ContainerName::Minted(first) => { + let (name, (container, lock)) = + mint_unused_container_name(first, |candidate| { + let container = LxcContainer::new(candidate, None); + let lock = LifecycleLock::acquire(&container, candidate)?; + if container + .is_defined() + .map_err(|e| probe_failed("exists", candidate, e))? + { + // Releases as it falls out of scope, so the + // next candidate is not taken while holding a + // lock on this one. + return Ok(None); + } + Ok(Some((container, lock))) + })?; + (name, container, lock, true) + } + }; + if created { + container + .create(&config.distribution, &config.release) + .map_err(|e| MxcError::backend_error(format!("Failed to create container: {e}")))?; + } + + Ok(ProvisionResult { + sandbox_id: format!("{}:{}", Self::ID_PREFIX, container_name), + metadata: Some(LxcProvisionMetadata { + container_name, + created, + }), + }) + } + + fn start( + &mut self, + sandbox_id: &str, + request: &ExecutionRequest, + _config: Option<()>, + ) -> Result, MxcError> { + let container_name = extract_container_name(sandbox_id)?; + let container = LxcContainer::new(container_name, None); + // Hold the sandbox still for the whole phase: the running/not-running + // decision below and the policy writes that depend on it have to be one + // critical section, or a second start slips past the `already_started` + // refusal and applies its policy to a container the first one is + // bringing up — and a concurrent stop tears this start's firewall back + // off between installing it and running the container. + let _lifecycle_lock = LifecycleLock::acquire(&container, container_name)?; + if !container + .is_defined() + .map_err(|e| probe_failed("exists", container_name, e))? + { + return Err(MxcError::not_provisioned(format!( + "LXC container {:?} is not provisioned", + container_name + ))); + } + let mut logger = Logger::new(Mode::Buffer); + if container + .is_running() + .map_err(|e| probe_failed("is running", container_name, e))? + { + // A running container cannot receive this start's chain, and an + // absent network section now owes one, so refusing is the only + // answer that does not report enforcement that never happened. + if has_filesystem_policy(&request.policy) + || has_network_policy(&request.policy) + || request.policy.requires_firewall() + { + return Err(MxcError::already_started( + "LXC container is already running; start policy cannot be reapplied", + )); + } + } else { + apply_filesystem_policy(&container, request, &mut logger)?; + // Install the firewall *before* the container is allowed to run. + // Applying it after start left a roughly 10-second window in which a + // container with a deny policy had unrestricted network. A + // firewall-install failure aborts the start (fail closed) rather + // than proceeding unfiltered. + // + // Register a signal rollback across that same window first. The + // chain is host state that outlives this process, so a SIGTERM + // between installing it and finishing the start would strand it with + // nobody to remove it. `set_active_network_only` is deliberately not + // the one-shot `set_active`: this container is provisioned and must + // survive, so only the firewall is rolled back. + signal_cleanup::set_active_network_only(container_name); + let installed = match apply_network_policy(&container, request, &mut logger) { + Ok(created) => created, + Err(e) => { + // Clear before returning. apply_network_policy already removed + // whatever it created, and when it failed because another start + // owns the chain it deliberately removed nothing — a signal + // arriving now would otherwise delete that owner's chain and + // leave its container running unfiltered. + signal_cleanup::clear_active(); + return Err(e); + } + }; + if let Err(e) = container.start() { + // A failed `lxc-start` is not evidence that the container is + // down. `LIVE_STATES` counts STARTING and ABORTING as live for + // exactly this reason: a start can fail with the container + // already up or still coming up. Removing the egress chain on + // the assumption that it never started would leave a running + // container unfiltered, so establish that it is stopped first. + // + // Only a definitive "not running" is evidence that unfiltering + // is safe -- an unreadable probe is not, so it is treated as + // live. + if container.is_running().unwrap_or(true) { + // Name the veth the rules were scoped to, then kill the + // container. Kill, not stop, for the reason the failed-ingress + // rollback below gives: a graceful stop waits up to 60 s, + // and every second of it is a container whose start already + // failed sitting there with a half-applied policy. + let veth = enforced_veth_name(container_name); + if let Err(stop_err) = container.kill() { + // The egress chain is the only part of the policy still + // in force, so removing it now would turn a filtered + // container into an unfiltered one. It stays -- the same + // trade the ingress rollback and `stop` both make. + signal_cleanup::clear_active(); + return Err(MxcError::backend_error(format!( + "Failed to start container: {e}; it could not be stopped afterwards \ + ({stop_err}), so it may still be running and its egress rules were \ + left in place" + ))); + } + cleanup_network_owned(container_name, veth.as_deref(), installed, &mut logger); + } else { + // Confirmed stopped, so there is no veth to discover and the + // FORWARD hooks are found by enumerating on the chain name. + // + // Ownership-scoped, not authoritative: `apply_network_policy` + // may have installed nothing, and a chain present without + // this attempt creating it belongs to a concurrent start + // whose container is running behind it. + cleanup_network_owned(container_name, None, installed, &mut logger); + } + signal_cleanup::clear_active(); + return Err(MxcError::backend_error(format!( + "Failed to start container: {e}" + ))); + } + // Inbound enforcement lands only once the container's network + // namespace exists, so unlike the egress chain it comes after start. + if let Err(e) = apply_ingress_policy(&container, container_name, request, &mut logger) { + // The container is up and its inbound deny is not in force, so + // leaving it running is exactly the fail-open this guard exists + // to prevent. Name the veth the rules were scoped to, then kill + // the container -- which also discards the netns holding any + // partial ingress chain -- and remove the egress state this + // start installed. + // + // Kill, not stop: a graceful `lxc-stop` waits up to 60 s and can + // fail outright under systemd-in-userns, and every second of + // that wait is a running container with no inbound filtering. + let veth = enforced_veth_name(container_name); + let stopped = container.kill(); + signal_cleanup::clear_active(); + if let Err(stop_err) = stopped { + // The container is still up, and the egress chain is the + // only half of its policy still in force. Removing it now + // would turn a half-filtered container into an unfiltered + // one, so it stays -- the same trade `stop` makes when it + // cannot stop the container, and the same one the signal + // rollback makes when its stop fails. + return Err(MxcError::backend_error(format!( + "{e}; the container could not be stopped afterwards ({stop_err}), so it \ + is still running and its egress rules were left in place" + ))); + } + cleanup_network_owned(container_name, veth.as_deref(), installed, &mut logger); + return Err(e); + } + // Past this point the chain and the container are both meant to + // persist, so a signal must not roll either back. + signal_cleanup::clear_active(); + } + Ok(StartResult { metadata: None }) + } + + fn exec( + &mut self, + sandbox_id: &str, + request: &ExecutionRequest, + _config: Option<()>, + consumer: ExecConsumer, + ) -> Result { + // Before any work: `attach_run` relays to this process's stdio and + // returns no pipes, so it cannot serve an in-process caller. Refusing + // after running it would make the refusal describe output that has + // already gone somewhere the caller never asked for. + if consumer == ExecConsumer::Library { + return Err(wxc_common::state_aware_backend::unsupported_library_exec( + "LXC", + )); + } + + let container_name = extract_container_name(sandbox_id)?; + reject_start_policy_on_other_phase("exec", &request.policy)?; + + let container = LxcContainer::new(container_name, None); + if !container + .is_defined() + .map_err(|e| probe_failed("exists", container_name, e))? + { + return Err(MxcError::not_provisioned(format!( + "LXC container {:?} is not provisioned", + container_name + ))); + } + if !container + .is_running() + .map_err(|e| probe_failed("is running", container_name, e))? + { + return Err(MxcError::not_started(format!( + "LXC container {:?} is not started", + container_name + ))); + } + + let timeout = if request.script_timeout == 0 { + None + } else { + Some(Duration::from_millis(u64::from(request.script_timeout))) + }; + + // Registered for the whole attach, not just the timed part: a signal + // kills this process without waiting for the timeout, and the container + // is persistent, so anything the script started would otherwise be + // inherited by the next exec. + let marker = mint_exec_marker(); + signal_cleanup::set_active_exec(container_name, &marker); + // Cleared unconditionally: an empty env would otherwise leave + // `lxc-attach` in keep-env mode, inheriting this process's environment + // and the credentials in it. + let outcome = container.attach_run( + &request.script_code, + &request.working_directory, + &request.env, + true, + timeout, + Some(&marker), + ); + signal_cleanup::clear_active(); + + let exit_code = outcome + .map(|(exit_code, _, _)| exit_code) + .map_err(|e| MxcError::backend_error(format!("Execution failed: {e}")))?; + + Ok(ExecHandle { + stdout: null_pipe_handle(), + stderr: null_pipe_handle(), + stdin: null_pipe_handle(), + waiter: Box::new(move || Ok(ExecOutcome::Exited(exit_code))), + terminator: Box::new(|| Ok(())), + }) + } + + fn stop( + &mut self, + sandbox_id: &str, + request: &ExecutionRequest, + _config: Option<()>, + ) -> Result, MxcError> { + let container_name = extract_container_name(sandbox_id)?; + reject_start_policy_on_other_phase("stop", &request.policy)?; + + let container = LxcContainer::new(container_name, None); + // Same lock the start takes. Without it this teardown can observe a + // start's container as not running, strip the FORWARD chain that start + // just installed, and return before it calls `container.start()`, + // leaving a running container with no egress filtering. + let _lifecycle_lock = LifecycleLock::acquire(&container, container_name)?; + if !container + .is_defined() + .map_err(|e| probe_failed("exists", container_name, e))? + { + return Err(MxcError::not_provisioned(format!( + "LXC container {:?} is not provisioned", + container_name + ))); + } + + let mut logger = Logger::new(Mode::Buffer); + // Name the veth the rules were scoped to. The interface disappears once + // the container stops, but iptables can still delete a FORWARD rule that + // names it. Stop the container *before* tearing down its firewall rules + // so no + // process runs without egress filtering during the shutdown drain. If + // the stop fails, propagate the error and leave the rules in place + // rather than exposing a still-running container. + let veth = enforced_veth_name(container_name); + if container + .is_running() + .map_err(|e| probe_failed("is running", container_name, e))? + { + container + .stop() + .map_err(|e| MxcError::backend_error(format!("Failed to stop container: {e}")))?; + } + cleanup_network_authoritative(container_name, veth.as_deref(), &mut logger)?; + Ok(StopResult { metadata: None }) + } + + fn deprovision( + &mut self, + sandbox_id: &str, + request: &ExecutionRequest, + _config: Option<()>, + ) -> Result, MxcError> { + let container_name = extract_container_name(sandbox_id)?; + reject_start_policy_on_other_phase("deprovision", &request.policy)?; + + let mut logger = Logger::new(Mode::Buffer); + let container = LxcContainer::new(container_name, None); + // Same lock the start takes, for the same reason as `stop`: this path + // also runs the authoritative network teardown, and must not do it + // while a start is between installing the firewall and running the + // container. + let _lifecycle_lock = LifecycleLock::acquire(&container, container_name)?; + let veth = enforced_veth_name(container_name); + // A probe that could not answer must not be read as "already gone". + // Skipping the destroy and then running the authoritative network + // teardown below would strip filtering from a container that is still + // running, which is the fail-open this ordering exists to prevent. The + // `?` leaves the rules in place and lets the caller retry. + if container + .is_defined() + .map_err(|e| probe_failed("exists", container_name, e))? + { + // `destroy` force-stops and removes the container, so once it + // returns no container process can run. Destroy *before* removing + // the firewall rules so nothing runs without egress filtering + // during teardown; if the destroy fails, leave the rules in place. + container.destroy().map_err(|e| { + MxcError::backend_error(format!("Failed to destroy container: {e}")) + })?; + } + cleanup_network_authoritative(container_name, veth.as_deref(), &mut logger)?; + // Terminal phase: the container is gone, so nothing will take this + // lock for that name again until something creates the container + // afresh. Reclaim the file rather than leave one zero-byte inode per + // sandbox name the host ever provisioned. + _lifecycle_lock.release_and_reclaim(&container, container_name); + Ok(DeprovisionResult { metadata: None }) + } + + fn validate_provision( + &self, + request: &ExecutionRequest, + config: Option<&LxcConfig>, + ) -> Result<(), MxcError> { + validate_lxc_config(config)?; + resolve_container_name(request)?; + reject_start_policy_on_other_phase("provision", &request.policy) + } + + fn validate_start( + &self, + sandbox_id: &str, + request: &ExecutionRequest, + _config: Option<&()>, + ) -> Result<(), MxcError> { + extract_container_name(sandbox_id)?; + validate_start_policy(request) + } + + fn validate_exec( + &self, + sandbox_id: &str, + request: &ExecutionRequest, + _config: Option<&()>, + ) -> Result<(), MxcError> { + extract_container_name(sandbox_id)?; + reject_start_policy_on_other_phase("exec", &request.policy) + } + + fn validate_stop( + &self, + sandbox_id: &str, + request: &ExecutionRequest, + _config: Option<&()>, + ) -> Result<(), MxcError> { + extract_container_name(sandbox_id)?; + reject_start_policy_on_other_phase("stop", &request.policy) + } + + fn validate_deprovision( + &self, + sandbox_id: &str, + request: &ExecutionRequest, + _config: Option<&()>, + ) -> Result<(), MxcError> { + extract_container_name(sandbox_id)?; + reject_start_policy_on_other_phase("deprovision", &request.policy) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use wxc_common::models::LifecycleConfig; + use wxc_common::models::NetworkEnforcementMode; + use wxc_common::models::NetworkPolicy; + use wxc_common::models::ProxyConfig; + use wxc_common::mxc_error::MxcErrorCode; + + fn provision_config() -> LxcConfig { + LxcConfig { + distribution: "alpine".to_string(), + release: "3.20".to_string(), + } + } + + fn restrictive_policy() -> ContainerPolicy { + ContainerPolicy { + blocked_hosts: vec!["evil.example.com".to_string()], + ..Default::default() + } + } + + #[test] + fn a_host_restriction_without_an_enforcement_mode_is_accepted_not_rejected() { + // The core change, seen through the start-only rejection: enforcement is + // policy-driven, so a host restriction no longer needs + // `enforcementMode: firewall` to be honored. Under the default + // (capabilities) mode the real start now installs iptables rules, so + // this must accept the policy rather than refuse it as unenforceable. + assert!(reject_unenforceable_network_policy(&restrictive_policy()).is_ok()); + assert!(reject_unenforceable_network_policy(&ContainerPolicy { + allowed_hosts: vec!["example.com".to_string()], + ..Default::default() + }) + .is_ok()); + } + + #[test] + fn an_explicit_default_block_without_an_enforcement_mode_is_accepted() { + // `defaultPolicy: "block"` with no `enforcementMode` is now enforced + // rather than silently dropped, so the start-only rejection no longer + // fires for it. + assert!(reject_unenforceable_network_policy(&ContainerPolicy { + default_network_policy: NetworkPolicy::Block, + ..Default::default() + }) + .is_ok()); + } + + #[test] + fn an_explicit_capabilities_mode_is_ignored_not_rejected() { + // A caller may set `enforcementMode: "capabilities"` explicitly. LXC + // ignores the value now, so a restriction carried alongside it is + // accepted and enforced -- the caller gets more enforcement than asked + // for, never a rejection. + let policy = ContainerPolicy { + network_enforcement_mode: NetworkEnforcementMode::Capabilities, + ..restrictive_policy() + }; + assert!(reject_unenforceable_network_policy(&policy).is_ok()); + } + + #[test] + fn allow_local_network_is_rejected_whatever_the_enforcement_mode() { + // `IngressManager` refuses this once invoked, and ingress now installs + // unconditionally, so it is always invoked. A dry run stops before the + // ingress apply, so this request-only rejection is what gives the dry + // run the same verdict as the real start. Mode no longer changes the + // outcome; looping the modes only shows the verdict does not depend on + // it. + for mode in [ + NetworkEnforcementMode::Capabilities, + NetworkEnforcementMode::Firewall, + NetworkEnforcementMode::Both, + ] { + let policy = ContainerPolicy { + network_enforcement_mode: mode, + allow_local_network: true, + ..Default::default() + }; + let err = reject_unenforceable_network_policy(&policy) + .expect_err("allowLocalNetwork has no enforceable LXC implementation"); + assert!( + format!("{err}").contains("allowLocalNetwork"), + "the refusal has to name the field the caller set, got {err}" + ); + } + } + + #[test] + fn a_policy_that_leaves_allow_local_network_alone_is_not_rejected_for_it() { + // The negative control: the gate above must not swallow ordinary + // starts. + assert!(reject_unenforceable_network_policy(&ContainerPolicy::default()).is_ok()); + } + + #[test] + fn a_network_block_that_expresses_no_restriction_is_still_a_network_block() { + // `network: {}`, and a block that only restates the `allowLocalNetwork: + // false` default, both parse to a policy identical to the struct + // default -- every narrower bit reads false. Only `network_specified` + // can see them. The phase contract says provision, exec, stop, and + // deprovision take no network section, so without this bit they were + // accepted there and silently ignored. + let empty_block = ContainerPolicy { + network_specified: true, + ..Default::default() + }; + assert!(has_network_policy(&empty_block)); + assert!(reject_start_policy_on_other_phase("exec", &empty_block).is_err()); + } + + #[test] + fn a_policy_with_no_network_section_is_not_a_network_policy() { + // The negative control for the test above: without it, "reject + // everything" would pass, and the plain start in + // run_lxc_state_aware_test.sh would start failing on every phase. + let none = ContainerPolicy::default(); + assert!(!has_network_policy(&none)); + assert!(reject_start_policy_on_other_phase("exec", &none).is_ok()); + } + + #[test] + fn backend_key_matches_wire_format() { + assert_eq!( + ::BACKEND_KEY, + "lxc" + ); + } + + #[test] + fn id_prefix_matches_wire_format() { + assert_eq!( + ::ID_PREFIX, + "lxc" + ); + } + + /// A `Library` exec is refused before the sandbox id is even parsed. + /// + /// `attach_run` relays the workload's output to this process's stdio and + /// returns no pipes, so this backend cannot serve an in-process caller. The + /// refusal has to precede any work, and this id would otherwise fail as + /// `MalformedId` first — so any error but the refusal means the consumer + /// check came too late. + #[test] + fn a_library_exec_is_refused_before_the_workload_runs() { + let mut runner = LxcStateAwareRunner; + let err = runner + .exec( + "not-a-valid-sandbox-id", + &ExecutionRequest::default(), + None, + ExecConsumer::Library, + ) + .expect_err("an in-process caller must be refused"); + assert!( + err.message + .contains("does not support exec for an in-process caller"), + "expected the shared refusal ahead of the id check, got: {}", + err.message + ); + } + + #[test] + fn extract_container_name_unwraps_lxc_prefix() { + assert_eq!( + extract_container_name("lxc:mxc-abcd1234").unwrap(), + "mxc-abcd1234" + ); + } + + #[test] + fn extract_container_name_rejects_other_prefix() { + let err = extract_container_name("iso:abc").unwrap_err(); + assert_eq!(err.code, MxcErrorCode::MalformedId); + } + + #[test] + fn extract_container_name_rejects_missing_colon() { + let err = extract_container_name("no-colon").unwrap_err(); + assert_eq!(err.code, MxcErrorCode::MalformedId); + } + + #[test] + fn extract_container_name_rejects_empty_payload() { + let err = extract_container_name("lxc:").unwrap_err(); + assert_eq!(err.code, MxcErrorCode::MalformedId); + } + + #[test] + fn extract_container_name_rejects_invalid_name_chars() { + let err = extract_container_name("lxc:name/with/slash").unwrap_err(); + assert_eq!(err.code, MxcErrorCode::MalformedId); + } + + #[test] + fn is_valid_container_name_rejects_dot() { + // '.' is stripped by the iptables chain-name derivation, so "a.b" and + // "ab" would collide onto the same chain; reject dotted names. + assert!(!is_valid_container_name("a.b")); + } + + #[test] + fn is_valid_container_name_rejects_overlong_name() { + // One character over the bound: the chain derivation would truncate it, + // letting names that differ only past the bound collide. + assert!(!is_valid_container_name( + &"a".repeat(MAX_CONTAINER_NAME_LEN + 1) + )); + } + + #[test] + fn is_valid_container_name_accepts_max_length_name() { + assert!(is_valid_container_name(&"a".repeat(MAX_CONTAINER_NAME_LEN))); + } + + #[test] + fn extract_container_name_rejects_dotted_name() { + let err = extract_container_name("lxc:a.b").unwrap_err(); + assert_eq!(err.code, MxcErrorCode::MalformedId); + } + + #[test] + fn generated_container_name_fits_iptables_chain_bound() { + // The auto-generated name must itself satisfy the tightened rules so the + // firewall chain derived from it stays within the netfilter length bound + // and is collision-resistant (a deterministic hash of the full name is + // folded in; the mapping is not injective, only hard to collide). + let ContainerName::Minted(name) = resolve_container_name(&ExecutionRequest::default()) + .expect("a default request carries an empty containerId") + else { + panic!("an empty containerId must mint a name rather than adopt one"); + }; + assert!( + is_valid_container_name(&name), + "generated name {name:?} is invalid" + ); + assert!(name.len() <= MAX_CONTAINER_NAME_LEN); + } + + #[test] + fn no_filesystem_block_at_all_is_not_a_filesystem_policy() { + // Plain provisions must stay outside the filesystem phase gate. + assert!( + !has_filesystem_policy(&ContainerPolicy::default()), + "a policy with no filesystem section must not look like one" + ); + } + + #[test] + fn any_one_path_list_alone_is_a_filesystem_policy() { + // The gate is an OR across the three lists, so a dropped clause still + // leaves the other two answering true. Each list is exercised on its + // own so that a phase stops refusing only the list that went missing. + let readwrite = ContainerPolicy { + readwrite_paths: vec!["/tmp/rw".to_string()], + ..Default::default() + }; + let readonly = ContainerPolicy { + readonly_paths: vec!["/tmp/ro".to_string()], + ..Default::default() + }; + let denied = ContainerPolicy { + denied_paths: vec!["/tmp/denied".to_string()], + ..Default::default() + }; + + assert!( + has_filesystem_policy(&readwrite), + "readwritePaths alone must be seen as a filesystem policy" + ); + assert!( + has_filesystem_policy(&readonly), + "readonlyPaths alone must be seen as a filesystem policy" + ); + assert!( + has_filesystem_policy(&denied), + "deniedPaths alone must be seen as a filesystem policy" + ); + } + + #[test] + fn a_minted_name_nobody_holds_is_the_one_used() { + let (chosen, ()) = + mint_unused_container_name("mxc-first".to_string(), |_| Ok(Some(()))).expect("free"); + assert_eq!( + chosen, "mxc-first", + "a name that is free must be used as minted" + ); + } + + #[test] + fn a_minted_name_that_collides_is_re_minted_rather_than_adopted() { + // provision computed `created = !is_defined()` and skipped the create + // when the name was taken, so a collision on a 32-bit token silently + // handed the caller a container that was already in use. Minting is + // the phase that has to resolve it: by start the name is all that is + // left, and a caller-supplied name is adopted on purpose. + let mut probes = 0; + let (chosen, ()) = mint_unused_container_name("mxc-taken".to_string(), |_| { + probes += 1; + Ok(if probes == 1 { None } else { Some(()) }) + }) + .expect("a free name after one collision"); + + assert_ne!( + chosen, "mxc-taken", + "a name that is already defined must not be adopted" + ); + assert!( + is_valid_container_name(&chosen), + "the re-minted name {chosen:?} must still be a valid container name" + ); + assert!(chosen.len() <= MAX_CONTAINER_NAME_LEN); + } + + #[test] + fn a_claim_that_succeeds_hands_back_what_it_took() { + // The claim exists so the caller can hold a lock across it. If the + // payload were dropped on the way out, the lock would release before + // the create it is meant to cover and the race would be back with the + // retry loop still passing its own tests. + let (chosen, payload) = mint_unused_container_name("mxc-first".to_string(), |candidate| { + Ok(Some(format!("locked:{candidate}"))) + }) + .expect("free"); + assert_eq!(chosen, "mxc-first"); + assert_eq!( + payload, "locked:mxc-first", + "the value the claim produced must reach the caller" + ); + } + + #[test] + fn a_host_that_holds_every_name_is_an_error_not_an_adoption() { + let err = mint_unused_container_name("mxc-taken".to_string(), |_| Ok(None::<()>)) + .expect_err("every candidate was taken"); + assert!( + err.message.contains("Could not mint an unused"), + "unexpected message: {}", + err.message + ); + } + + #[test] + fn a_probe_that_cannot_answer_is_not_read_as_free() { + // The whole point of the retry is that "taken" is detected. A probe + // that fails and is treated as "free" reinstates the adoption this + // guards against, so the failure has to propagate. + let err = mint_unused_container_name("mxc-unknown".to_string(), |_| { + Err::, _>(MxcError::backend_error("lxc-info could not be run")) + }) + .expect_err("the probe failed"); + assert!( + err.message.contains("lxc-info could not be run"), + "unexpected message: {}", + err.message + ); + } + + #[cfg(target_os = "linux")] + #[test] + fn a_lifecycle_lock_excludes_a_concurrent_phase_and_releases_on_drop() { + // The lock is the whole of the fix for the check-then-act races, so + // this asserts exclusion directly rather than racing two threads and + // timing them: a second, non-blocking attempt must be refused while + // the lock is held, and must succeed once it is dropped. + let dir = std::env::temp_dir().join(format!("mxc-lifecycle-lock-{}", mint_random_token())); + let name = "locked"; + std::fs::create_dir_all(dir.join(name)).expect("container directory"); + let container = LxcContainer::new(name, Some(dir.to_str().expect("utf-8 temp dir"))); + + let held = + LifecycleLock::acquire(&container, name).expect("the first phase takes the lock"); + + let path = dir.join(format!(".mxc-lifecycle-{name}.lock")); + let probe = || { + let file = std::fs::OpenOptions::new() + .write(true) + .open(&path) + .expect("the lock file the first phase created"); + nix::fcntl::Flock::lock(file, nix::fcntl::FlockArg::LockExclusiveNonblock) + }; + + assert!( + probe().is_err(), + "a second phase must not enter while the first holds the lock" + ); + drop(held); + assert!( + probe().is_ok(), + "the lock must be released when the phase that took it returns" + ); + + let _ = std::fs::remove_dir_all(&dir); + } + + #[cfg(target_os = "linux")] + #[test] + fn a_lifecycle_lock_is_skipped_when_there_is_no_lxc_root() { + // `stop` and `deprovision` have to stay idempotent on a host that never + // had LXC. No root means no container directory under it, so no start + // can have cleared its `is_defined` gate and there is no transition to + // serialize against. + let dir = + std::env::temp_dir().join(format!("mxc-lifecycle-noroot-{}", mint_random_token())); + let container = LxcContainer::new("absent", Some(dir.to_str().expect("utf-8 temp dir"))); + + assert!( + LifecycleLock::acquire(&container, "absent").is_ok(), + "a missing LXC root must not refuse the phase" + ); + assert!( + !dir.exists(), + "the lock must not create the LXC root as a side effect" + ); + } + + #[cfg(target_os = "linux")] + #[test] + fn a_lifecycle_lock_outlives_the_container_directory() { + // `deprovision` holds this lock across the destroy that removes the + // container directory. If the lock file lived in there it would be + // unlinked while held, and the next phase would create a fresh file and + // take a different lock — so the file has to sit in the LXC root and + // still be acquirable once the container is gone. + let dir = std::env::temp_dir().join(format!("mxc-lifecycle-gone-{}", mint_random_token())); + let name = "destroyed"; + std::fs::create_dir_all(dir.join(name)).expect("container directory"); + let container = LxcContainer::new(name, Some(dir.to_str().expect("utf-8 temp dir"))); + + let held = + LifecycleLock::acquire(&container, name).expect("the deprovision takes the lock"); + std::fs::remove_dir_all(dir.join(name)).expect("the destroy removes the container"); + + assert!( + dir.join(format!(".mxc-lifecycle-{name}.lock")).exists(), + "the lock file must survive the container it guards" + ); + drop(held); + assert!( + LifecycleLock::acquire(&container, name).is_ok(), + "a later phase must still be able to take the lock" + ); + + let _ = std::fs::remove_dir_all(&dir); + } + + #[cfg(target_os = "linux")] + #[test] + fn the_terminal_phase_reclaims_the_lock_file() { + // One zero-byte inode per sandbox name, forever, on a host that + // provisions many short-lived sandboxes. + let dir = + std::env::temp_dir().join(format!("mxc-lifecycle-reclaim-{}", mint_random_token())); + let name = "reclaimed"; + std::fs::create_dir_all(&dir).expect("lxc root"); + let container = LxcContainer::new(name, Some(dir.to_str().expect("utf-8 temp dir"))); + let path = dir.join(format!(".mxc-lifecycle-{name}.lock")); + + let held = LifecycleLock::acquire(&container, name).expect("deprovision takes the lock"); + assert!(path.exists(), "the lock file must exist while held"); + held.release_and_reclaim(&container, name); + assert!( + !path.exists(), + "the terminal phase must not leave its lock file behind" + ); + + // And the next provision of the same name still works. + assert!( + LifecycleLock::acquire(&container, name).is_ok(), + "reclaiming must not wedge the name" + ); + + let _ = std::fs::remove_dir_all(&dir); + } + + #[cfg(target_os = "linux")] + #[test] + fn a_lock_on_a_replaced_file_is_not_accepted_as_held() { + // `flock` follows the descriptor, not the name. A lock taken on a file + // that has since been replaced excludes nobody, because the next phase + // opens the new inode and locks that instead. Accepting it would hand + // out two simultaneous "exclusive" locks on one sandbox. + let dir = std::env::temp_dir().join(format!("mxc-lifecycle-inode-{}", mint_random_token())); + std::fs::create_dir_all(&dir).expect("lxc root"); + let name = "replaced"; + let container = LxcContainer::new(name, Some(dir.to_str().expect("utf-8 temp dir"))); + let path = dir.join(format!(".mxc-lifecycle-{name}.lock")); + + let held = LifecycleLock::acquire(&container, name).expect("first acquire"); + // Stand a different inode at the same path, exactly as an unlink plus a + // later create would. + std::fs::remove_file(&path).expect("unlink the locked inode"); + std::fs::write(&path, b"").expect("a fresh inode at the same path"); + + let guard = held._guard.as_ref().expect("the lock is held on Linux"); + assert!( + !LifecycleLock::path_still_names(guard, &path.to_string_lossy()), + "a lock on a replaced inode must not read as current" + ); + + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn validate_provision_requires_distribution_and_release() { + let runner = LxcStateAwareRunner::new(); + let err = runner + .validate_provision(&ExecutionRequest::default(), Some(&LxcConfig::default())) + .unwrap_err(); + assert_eq!(err.code, MxcErrorCode::MalformedRequest); + } + + #[test] + fn validate_provision_accepts_config_and_generated_id() { + let runner = LxcStateAwareRunner::new(); + runner + .validate_provision(&ExecutionRequest::default(), Some(&provision_config())) + .unwrap(); + } + + #[test] + fn validate_provision_rejects_invalid_container_id() { + let runner = LxcStateAwareRunner::new(); + let req = ExecutionRequest { + container_id: "bad/name".to_string(), + ..Default::default() + }; + let err = runner + .validate_provision(&req, Some(&provision_config())) + .unwrap_err(); + assert_eq!(err.code, MxcErrorCode::MalformedRequest); + } + + #[test] + fn validate_provision_rejects_dotted_container_id() { + // A dotted containerId would collide with its dot-stripped sibling on + // the derived iptables chain, so provisioning must reject it up front. + let runner = LxcStateAwareRunner::new(); + let req = ExecutionRequest { + container_id: "has.dot".to_string(), + ..Default::default() + }; + let err = runner + .validate_provision(&req, Some(&provision_config())) + .unwrap_err(); + assert_eq!(err.code, MxcErrorCode::MalformedRequest); + } + + #[test] + fn validate_provision_rejects_start_phase_policy() { + let runner = LxcStateAwareRunner::new(); + let req = ExecutionRequest { + policy: ContainerPolicy { + readonly_paths: vec!["/workspace".to_string()], + ..Default::default() + }, + ..Default::default() + }; + let err = runner + .validate_provision(&req, Some(&provision_config())) + .unwrap_err(); + assert_eq!(err.code, MxcErrorCode::PolicyValidation); + } + + #[test] + fn validate_start_accepts_policy_and_lxc_id() { + let runner = LxcStateAwareRunner::new(); + let req = ExecutionRequest { + policy: ContainerPolicy { + readonly_paths: vec!["/workspace".to_string()], + ..Default::default() + }, + ..Default::default() + }; + runner + .validate_start("lxc:mxc-abcd1234", &req, None) + .unwrap(); + } + + #[test] + fn validate_start_rejects_a_start_the_real_start_would_refuse() { + // A dry run stops after `validate_start` and answers with an empty + // success envelope (`state_aware_dispatch.rs`, `Phase::Start`). While + // the start-only rejections lived inside `apply_network_policy`, the + // dry run could not see them, so it answered "this start is fine" for + // policies the real start refuses outright -- the one question a dry run + // exists to answer, answered wrongly. These are the two rejections that + // need nothing but the request. + let runner = LxcStateAwareRunner::new(); + + let proxy = ExecutionRequest { + policy: ContainerPolicy { + network_proxy: ProxyConfig { + builtin_test_server: true, + ..Default::default() + }, + ..Default::default() + }, + ..Default::default() + }; + let err = runner + .validate_start("lxc:mxc-abcd1234", &proxy, None) + .unwrap_err(); + assert_eq!(err.code, MxcErrorCode::PolicyValidation); + assert!( + err.message.contains("network.proxy"), + "expected the proxy rejection, got: {}", + err.message + ); + + // allowLocalNetwork: the inbound chain can only open every source, which + // is broader than the local-network access requested, so IngressManager + // refuses it and the real start aborts. This is the other rejection that + // needs only the request, so the dry run must give the same verdict. + let permissive_inbound = ExecutionRequest { + policy: ContainerPolicy { + allow_local_network: true, + ..Default::default() + }, + ..Default::default() + }; + let err = runner + .validate_start("lxc:mxc-abcd1234", &permissive_inbound, None) + .unwrap_err(); + assert_eq!(err.code, MxcErrorCode::PolicyValidation); + assert!( + err.message.contains("allowLocalNetwork"), + "expected the allowLocalNetwork rejection, got: {}", + err.message + ); + } + + #[test] + fn validate_start_accepts_a_restriction_without_an_enforcement_mode() { + // The negative control for the test above, and the core behavior change + // seen through the caller: the same host restriction with no + // `enforcementMode` is now enforceable through iptables, so validation + // accepts it instead of refusing it as unenforceable. Without this, + // "reject everything" would pass. + let runner = LxcStateAwareRunner::new(); + let req = ExecutionRequest { + policy: restrictive_policy(), + ..Default::default() + }; + runner + .validate_start("lxc:mxc-abcd1234", &req, None) + .unwrap(); + } + + #[test] + fn validate_exec_rejects_policy() { + let runner = LxcStateAwareRunner::new(); + let req = ExecutionRequest { + policy: ContainerPolicy { + blocked_hosts: vec!["example.com".to_string()], + network_enforcement_mode: NetworkEnforcementMode::Firewall, + ..Default::default() + }, + ..Default::default() + }; + let err = runner + .validate_exec("lxc:mxc-abcd1234", &req, None) + .unwrap_err(); + assert_eq!(err.code, MxcErrorCode::PolicyValidation); + } + + #[test] + fn state_aware_runner_is_constructible_next_to_one_shot_lifecycle() { + let _runner = LxcStateAwareRunner::new(); + let lifecycle = LifecycleConfig::default(); + assert!(lifecycle.destroy_on_exit); + } +} diff --git a/src/core/lxc/Cargo.toml b/src/core/lxc/Cargo.toml index 6b0b692d7..551ac5ca9 100644 --- a/src/core/lxc/Cargo.toml +++ b/src/core/lxc/Cargo.toml @@ -22,6 +22,7 @@ mxc_engine = { workspace = true } lxc_common = { workspace = true } anyhow = { workspace = true } clap = { workspace = true } +serde_json = { workspace = true } nanvix_binaries = { path = "../../backends/nanvix/binaries", optional = true } [target.'cfg(target_arch = "x86_64")'.dependencies] diff --git a/src/core/lxc/src/main.rs b/src/core/lxc/src/main.rs index 3aab93a1b..8ff32c9c8 100644 --- a/src/core/lxc/src/main.rs +++ b/src/core/lxc/src/main.rs @@ -6,10 +6,13 @@ use std::process; use std::time::Instant; use clap::Parser; -use wxc_common::config_parser::load_request; +use wxc_common::config_parser::{load_mxc_request, ParseError}; use wxc_common::logger::{Logger, Mode}; use wxc_common::models::{ExecutionRequest, ScriptResponse}; +use wxc_common::mxc_error::{MxcError, ResponseEnvelope}; use wxc_common::script_runner::handle_dry_run_exit; +use wxc_common::state_aware_dispatch::DispatchOutcome; +use wxc_common::state_aware_request::{MxcRequest, ParsedStateAwareRequest}; use wxc_common::telemetry; use lxc_common::signal_cleanup; @@ -95,9 +98,22 @@ fn delete_lxc_container(name: &str, logger: &mut Logger) -> bool { let container = LxcContainer::new(name, None); - if !container.is_defined() { - logger.log_line(&format!("Container '{}' does not exist.", name)); - return false; + match container.is_defined() { + Ok(true) => {} + Ok(false) => { + logger.log_line(&format!("Container '{}' does not exist.", name)); + return false; + } + Err(e) => { + // An unanswered probe is not an answer of "absent". The caller + // asked for the container to be gone, so try the destroy and let + // its outcome be the report rather than leaving a container behind + // on the strength of a question we could not ask. + logger.log_line(&format!( + "Could not determine whether container '{}' exists ({}); attempting delete anyway.", + name, e + )); + } } match container.destroy() { @@ -112,6 +128,60 @@ fn delete_lxc_container(name: &str, logger: &mut Logger) -> bool { } } +fn run_state_aware_main( + mut parsed: ParsedStateAwareRequest, + dry_run: bool, + experimental: bool, + testing_features: bool, + logger: &mut Logger, +) -> ! { + parsed.request.experimental_enabled = experimental; + parsed.request.testing_features_enabled = testing_features; + parsed.request.dry_run = dry_run; + + // Shares the `wxc` executor's telemetry / correlation-vector orchestration + // rather than calling dispatch directly, so a Linux lifecycle emits the same + // events, carries the same MS-CV, and installs the same crash hook. + let outcome = mxc_engine::run_state_aware_with_telemetry(parsed, dry_run, experimental, logger); + + // Mirrors the wxc executor: on dispatch failure the error goes only to the + // auxiliary diagnostic sinks (log file / diagnostic pipe), never the primary + // buffer/stderr, so the client-facing error envelope below is not shadowed + // by a duplicate. + if let Err(error) = &outcome { + logger.log_diagnostic_line(&error.to_string()); + } + + let buffered = logger.get_buffer().to_string(); + if !buffered.is_empty() { + eprint!("{}", buffered); + } + match outcome { + Ok(DispatchOutcome::Envelope(value)) => { + println!("{}", value); + process::exit(0); + } + Ok(DispatchOutcome::ExecCompleted { exit_code }) => process::exit(exit_code), + Err(e) => { + println!("{}", error_envelope_string(&e)); + process::exit(1); + } + } +} + +/// Serialise `error` to its JSON response-envelope string, including the +/// last-resort fallback for when the envelope itself fails to serialise. +/// +/// Every caller writes it to stdout, which the cross-backend contract (§7.3) +/// reserves for the response envelope in every phase. +fn error_envelope_string(error: &MxcError) -> String { + let envelope: ResponseEnvelope<()> = ResponseEnvelope::from_error(error); + serde_json::to_string(&envelope).unwrap_or_else(|_| { + r#"{"error":{"code":"backend_error","message":"failed to serialise error envelope"}}"# + .to_string() + }) +} + fn main() { // Install before spawning any other threads so the signal mask propagates. // Failure here is fatal: install() either succeeds with the watchdog @@ -168,11 +238,14 @@ fn main() { (String::new(), false) }; - let mut logger = Logger::new(if cli.debug { - Mode::Console - } else { - Mode::Buffer - }); + // Always buffered until the request is parsed. `--debug` selects console + // logging, which writes diagnostics straight to stdout -- but a state-aware + // request needs stdout to carry nothing but its JSON envelope, and config + // parsing emits warnings (overlapping path lists, a mount path missing on + // the host) through this logger before the phase is even known. Promoting + // to console only on the one-shot branch keeps `{ debug: true }` from + // corrupting the state-aware channel. + let mut logger = Logger::new(Mode::Buffer); if let Some(ref log_path) = cli.log_file { if let Err(e) = logger.enable_file_sink(std::path::Path::new(log_path)) { @@ -195,14 +268,36 @@ fn main() { } // Load request - let mut request = match load_request(&config_data, &mut logger, is_base64) { - Ok(r) => r, - Err(_) => { + let request = match load_mxc_request(&config_data, &mut logger, is_base64) { + Ok(MxcRequest::OneShot(req)) => { + // One-shot owns stdout outright, so `--debug` streams there as it + // always has. Flush what was buffered while the phase was still + // unknown so the switch costs no output. + if cli.debug { + let buffered = logger.promote_to_console(); + print!("{}", buffered); + } + req + } + Ok(MxcRequest::StateAware(parsed)) => run_state_aware_main( + parsed, + cli.dry_run, + cli.experimental, + cli.allow_testing_features, + &mut logger, + ), + Err(ParseError::OneShot(_)) | Err(ParseError::Decode(_)) => { eprint!("Request error\n{}", logger.get_buffer()); process::exit(1); } + Err(ParseError::StateAware(e)) => { + println!("{}", error_envelope_string(&e)); + eprint!("{}", logger.get_buffer()); + process::exit(1); + } }; + let mut request = request; request.experimental_enabled = cli.experimental; request.testing_features_enabled = cli.allow_testing_features; request.dry_run = cli.dry_run; diff --git a/src/core/mxc_engine/src/lib.rs b/src/core/mxc_engine/src/lib.rs index e5fe40fb0..f2aa02842 100644 --- a/src/core/mxc_engine/src/lib.rs +++ b/src/core/mxc_engine/src/lib.rs @@ -23,6 +23,8 @@ //! - [`run`] / [`resolve_runner`] (Windows) — run-to-completion backend //! selection and execution. //! - [`run_state_aware`] — state-aware lifecycle backend resolution + dispatch. +//! - [`run_state_aware_with_telemetry`] — the same dispatch wrapped in the +//! shared telemetry / correlation-vector orchestration every executor needs. //! - [`platform_support`] / [`PlatformSupport`] — host support detection. //! - [`available_backends`] / [`AvailableBackend`] — read-only host //! backend-availability probe (with effective isolation tier). @@ -39,6 +41,7 @@ mod probe; #[cfg(any(target_os = "windows", target_os = "linux", target_os = "macos"))] mod run; mod state_aware; +mod state_aware_telemetry; pub use error::{Error, ErrorCode}; #[cfg(all(target_os = "windows", feature = "isolation_session"))] @@ -53,6 +56,7 @@ pub use probe::{available_backends, AvailableBackend, BackendCapability}; #[cfg(any(target_os = "windows", target_os = "linux", target_os = "macos"))] pub use run::{resolve_runner, run, ResolvedRunner}; pub use state_aware::{exec_state_aware_json, run_state_aware, run_state_aware_json}; +pub use state_aware_telemetry::run_state_aware_with_telemetry; use wxc_common::logger::{Logger, Mode}; use wxc_common::sandbox_process::{SandboxProcess, StreamCloser}; diff --git a/src/core/mxc_engine/src/state_aware.rs b/src/core/mxc_engine/src/state_aware.rs index 2c2d4e532..86452078b 100644 --- a/src/core/mxc_engine/src/state_aware.rs +++ b/src/core/mxc_engine/src/state_aware.rs @@ -81,6 +81,11 @@ pub fn run_state_aware( let mut runner = isolation_session_common::IsolationSessionRunner::new(); wxc_common::state_aware_dispatch::dispatch_state_aware(&mut runner, parsed, dry_run) } + #[cfg(target_os = "linux")] + wxc_common::models::ContainmentBackend::Lxc => { + let mut runner = lxc_common::state_aware::LxcStateAwareRunner::new(); + wxc_common::state_aware_dispatch::dispatch_state_aware(&mut runner, parsed, dry_run) + } #[cfg(all(target_os = "windows", feature = "wslc"))] wxc_common::models::ContainmentBackend::Wslc => { let mut runner = wslc_common::WslcStateAwareRunner::new(); @@ -129,10 +134,51 @@ pub fn exec_state_aware( // rather than falling through to the generic `unsupported_phase`. #[cfg(not(all(target_os = "windows", feature = "wslc")))] wxc_common::models::ContainmentBackend::Wslc => Err(wslc_unavailable()), - _ => Err(MxcError::unsupported_phase(format!( - "backend {:?} does not implement the state-aware lifecycle", - backend - ))), + _ => Err(exec_unsupported_error(&backend)), + } +} + +/// The error returned when a backend cannot serve a **streaming** exec. +/// +/// Two distinct situations reach here and the caller needs to tell them apart: +/// +/// - The backend has no state-aware lifecycle at all, so nothing about it works +/// through these APIs. +/// - The backend does implement the lifecycle ([`run_state_aware`] dispatches +/// provision/start/exec/stop/deprovision for it) but has no streaming +/// [`SandboxProcess`], so only this one API is unavailable. +/// +/// LXC is the second case. Reporting it as "does not implement the state-aware +/// lifecycle" sent callers off to debug a provision path that works fine, so +/// the message names the real gap and points at the API that does work. +fn exec_unsupported_error(backend: &wxc_common::models::ContainmentBackend) -> MxcError { + if backend_has_state_aware_lifecycle(backend) { + MxcError::unsupported_phase(format!( + "backend {backend:?} implements the state-aware lifecycle but not streaming exec; \ + use the non-streaming exec phase instead" + )) + } else { + MxcError::unsupported_phase(format!( + "backend {backend:?} does not implement the state-aware lifecycle" + )) + } +} + +/// Whether `backend` has a `StatefulSandboxBackend` impl wired into +/// [`run_state_aware`] on this target. Kept next to that `match` so the two stay +/// in step — a backend added there without being added here would be described +/// by the wrong error. +fn backend_has_state_aware_lifecycle(backend: &wxc_common::models::ContainmentBackend) -> bool { + match backend { + #[cfg(target_os = "windows")] + wxc_common::models::ContainmentBackend::WindowsSandbox => true, + #[cfg(all(target_os = "windows", feature = "isolation_session"))] + wxc_common::models::ContainmentBackend::IsolationSession => true, + #[cfg(all(target_os = "windows", feature = "wslc"))] + wxc_common::models::ContainmentBackend::Wslc => true, + #[cfg(target_os = "linux")] + wxc_common::models::ContainmentBackend::Lxc => true, + _ => false, } } @@ -254,6 +300,32 @@ mod tests { assert!(error.message.contains("experimental")); } + #[test] + #[cfg(target_os = "linux")] + fn exec_error_distinguishes_missing_streaming_from_missing_lifecycle() { + // LXC dispatches the lifecycle but has no streaming SandboxProcess. The + // old blanket message sent callers off to debug a provision path that + // works, so the two cases must read differently. + let lxc = exec_unsupported_error(&ContainmentBackend::Lxc); + assert_eq!(lxc.code, MxcErrorCode::UnsupportedPhase); + assert!( + lxc.message.contains("not streaming exec"), + "expected the streaming-specific message, got {:?}", + lxc.message + ); + assert!(!lxc.message.contains("does not implement")); + + let bwrap = exec_unsupported_error(&ContainmentBackend::Bubblewrap); + assert_eq!(bwrap.code, MxcErrorCode::UnsupportedPhase); + assert!( + bwrap + .message + .contains("does not implement the state-aware lifecycle"), + "expected the no-lifecycle message, got {:?}", + bwrap.message + ); + } + #[test] fn exec_experimental_backend_requires_optin() { // The streaming exec entry point applies the same opt-in gate as the diff --git a/src/core/mxc_engine/src/state_aware_telemetry.rs b/src/core/mxc_engine/src/state_aware_telemetry.rs new file mode 100644 index 000000000..14db1470e --- /dev/null +++ b/src/core/mxc_engine/src/state_aware_telemetry.rs @@ -0,0 +1,292 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +//! Telemetry and correlation-vector orchestration for the state-aware +//! lifecycle. +//! +//! This wraps [`crate::run_state_aware`] with everything a lifecycle dispatch +//! needs around it: telemetry init/shutdown, process attribution (backend + +//! phase), the Microsoft Correlation Vector (MS-CV) seed/spin plan, the crash +//! panic hook, and the terminal `emit_state_aware` event. +//! +//! It lives here — rather than in an executor binary — because every executor +//! that dispatches the state-aware lifecycle needs identical behavior. When +//! this logic lived in `wxc`'s `main.rs`, the `lxc` executor called +//! [`crate::run_state_aware`] directly and so silently produced no lifecycle +//! telemetry and no correlation vector: a Linux lifecycle was invisible to the +//! same dashboards that observed a Windows one, and a client relaying a +//! provision-seeded cV had nothing to relay. Sharing the orchestration is what +//! keeps the two executors from drifting apart again. + +use std::time::Instant; + +use wxc_common::logger::Logger; +use wxc_common::mxc_error::MxcError; +use wxc_common::state_aware_dispatch::{resolve_backend, DispatchOutcome}; +use wxc_common::state_aware_request::ParsedStateAwareRequest; +use wxc_common::telemetry; + +/// The correlation-vector action a state-aware phase should take, decided purely +/// from the phase and the relayed value. Returned by [`plan_correlation_vector`] +/// so the seed-vs-spin decision is unit-testable without touching the RNG/clock; +/// the caller executes the plan against the (nondeterministic) operators. +#[derive(Debug, PartialEq, Eq)] +enum CvPlan<'a> { + /// Mint a fresh vector. Used for `provision`, and for any non-provision phase + /// whose relayed value is missing, empty, or not relayable (malformed / + /// hostile) — so garbage never even reaches the `spin` operator. + Seed, + /// Spin the relayed value to derive this phase's child vector. Only planned + /// for a value [`is_relayable`](telemetry::correlation_vector::is_relayable) + /// vouches for, so `spin` here always builds on a real parent rather than + /// silently reseeding. + Spin(&'a str), +} + +/// Plans the Microsoft Correlation Vector (MS-CV) action for a state-aware phase. +/// +/// Provision always seeds a fresh random base. Every later phase spins the +/// relayed `incoming_cv` so sibling phases get distinct vectors that still share +/// the lifecycle prefix — but only when the relayed value is actually relayable +/// (a valid mutable or frozen vector). A missing, empty, or malformed relayed +/// value is planned as [`CvPlan::Seed`] so the `Spin` arm never stands in for a +/// reseed; the decision stays a pure function of `(is_provision, incoming_cv)` +/// with no RNG, so it is deterministically testable. +fn plan_correlation_vector(is_provision: bool, incoming_cv: Option<&str>) -> CvPlan<'_> { + match incoming_cv { + Some(cv) if !is_provision && telemetry::correlation_vector::is_relayable(cv) => { + CvPlan::Spin(cv) + } + _ => CvPlan::Seed, + } +} + +/// Executes the pure [`plan_correlation_vector`] plan against the (nondeterministic) +/// MS-CV operators, returning this phase's correlation vector. Empty when +/// telemetry is inactive so an inactive provider does no RNG/clock work and +/// provision output is unchanged. +fn compute_phase_correlation( + telemetry_active: bool, + is_provision: bool, + incoming_cv: Option<&str>, +) -> String { + if !telemetry_active { + return String::new(); + } + match plan_correlation_vector(is_provision, incoming_cv) { + CvPlan::Seed => telemetry::correlation_vector::seed(), + CvPlan::Spin(cv) => telemetry::correlation_vector::spin(cv), + } +} + +/// Injects the freshly-seeded correlation vector into a provision result +/// envelope (`{ "result": { ..., "correlationVector": "" } }`) so the client +/// can relay it into every later phase of the lifecycle. No-op when the outcome +/// is not a result envelope (exec-completed / error paths carry no cV). +fn inject_correlation_vector(outcome: &mut Result, cv: &str) { + if let Ok(DispatchOutcome::Envelope(value)) = outcome { + if let Some(result) = value.get_mut("result").and_then(|r| r.as_object_mut()) { + result.insert( + "correlationVector".to_string(), + serde_json::Value::String(cv.to_string()), + ); + } + } +} + +/// Run a state-aware phase with full telemetry and correlation-vector +/// orchestration, returning the dispatch outcome for the caller to render. +/// +/// The caller still owns the terminal behavior (flushing `logger`'s buffer, +/// writing the envelope to stdout, choosing an exit code) — only the +/// observability wrapper is shared. Telemetry is gated on `experimental` +/// exactly like the one-shot path and reads the same typed +/// `experimental.telemetry` field; a malformed telemetry block is already +/// rejected at parse time, so there is no client-error handling here. +pub fn run_state_aware_with_telemetry( + parsed: ParsedStateAwareRequest, + dry_run: bool, + experimental: bool, + logger: &mut Logger, +) -> Result { + // Resolve attribution (phase + backend) BEFORE dispatch consumes `parsed`. + let phase = parsed.phase.as_str(); + // Whether this invocation is the provision phase. Provision seeds a fresh + // random correlation-vector base and returns it in the result envelope; + // every later phase relays that base back and spins it. We deliberately + // ignore any client-supplied `correlationVector` on provision so a lifecycle + // can never be seeded with a stale or foreign vector. + let is_provision = phase == "provision"; + // The relayed correlation vector for non-provision phases (the base seeded at + // provision). Captured before `dispatch` consumes `parsed`. `None` for + // provision (which seeds its own below). + let incoming_cv = if is_provision { + None + } else { + parsed.correlation_vector.clone() + }; + let resolved_backend = resolve_backend(&parsed).ok(); + let backend = resolved_backend + .as_ref() + .map(|b| b.wire_name()) + .unwrap_or("unknown"); + let telemetry_active = if experimental { + parsed + .request + .experimental + .telemetry + .as_ref() + .map(|c| telemetry::init(c, logger)) + .unwrap_or(false) + } else { + false + }; + + // Compute this phase's MS-CV, executing the pure seed-vs-spin plan against + // the operators. Only computed when telemetry is active so an inactive + // provider does no work and provision output is unchanged. + let correlation = + compute_phase_correlation(telemetry_active, is_provision, incoming_cv.as_deref()); + + // Attribute out-of-band emit paths (the console-control handler installed in + // `main`, and the panic hook installed just below) to the resolved backend + // and the lifecycle phase, and install a crash-telemetry panic hook for this + // dispatch — mirroring the one-shot path, which the `-> !` entry points + // bypass. The shared hook chains the previous hook (default stderr backtrace + // still prints) and is panic-free. + if telemetry_active { + if let Some(containment) = resolved_backend.as_ref() { + telemetry::set_process_context(containment); + } + telemetry::set_process_phase(phase); + // Stash this phase's correlation vector so out-of-band events + // (panic / cancellation) carry the same cV as the terminal emit below. + telemetry::set_process_correlation_vector(&correlation); + telemetry::install_panic_hook(); + } + + let started = Instant::now(); + let mut outcome = crate::run_state_aware(parsed, dry_run); + let elapsed = started.elapsed(); + + // For provision, return the freshly-seeded correlation vector to the client + // by injecting it into the result envelope so it can be relayed into later + // phases. Gated on telemetry so provision output is unchanged when telemetry + // is off. + if is_provision && telemetry_active { + inject_correlation_vector(&mut outcome, &correlation); + } + + // Emit lifecycle telemetry (and shut the provider down) before the caller + // flushes the diagnostic buffer / envelope. Terminal path — safe to shut + // down here. + telemetry::emit_state_aware( + telemetry_active, + telemetry::TelemetryContext { + backend, + phase, + correlation_vector: &correlation, + }, + &outcome, + elapsed, + ); + + outcome +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn plan_correlation_vector_provision_seeds_fresh_vector() { + // Provision ignores any relayed value and always plans a fresh seed. + assert_eq!( + plan_correlation_vector(true, Some("BBBBBBBBBBBBBBBBBBBBBB.5")), + CvPlan::Seed + ); + assert_eq!(plan_correlation_vector(true, None), CvPlan::Seed); + } + + #[test] + fn plan_correlation_vector_phase_spins_relayed_base() { + // A relayable relayed value is spun to derive this phase's child vector. + let base = "AAAAAAAAAAAAAAAAAAAAAA.0"; + assert_eq!( + plan_correlation_vector(false, Some(base)), + CvPlan::Spin(base) + ); + // A valid frozen relayed value is also relayable (spin passes it through). + let frozen = "AAAAAAAAAAAAAAAAAAAAAA.0!"; + assert_eq!( + plan_correlation_vector(false, Some(frozen)), + CvPlan::Spin(frozen) + ); + } + + #[test] + fn plan_correlation_vector_phase_reseeds_for_missing_empty_or_malformed() { + // Missing / empty / non-relayable relayed values plan a fresh `Seed`, so + // the `Spin` arm never stands in for a reseed (garbage never reaches the + // `spin` operator). + for incoming in [None, Some(""), Some("garbage"), Some("short.0")] { + assert_eq!( + plan_correlation_vector(false, incoming), + CvPlan::Seed, + "non-relayable relay {incoming:?} must plan Seed" + ); + } + } + + #[test] + fn compute_phase_correlation_is_empty_when_telemetry_inactive() { + // Inactive telemetry does no RNG/clock work regardless of phase/relay. + assert!(compute_phase_correlation(false, true, None).is_empty()); + assert!( + compute_phase_correlation(false, false, Some("AAAAAAAAAAAAAAAAAAAAAA.0")).is_empty() + ); + } + + #[test] + fn compute_phase_correlation_spins_relayable_and_reseeds_garbage() { + // Active provision seeds a fresh valid vector. + let provisioned = compute_phase_correlation(true, true, None); + assert!(telemetry::correlation_vector::is_relayable(&provisioned)); + // Active non-provision spins a relayable relay onto the shared prefix. + let base = "AAAAAAAAAAAAAAAAAAAAAA.0"; + let spun = compute_phase_correlation(true, false, Some(base)); + assert!(spun.starts_with(&format!("{base}.")), "{spun:?}"); + // Active non-provision with garbage reseeds to a fresh, unrelated vector. + let reseeded = compute_phase_correlation(true, false, Some("garbage")); + assert!(telemetry::correlation_vector::is_relayable(&reseeded)); + assert!(!reseeded.starts_with("garbage")); + } + + #[test] + fn inject_correlation_vector_sets_field_on_envelope() { + let mut outcome: Result = Ok(DispatchOutcome::Envelope( + serde_json::json!({ "result": { "sandboxId": "iso:wxc-abc" } }), + )); + inject_correlation_vector(&mut outcome, "AAAAAAAAAAAAAAAAAAAAAA.0"); + match outcome { + Ok(DispatchOutcome::Envelope(v)) => assert_eq!( + v["result"]["correlationVector"], + serde_json::json!("AAAAAAAAAAAAAAAAAAAAAA.0") + ), + _ => panic!("expected envelope"), + } + } + + #[test] + fn inject_correlation_vector_noop_on_non_envelope() { + // Exec-completed / error outcomes carry no result envelope: injection is + // a no-op and must not panic. + let mut exit: Result = + Ok(DispatchOutcome::ExecCompleted { exit_code: 0 }); + inject_correlation_vector(&mut exit, "AAAAAAAAAAAAAAAAAAAAAA.0"); + assert!(matches!( + exit, + Ok(DispatchOutcome::ExecCompleted { exit_code: 0 }) + )); + } +} diff --git a/src/core/mxc_pty/src/lib.rs b/src/core/mxc_pty/src/lib.rs index 7ebd8e94e..4758851d7 100644 --- a/src/core/mxc_pty/src/lib.rs +++ b/src/core/mxc_pty/src/lib.rs @@ -257,6 +257,10 @@ pub fn run_with_pty(mut command: Command, options: PtyOptions) -> Result(); + // Lets the drain below tell a reader that is still forwarding output from a + // reader that is merely blocked on a pty a leaked process left open. + let bytes_forwarded = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0)); + let reader_progress = std::sync::Arc::clone(&bytes_forwarded); let output_thread = thread::spawn(move || { let mut buf = [0u8; 4096]; let mut signaled = false; @@ -271,6 +275,7 @@ pub fn run_with_pty(mut command: Command, options: PtyOptions) -> Result break, } @@ -312,6 +317,26 @@ pub fn run_with_pty(mut command: Command, options: PtyOptions) -> Result { let status = child.wait().map_err(|e| format!("wait: {}", e))?; @@ -322,29 +347,153 @@ pub fn run_with_pty(mut command: Command, options: PtyOptions) -> Result break PtyOutcome::Exited(status), Ok(None) => { if Instant::now() >= deadline { - let _ = child.kill(); - let _ = child.wait(); + kill_child_and_group(&mut child); break PtyOutcome::TimedOut; } thread::sleep(PtyOptions::POLL_INTERVAL); } Err(e) => { - let _ = child.kill(); - let _ = child.wait(); + kill_child_and_group(&mut child); return Err(format!("try_wait: {}", e)); } } }, }; - // Drain remaining output before returning. The secondary fds are closed - // on child exit, so primary_reader hits EOF and the thread exits. - let _ = output_thread.join(); + // Drain remaining output before returning. + const DRAIN_GRACE: Duration = Duration::from_secs(2); + match outcome { + PtyOutcome::Exited(_) => { + // Normally the child's exit closes its secondary fds, primary_reader + // hits EOF, and the drain thread finishes on its own — so join + // unbounded and every last byte is flushed. + // + // But the direct child exiting does not guarantee EOF. A process it + // left running that escaped into its own session still holds the pty + // secondary open, and then this join blocks forever. That defeats a + // requested timeout exactly as the killed-child case does, just via + // a different outcome. So whenever a deadline was asked for, bound + // the drain. + // + // Neither a flat grace nor the remaining budget is the right bound + // on its own. A flat grace charges the common case -- a child that + // exits quickly having buffered a lot of output -- for a hazard a + // different one creates, and truncates output this function promises + // to forward. The remaining budget alone lets a silent leaked holder + // keep the caller for the whole timeout the run never needed, which + // is worse than the flat grace for precisely the case the bound + // exists for. + // + // The two are distinguishable: a real drain keeps producing bytes, + // and a leaked holder of an idle pty produces none. So the bound is + // silence, capped by the budget. Output flowing extends the wait, + // DRAIN_GRACE of nothing ends it, and a pathological slow drip still + // cannot outlast what the caller already authorized. + match deadline { + Some(d) => { + let _ = join_while_draining( + output_thread, + &bytes_forwarded, + DRAIN_GRACE, + d + DRAIN_GRACE, + ); + } + None => { + let _ = output_thread.join(); + } + } + } + PtyOutcome::TimedOut => { + // We killed the child's process group, but a process it left + // running inside the sandbox that escaped into its own session can + // still hold the pty secondary open — primary_reader would then + // never see EOF and an unbounded join would block forever, + // defeating the very timeout we just enforced. Give the drain a + // short grace to flush buffered output, then abandon the reader. + // A one-shot executor exits immediately afterward and the OS reaps + // the thread, but an in-process caller does not, so the thread and + // the pty fd survive until its host exits. + let _ = join_with_timeout(output_thread, DRAIN_GRACE); + } + } Ok(outcome) } -/// Background thread that watches for SIGWINCH on the outer pty +/// Join `handle`, returning once the thread finishes or `grace` elapses — +/// whichever comes first. Returns `true` if the thread finished within the +/// grace period, `false` if it was abandoned. +/// +/// [`run_with_pty`] uses this to drain a killed child's buffered output on the +/// timeout path without risking a permanent hang: a process the timed-out +/// child left running inside the sandbox can keep the pty open, so the reader +/// thread never sees EOF and a plain `JoinHandle::join` would block forever. +/// The abandoned reader thread lives until its host process exits, which is +/// immediate for a one-shot executor and not immediate for an in-process +/// caller such as the SDK or the FFI layer. +/// +/// Implemented with a helper thread that performs the blocking join and +/// signals a channel, since the standard library has no join-with-timeout. +#[cfg_attr(not(any(target_os = "linux", target_os = "macos")), allow(dead_code))] +fn join_with_timeout(handle: std::thread::JoinHandle<()>, grace: Duration) -> bool { + let (tx, rx) = std::sync::mpsc::channel(); + std::thread::spawn(move || { + let _ = handle.join(); + let _ = tx.send(()); + }); + rx.recv_timeout(grace).is_ok() +} + +/// Join `handle`, waiting for as long as it keeps making progress but giving up +/// after `idle_grace` of silence, and in no case later than `hard_deadline`. +/// +/// Returns whether the thread finished. +/// +/// `progress` is a monotonically increasing count the joined thread bumps +/// whenever it does the work being waited on. That is what separates the two +/// situations a plain timeout cannot tell apart: a reader still forwarding a +/// large buffered backlog, which deserves as long as it needs, and a reader +/// blocked on a pty a leaked process is holding open, which will never finish +/// and must not hold the caller. `hard_deadline` keeps even a pathological slow +/// drip from outlasting the budget the caller authorized. +#[cfg_attr(not(any(target_os = "linux", target_os = "macos")), allow(dead_code))] +fn join_while_draining( + handle: std::thread::JoinHandle<()>, + progress: &std::sync::atomic::AtomicU64, + idle_grace: Duration, + hard_deadline: std::time::Instant, +) -> bool { + use std::sync::atomic::Ordering; + + let (tx, rx) = std::sync::mpsc::channel(); + std::thread::spawn(move || { + let _ = handle.join(); + let _ = tx.send(()); + }); + + let mut last_seen = progress.load(Ordering::Relaxed); + loop { + let budget = + idle_grace.min(hard_deadline.saturating_duration_since(std::time::Instant::now())); + match rx.recv_timeout(budget) { + Ok(()) => return true, + // The joining helper vanished without reporting; nothing further to + // wait for. + Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => return true, + Err(std::sync::mpsc::RecvTimeoutError::Timeout) => { + if std::time::Instant::now() >= hard_deadline { + return false; + } + let seen = progress.load(Ordering::Relaxed); + if seen == last_seen { + return false; + } + last_seen = seen; + } + } + } +} + /// (delivered to *some* thread because fd 0 is the outer secondary) and /// forwards the new window size to the inner pty primary via TIOCSWINSZ. /// @@ -526,6 +675,126 @@ mod tests { assert_eq!(PtyOptions::POLL_INTERVAL, Duration::from_millis(500)); } + #[test] + fn join_with_timeout_returns_true_when_thread_finishes() { + let handle = std::thread::spawn(|| {}); + assert!(join_with_timeout(handle, Duration::from_secs(5))); + } + + #[test] + fn join_with_timeout_gives_up_on_a_blocked_thread() { + // A thread that blocks far past the grace must not make + // join_with_timeout wait for it — this is what stops a leaked + // pty-holding child from hanging exec forever. + let (tx, rx) = std::sync::mpsc::channel::<()>(); + let handle = std::thread::spawn(move || { + // Blocks until the test drops `tx`, well past the grace below. + let _ = rx.recv(); + }); + let start = std::time::Instant::now(); + assert!(!join_with_timeout(handle, Duration::from_millis(200))); + assert!( + start.elapsed() < Duration::from_secs(5), + "join_with_timeout must return near the grace, not block" + ); + drop(tx); // let the abandoned thread exit before the test ends + } + + #[test] + fn a_drain_that_keeps_producing_output_is_allowed_past_the_idle_grace() { + // The point of the idle bound: a child that exits having buffered a lot + // of output has done nothing wrong, and a flat grace would truncate it. + // This thread keeps reporting progress well past the grace and must be + // joined in full. + use std::sync::atomic::{AtomicU64, Ordering}; + use std::sync::Arc; + + let progress = Arc::new(AtomicU64::new(0)); + let worker_progress = Arc::clone(&progress); + let handle = std::thread::spawn(move || { + for _ in 0..10 { + std::thread::sleep(Duration::from_millis(60)); + worker_progress.fetch_add(4096, Ordering::Relaxed); + } + }); + + assert!( + join_while_draining( + handle, + &progress, + Duration::from_millis(100), + std::time::Instant::now() + Duration::from_secs(30), + ), + "a reader still forwarding bytes must be waited for, not abandoned" + ); + assert_eq!(progress.load(Ordering::Relaxed), 40960); + } + + #[test] + fn a_silent_drain_is_abandoned_after_the_idle_grace_not_at_the_deadline() { + // The mirror case, and the one the bound exists for: a process the child + // left behind holds the pty open, so the reader blocks and produces + // nothing. Waiting out the caller's remaining budget here would hand a + // leaked descendant the whole timeout the run never needed. + use std::sync::atomic::AtomicU64; + + let progress = AtomicU64::new(0); + let (tx, rx) = std::sync::mpsc::channel::<()>(); + let handle = std::thread::spawn(move || { + let _ = rx.recv(); + }); + + let start = std::time::Instant::now(); + assert!(!join_while_draining( + handle, + &progress, + Duration::from_millis(150), + std::time::Instant::now() + Duration::from_secs(30), + )); + assert!( + start.elapsed() < Duration::from_secs(5), + "a silent reader must be abandoned at the idle grace, not at the \ + far-off deadline; took {:?}", + start.elapsed() + ); + drop(tx); + } + + #[test] + fn even_a_steadily_progressing_drain_cannot_outlast_the_deadline() { + // Progress extends the wait, so on its own it would let a leak that + // drips a byte at a time hold the caller forever -- this test hangs + // outright without the backstop. The budget the caller authorized is + // what bounds it. + use std::sync::atomic::{AtomicU64, Ordering}; + use std::sync::Arc; + + let progress = Arc::new(AtomicU64::new(0)); + let worker_progress = Arc::clone(&progress); + let (tx, rx) = std::sync::mpsc::channel::<()>(); + let handle = std::thread::spawn(move || { + while rx.try_recv().is_err() { + std::thread::sleep(Duration::from_millis(20)); + worker_progress.fetch_add(1, Ordering::Relaxed); + } + }); + + let start = std::time::Instant::now(); + assert!(!join_while_draining( + handle, + &progress, + Duration::from_millis(50), + std::time::Instant::now() + Duration::from_millis(300), + )); + assert!( + start.elapsed() < Duration::from_secs(5), + "the hard deadline must cap a drain that never stops dripping; \ + took {:?}", + start.elapsed() + ); + drop(tx); + } + #[cfg(any(target_os = "linux", target_os = "macos"))] #[test] fn echo_runs_under_pty() { @@ -553,6 +822,36 @@ mod tests { assert!(matches!(outcome, PtyOutcome::TimedOut)); } + /// Regression for the exec-timeout hang: a timed-out command that left a + /// process running (here a backgrounded `sleep` sibling of the foreground + /// command) must not stall `run_with_pty`. The leaked process inherits the + /// pty secondary, so without the process-group kill + bounded drain the + /// output reader never sees EOF and the join blocks until the straggler + /// exits (~30s). With the fix the whole process group is killed and the + /// call returns right after the (sub-second) timeout. + #[cfg(any(target_os = "linux", target_os = "macos"))] + #[test] + fn timeout_returns_promptly_despite_leaked_background_process() { + let _guard = RUN_WITH_PTY_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let mut cmd = Command::new("/bin/sh"); + // Background one sleep, then become another via exec. `sleep 30` + // (not 300) bounds any worst-case leak if this ever regresses. + cmd.arg("-c").arg("sleep 30 & exec sleep 30"); + let opts = PtyOptions { + timeout: Some(Duration::from_millis(750)), + ..PtyOptions::default() + }; + let start = std::time::Instant::now(); + let outcome = run_with_pty(cmd, opts).expect("bridge spawns"); + let elapsed = start.elapsed(); + assert!(matches!(outcome, PtyOutcome::TimedOut)); + assert!( + elapsed < Duration::from_secs(20), + "exec must return promptly after timeout, took {elapsed:?} (leaked \ + process hung the drain?)" + ); + } + /// Documents the libc invariant motivating the FD_CLOEXEC fixup /// inside `run_with_pty`. If a future libc starts defaulting to /// `FD_CLOEXEC` on `openpty(3)`, this test skips with a message — diff --git a/src/core/wxc/src/main.rs b/src/core/wxc/src/main.rs index 52ef9e01a..ef39c2775 100644 --- a/src/core/wxc/src/main.rs +++ b/src/core/wxc/src/main.rs @@ -251,74 +251,6 @@ fn apply_command_override( } } -/// The plan for producing this phase's correlation vector, derived -/// from the phase and the relayed value. Returned by [`plan_correlation_vector`] -/// so the seed-vs-spin decision is unit-testable without touching the RNG/clock; -/// the caller executes the plan against the (nondeterministic) operators. -#[derive(Debug, PartialEq, Eq)] -enum CvPlan<'a> { - /// Mint a fresh vector. Used for `provision`, and for any non-provision phase - /// whose relayed value is missing, empty, or not relayable (malformed / - /// hostile) — so garbage never even reaches the `spin` operator. - Seed, - /// Spin the relayed value to derive this phase's child vector. Only planned - /// for a value [`is_relayable`](telemetry::correlation_vector::is_relayable) - /// vouches for, so `spin` here always builds on a real parent rather than - /// silently reseeding. - Spin(&'a str), -} - -/// Plans the Microsoft Correlation Vector (MS-CV) action for a state-aware phase. -/// -/// Provision always seeds a fresh random base. Every later phase spins the -/// relayed `incoming_cv` so sibling phases get distinct vectors that still share -/// the lifecycle prefix — but only when the relayed value is actually relayable -/// (a valid mutable or frozen vector). A missing, empty, or malformed relayed -/// value is planned as [`CvPlan::Seed`] so the `Spin` arm never stands in for a -/// reseed; the decision stays a pure function of `(is_provision, incoming_cv)` -/// with no RNG, so it is deterministically testable. -fn plan_correlation_vector(is_provision: bool, incoming_cv: Option<&str>) -> CvPlan<'_> { - match incoming_cv { - Some(cv) if !is_provision && telemetry::correlation_vector::is_relayable(cv) => { - CvPlan::Spin(cv) - } - _ => CvPlan::Seed, - } -} - -/// Executes the pure [`plan_correlation_vector`] plan against the (nondeterministic) -/// MS-CV operators, returning this phase's correlation vector. Empty when -/// telemetry is inactive so an inactive provider does no RNG/clock work and -/// provision output is unchanged. -fn compute_phase_correlation( - telemetry_active: bool, - is_provision: bool, - incoming_cv: Option<&str>, -) -> String { - if !telemetry_active { - return String::new(); - } - match plan_correlation_vector(is_provision, incoming_cv) { - CvPlan::Seed => telemetry::correlation_vector::seed(), - CvPlan::Spin(cv) => telemetry::correlation_vector::spin(cv), - } -} - -/// Injects the freshly-seeded correlation vector into a provision result -/// envelope (`{ "result": { ..., "correlationVector": "" } }`) so the client -/// can relay it into every later phase of the lifecycle. No-op when the outcome -/// is not a result envelope (exec-completed / error paths carry no cV). -fn inject_correlation_vector(outcome: &mut Result, cv: &str) { - if let Ok(DispatchOutcome::Envelope(value)) = outcome { - if let Some(result) = value.get_mut("result").and_then(|r| r.as_object_mut()) { - result.insert( - "correlationVector".to_string(), - serde_json::Value::String(cv.to_string()), - ); - } - } -} - /// On a state-aware dispatch failure, record the error only on the auxiliary /// diagnostic sinks (`--log-file` and the diagnostic pipe) via /// [`Logger::log_diagnostic_line`]. It is deliberately kept out of the primary @@ -341,93 +273,11 @@ fn run_state_aware_main( experimental: bool, logger: &mut Logger, ) -> ! { - // Resolve attribution (phase + backend) and telemetry enablement BEFORE - // dispatch consumes `parsed`. State-aware telemetry is gated on - // `--experimental` exactly like the one-shot path, and reads the same typed - // `experimental.telemetry` field — the state-aware parser populates it while - // keeping the per-backend `experimental_raw` block for dispatch. A malformed - // telemetry block is rejected at parse time (as a state-aware envelope), so - // no client-error handling is needed here. - let phase = parsed.phase.as_str(); - // Whether this invocation is the provision phase. Provision seeds a fresh - // random correlation-vector base and returns it in the result envelope; - // every later phase relays that base back and spins it. We deliberately - // ignore any client-supplied `correlationVector` on provision so a lifecycle - // can never be seeded with a stale or foreign vector. - let is_provision = phase == "provision"; - // The relayed correlation vector for non-provision phases (the base seeded at - // provision). Captured before `dispatch` consumes `parsed`. `None` for - // provision (which seeds its own below). - let incoming_cv = if is_provision { - None - } else { - parsed.correlation_vector.clone() - }; - let resolved_backend = resolve_backend(&parsed).ok(); - let backend = resolved_backend - .as_ref() - .map(|b| b.wire_name()) - .unwrap_or("unknown"); - let telemetry_active = if experimental { - parsed - .request - .experimental - .telemetry - .as_ref() - .map(|c| telemetry::init(c, logger)) - .unwrap_or(false) - } else { - false - }; - - // Compute this phase's Microsoft Correlation Vector (MS-CV), executing the - // pure seed-vs-spin plan against the operators. Only computed when telemetry - // is active so an inactive provider does no work and provision output is - // unchanged. - let correlation = - compute_phase_correlation(telemetry_active, is_provision, incoming_cv.as_deref()); - - // Attribute out-of-band emit paths (the console-control handler installed in - // `main`, and the panic hook installed just below) to the resolved backend - // and the lifecycle phase, and install a crash-telemetry panic hook for this - // dispatch — mirroring the one-shot path, which this `-> !` entry point - // bypasses. The shared hook chains the previous hook (default stderr - // backtrace still prints) and is panic-free. - if telemetry_active { - if let Some(containment) = resolved_backend.as_ref() { - telemetry::set_process_context(containment); - } - telemetry::set_process_phase(phase); - // Stash this phase's correlation vector so out-of-band events - // (panic / cancellation) carry the same cV as the terminal emit below. - telemetry::set_process_correlation_vector(&correlation); - telemetry::install_panic_hook(); - } - - let started = Instant::now(); - let mut outcome = mxc_engine::run_state_aware(parsed, dry_run); - let elapsed = started.elapsed(); - - // For provision, return the freshly-seeded correlation vector to the client - // by injecting it into the result envelope so it can be relayed into later - // phases. Gated on telemetry so provision output is unchanged when telemetry - // is off. - if is_provision && telemetry_active { - inject_correlation_vector(&mut outcome, &correlation); - } - - // Emit lifecycle telemetry (and shut the provider down) before flushing the - // diagnostic buffer / envelope. Terminal path — safe to shutdown here. - telemetry::emit_state_aware( - telemetry_active, - telemetry::TelemetryContext { - backend, - phase, - correlation_vector: &correlation, - }, - &outcome, - elapsed, - ); + // Telemetry, correlation-vector, and panic-hook orchestration lives in + // `mxc_engine` so the `lxc` executor's entry point shares it verbatim — a + // Linux lifecycle is observed exactly like a Windows one, and the two + // cannot drift apart. + let outcome = mxc_engine::run_state_aware_with_telemetry(parsed, dry_run, experimental, logger); // On dispatch failure, route the error to the auxiliary diagnostic sinks // only (log file / diagnostic pipe) — never the primary buffer/stderr — so @@ -1771,98 +1621,6 @@ mod tests { ); } - #[test] - fn plan_correlation_vector_provision_seeds_fresh_vector() { - // Provision ignores any relayed value and always plans a fresh seed. - assert_eq!( - plan_correlation_vector(true, Some("BBBBBBBBBBBBBBBBBBBBBB.5")), - CvPlan::Seed - ); - assert_eq!(plan_correlation_vector(true, None), CvPlan::Seed); - } - - #[test] - fn plan_correlation_vector_phase_spins_relayed_base() { - // A relayable relayed value is spun to derive this phase's child vector. - let base = "AAAAAAAAAAAAAAAAAAAAAA.0"; - assert_eq!( - plan_correlation_vector(false, Some(base)), - CvPlan::Spin(base) - ); - // A valid frozen relayed value is also relayable (spin passes it through). - let frozen = "AAAAAAAAAAAAAAAAAAAAAA.0!"; - assert_eq!( - plan_correlation_vector(false, Some(frozen)), - CvPlan::Spin(frozen) - ); - } - - #[test] - fn plan_correlation_vector_phase_reseeds_for_missing_empty_or_malformed() { - // Missing / empty / non-relayable relayed values plan a fresh `Seed`, so - // the `Spin` arm never stands in for a reseed (garbage never reaches the - // `spin` operator). - for incoming in [None, Some(""), Some("garbage"), Some("short.0")] { - assert_eq!( - plan_correlation_vector(false, incoming), - CvPlan::Seed, - "non-relayable relay {incoming:?} must plan Seed" - ); - } - } - - #[test] - fn compute_phase_correlation_is_empty_when_telemetry_inactive() { - // Inactive telemetry does no RNG/clock work regardless of phase/relay. - assert!(compute_phase_correlation(false, true, None).is_empty()); - assert!( - compute_phase_correlation(false, false, Some("AAAAAAAAAAAAAAAAAAAAAA.0")).is_empty() - ); - } - - #[test] - fn compute_phase_correlation_spins_relayable_and_reseeds_garbage() { - // Active provision seeds a fresh valid vector. - let provisioned = compute_phase_correlation(true, true, None); - assert!(telemetry::correlation_vector::is_relayable(&provisioned)); - // Active non-provision spins a relayable relay onto the shared prefix. - let base = "AAAAAAAAAAAAAAAAAAAAAA.0"; - let spun = compute_phase_correlation(true, false, Some(base)); - assert!(spun.starts_with(&format!("{base}.")), "{spun:?}"); - // Active non-provision with garbage reseeds to a fresh, unrelated vector. - let reseeded = compute_phase_correlation(true, false, Some("garbage")); - assert!(telemetry::correlation_vector::is_relayable(&reseeded)); - assert!(!reseeded.starts_with("garbage")); - } - - #[test] - fn inject_correlation_vector_sets_field_on_envelope() { - let mut outcome: Result = Ok(DispatchOutcome::Envelope( - serde_json::json!({ "result": { "sandboxId": "iso:wxc-abc" } }), - )); - inject_correlation_vector(&mut outcome, "AAAAAAAAAAAAAAAAAAAAAA.0"); - match outcome { - Ok(DispatchOutcome::Envelope(v)) => assert_eq!( - v["result"]["correlationVector"], - serde_json::json!("AAAAAAAAAAAAAAAAAAAAAA.0") - ), - _ => panic!("expected envelope"), - } - } - - #[test] - fn inject_correlation_vector_noop_on_non_envelope() { - // Exec-completed / error outcomes carry no result envelope: injection is - // a no-op and must not panic. - let mut exit: Result = - Ok(DispatchOutcome::ExecCompleted { exit_code: 0 }); - inject_correlation_vector(&mut exit, "AAAAAAAAAAAAAAAAAAAAAA.0"); - assert!(matches!( - exit, - Ok(DispatchOutcome::ExecCompleted { exit_code: 0 }) - )); - } - #[test] fn finalize_maps_envelope_to_stdout_exit_zero() { let outcome: Result = diff --git a/src/core/wxc_common/src/config_contract_adapters/dev/one_shot.rs b/src/core/wxc_common/src/config_contract_adapters/dev/one_shot.rs index 8859ec18c..f2ccf83a0 100644 --- a/src/core/wxc_common/src/config_contract_adapters/dev/one_shot.rs +++ b/src/core/wxc_common/src/config_contract_adapters/dev/one_shot.rs @@ -248,6 +248,7 @@ fn convert_experimental(value: contract::OneShotExperimental) -> wire::Experimen windows_sandbox: windows_sandbox.into_option().map(convert_windows_sandbox), wslc: wslc.into_option().map(convert_wslc), isolation_session: None, + lxc: None, seatbelt: None, telemetry: telemetry.into_option().map(convert_telemetry), } diff --git a/src/core/wxc_common/src/config_contract_adapters/dev/one_shot_tests/experimental.rs b/src/core/wxc_common/src/config_contract_adapters/dev/one_shot_tests/experimental.rs index cc5d42df8..5e690f6b9 100644 --- a/src/core/wxc_common/src/config_contract_adapters/dev/one_shot_tests/experimental.rs +++ b/src/core/wxc_common/src/config_contract_adapters/dev/one_shot_tests/experimental.rs @@ -64,6 +64,22 @@ const WSLC_REQUEST_JSON: &str = r#"{ } }"#; +const LXC_PROVISION_REQUEST_JSON: &str = r#"{ + "version": "0.8.0-alpha", + "containment": "lxc", + "process": { + "commandLine": "echo hello" + }, + "experimental": { + "lxc": { + "provision": { + "distribution": "alpine", + "release": "3.23" + } + } + } +}"#; + #[test] fn windows_sandbox_maps_expected_wire_fields() { let wire = adapt(WINDOWS_SANDBOX_REQUEST_JSON); @@ -89,6 +105,7 @@ fn windows_sandbox_maps_expected_wire_fields() { assert!(experimental.test.is_none()); assert!(experimental.wslc.is_none()); assert!(experimental.isolation_session.is_none()); + assert!(experimental.lxc.is_none()); assert!(experimental.seatbelt.is_none()); assert!(experimental.telemetry.is_none()); } @@ -110,6 +127,7 @@ fn test_feature_and_telemetry_map_expected_wire_fields() { assert!(experimental.windows_sandbox.is_none()); assert!(experimental.wslc.is_none()); assert!(experimental.isolation_session.is_none()); + assert!(experimental.lxc.is_none()); assert!(experimental.seatbelt.is_none()); } @@ -156,10 +174,25 @@ fn wslc_maps_expected_wire_fields() { assert!(experimental.test.is_none()); assert!(experimental.windows_sandbox.is_none()); assert!(experimental.isolation_session.is_none()); + assert!(experimental.lxc.is_none()); assert!(experimental.seatbelt.is_none()); assert!(experimental.telemetry.is_none()); } +#[test] +fn lxc_is_rejected_as_a_one_shot_experimental_section() { + // A one-shot run never executes the provision phase this section configures, + // so accepting it would take a container image the caller asked for and + // silently drop it. + let error = serde_json::from_str::(LXC_PROVISION_REQUEST_JSON) + .expect_err("experimental.lxc must not be accepted on a one-shot request"); + + assert!( + error.to_string().contains("lxc"), + "the rejection should name the offending field, got: {error}" + ); +} + #[test] fn windows_sandbox_matches_current_wire_deserialization() { assert_matches_current_wire_deserialization(WINDOWS_SANDBOX_REQUEST_JSON); diff --git a/src/core/wxc_common/src/config_contract_adapters/dev/state_aware.rs b/src/core/wxc_common/src/config_contract_adapters/dev/state_aware.rs index 8ad394b64..65723afa4 100644 --- a/src/core/wxc_common/src/config_contract_adapters/dev/state_aware.rs +++ b/src/core/wxc_common/src/config_contract_adapters/dev/state_aware.rs @@ -54,6 +54,7 @@ fn convert_isolation_session_provision_experimental( isolation_session: isolation_session .into_option() .map(convert_state_aware_isolation_session), + lxc: None, seatbelt: None, telemetry: telemetry.into_option().map(convert_telemetry), } @@ -85,6 +86,7 @@ fn convert_windows_sandbox_provision_experimental( windows_sandbox: None, wslc: None, isolation_session: None, + lxc: None, seatbelt: None, telemetry: telemetry.into_option().map(convert_telemetry), } @@ -125,6 +127,7 @@ fn convert_wslc_provision_experimental( windows_sandbox: None, wslc: wslc.into_option().map(convert_state_aware_wslc), isolation_session: None, + lxc: None, seatbelt: None, telemetry: telemetry.into_option().map(convert_telemetry), } @@ -137,6 +140,7 @@ fn convert_start_experimental(value: contract::StartExperimental) -> wire::Exper windows_sandbox: None, wslc: None, isolation_session: None, + lxc: None, seatbelt: None, telemetry: telemetry.into_option().map(convert_telemetry), } @@ -149,6 +153,7 @@ fn convert_exec_experimental(value: contract::ExecExperimental) -> wire::Experim windows_sandbox: None, wslc: None, isolation_session: None, + lxc: None, seatbelt: None, telemetry: telemetry.into_option().map(convert_telemetry), } @@ -161,6 +166,7 @@ fn convert_stop_experimental(value: contract::StopExperimental) -> wire::Experim windows_sandbox: None, wslc: None, isolation_session: None, + lxc: None, seatbelt: None, telemetry: telemetry.into_option().map(convert_telemetry), } @@ -175,6 +181,7 @@ fn convert_deprovision_experimental( windows_sandbox: None, wslc: None, isolation_session: None, + lxc: None, seatbelt: None, telemetry: telemetry.into_option().map(convert_telemetry), } diff --git a/src/core/wxc_common/src/config_parser.rs b/src/core/wxc_common/src/config_parser.rs index 517aa74f0..5dc5ddbce 100644 --- a/src/core/wxc_common/src/config_parser.rs +++ b/src/core/wxc_common/src/config_parser.rs @@ -336,7 +336,8 @@ const CURRENT_SCHEMA_VERSION: &str = "0.8.0-alpha"; /// experimental backend sections that don't match the selected /// `containment`. Add a new entry when promoting a backend to a top-level /// section or graduating one from experimental. -const KNOWN_EXPERIMENTAL_BACKENDS: &[&str] = &["windows_sandbox", "wslc", "isolation_session"]; +const KNOWN_EXPERIMENTAL_BACKENDS: &[&str] = + &["windows_sandbox", "wslc", "isolation_session", "lxc"]; /// Validate that the schema version (semver) is supported by this binary. /// Compares major.minor only — patch and pre-release labels are ignored. @@ -568,9 +569,12 @@ fn validate_experimental_backend_keys( return Ok(()); }; - let matching_key = containment - .and_then(|c| c.section_path()) - .and_then(|path| path.strip_prefix("experimental.")); + let matching_key = match containment { + Some(ContainmentBackend::Lxc) => Some("lxc"), + _ => containment + .and_then(|c| c.section_path()) + .and_then(|path| path.strip_prefix("experimental.")), + }; let present: Vec<&'static str> = KNOWN_EXPERIMENTAL_BACKENDS .iter() @@ -916,7 +920,7 @@ fn convert_wire_config( process_container_network = ac.network; } - // Filesystem section + // Filesystem section. if let Some(fscfg) = cfg.filesystem { if let Some(v) = fscfg.denied_paths { policy.denied_paths = v; @@ -1152,30 +1156,6 @@ fn convert_wire_config( return Err(WxcError::ConfigParse(msg.to_string())); } - // LXC is the inverse of the two guards above: it *does* have a - // privileged packet-filter layer, and that layer is the only thing that - // makes the proxy an exception rather than a suggestion. Under the - // default `Capabilities` mode `apply_firewall_rules` installs nothing, - // so the runner would inject HTTP(S)_PROXY while leaving direct egress - // wide open -- a config that reads as deny-all-except-proxy and - // enforces neither half. Reject it rather than auto-promoting, so the - // user's stated enforcement is never silently rewritten. - if containment == ContainmentBackend::Lxc - && policy.network_proxy.is_enabled() - && !matches!( - policy.network_enforcement_mode, - NetworkEnforcementMode::Firewall | NetworkEnforcementMode::Both - ) - { - let msg = "LXC: network.proxy requires network.enforcementMode='firewall' \ - or 'both'. Under the default 'capabilities' mode no iptables \ - rules are installed, so the proxy environment variables would be \ - injected while direct egress stayed unrestricted -- any client \ - that ignores HTTP_PROXY would bypass the proxy entirely."; - logger.log_line(msg); - return Err(WxcError::ConfigParse(msg.to_string())); - } - // A proxy URL may carry `user:pass@` userinfo, and neither LXC nor // Bubblewrap keeps that value out of process argv: LXC turns each env // entry into an `lxc-attach --set-var=KEY=VALUE` argument, and @@ -3715,10 +3695,10 @@ mod tests { #[test] fn proxy_accepted_with_lxc() { // LXC requires a routable proxy host: localhost/127.0.0.1 is the - // container loopback and unreachable, so use network.proxy.url. - // A firewall mode is required, because that is what makes the proxy an - // exception to deny-all rather than an unenforced suggestion. - let json = r#"{"process":{"commandLine":"x"},"containment":"lxc","network":{"proxy":{"url":"http://proxy.example.com:8080"},"enforcementMode":"firewall"}}"#; + // container loopback and unreachable, so use network.proxy.url. A proxy + // makes the policy require the firewall, so LXC installs the rules that + // make it an exception to deny-all; no enforcementMode is needed. + let json = r#"{"process":{"commandLine":"x"},"containment":"lxc","network":{"proxy":{"url":"http://proxy.example.com:8080"}}}"#; let encoded = base64_encode(json.as_bytes()); let mut logger = test_logger(); @@ -3730,48 +3710,32 @@ mod tests { } #[test] - fn proxy_with_lxc_accepts_both_mode() { - // 'both' also installs the iptables rules, so it satisfies the guard. - let json = r#"{"process":{"commandLine":"x"},"containment":"lxc","network":{"proxy":{"url":"http://proxy.example.com:8080"},"enforcementMode":"both"}}"#; - let encoded = base64_encode(json.as_bytes()); - let mut logger = test_logger(); - - let req = load_request(&encoded, &mut logger, true).unwrap(); - assert!(req.policy.network_proxy.is_enabled()); - } - - #[test] - fn proxy_with_lxc_and_omitted_enforcement_mode_is_rejected() { - // enforcementMode defaults to 'capabilities', under which - // apply_firewall_rules installs nothing. Accepting this config would - // inject HTTP(S)_PROXY while leaving direct egress unrestricted, so - // anything ignoring the environment variables bypasses the proxy. - let json = r#"{"process":{"commandLine":"x"},"containment":"lxc","network":{"proxy":{"url":"http://proxy.example.com:8080"}}}"#; - let encoded = base64_encode(json.as_bytes()); - let mut logger = test_logger(); - - let err = load_request(&encoded, &mut logger, true).unwrap_err(); - assert!( - format!("{}", err).contains("network.proxy requires network.enforcementMode"), - "expected the LXC enforcement-mode rejection, got: {}", - err - ); - } - - #[test] - fn proxy_with_lxc_and_explicit_capabilities_mode_is_rejected() { - // Stating 'capabilities' explicitly is the same fail-open as omitting - // it, so it must be rejected identically rather than read as consent. - let json = r#"{"process":{"commandLine":"x"},"containment":"lxc","network":{"proxy":{"url":"http://proxy.example.com:8080"},"enforcementMode":"capabilities"}}"#; - let encoded = base64_encode(json.as_bytes()); - let mut logger = test_logger(); + fn proxy_with_lxc_is_accepted_whatever_the_enforcement_mode() { + // The behavior change, seen through the parser: an LXC proxy used to be + // rejected unless enforcementMode was firewall/both. It is now accepted + // regardless -- omitted, capabilities, firewall, and both all parse to + // an enabled proxy, because the proxy alone drives the install. + for mode in [ + r#""enforcementMode":"capabilities","#, + r#""enforcementMode":"firewall","#, + r#""enforcementMode":"both","#, + "", + ] { + let json = format!( + r#"{{"process":{{"commandLine":"x"}},"containment":"lxc","network":{{{}"proxy":{{"url":"http://proxy.example.com:8080"}}}}}}"#, + mode + ); + let encoded = base64_encode(json.as_bytes()); + let mut logger = test_logger(); - let err = load_request(&encoded, &mut logger, true).unwrap_err(); - assert!( - format!("{}", err).contains("network.proxy requires network.enforcementMode"), - "expected the LXC enforcement-mode rejection, got: {}", - err - ); + let req = load_request(&encoded, &mut logger, true).unwrap_or_else(|err| { + panic!("an LXC proxy must be accepted (mode fragment {mode:?}), got: {err}") + }); + assert!( + req.policy.network_proxy.is_enabled(), + "mode fragment {mode:?}: the proxy must be enabled" + ); + } } // The credential guard runs after `convert_wire_proxy`, so a @@ -6453,6 +6417,33 @@ mod tests { ); } + #[test] + fn state_aware_lxc_experimental_backend_key_is_accepted() { + let json = r#"{ + "phase": "provision", + "containment": "lxc", + "experimental": { + "lxc": {"provision": {"distribution": "alpine", "release": "3.20"}} + } + }"#; + let encoded = base64_encode(json.as_bytes()); + let mut logger = test_logger(); + let req = load_mxc_request(&encoded, &mut logger, true) + .expect("state-aware lxc config should parse"); + match req { + MxcRequest::StateAware(p) => { + assert_eq!(p.phase, Phase::Provision); + assert_eq!(p.containment, Some(ContainmentBackend::Lxc)); + assert!(p + .experimental_raw + .as_ref() + .and_then(|v| v.get("lxc")) + .is_some()); + } + other => panic!("expected state-aware request, got {other:?}"), + } + } + // ---- Abstract-intent coverage ---- // Backend sections paired with `containment: "process"` / "vm" must be // accepted iff the intent resolves to the owning backend on this OS. diff --git a/src/core/wxc_common/src/logger.rs b/src/core/wxc_common/src/logger.rs index 69005038d..18f6546b8 100644 --- a/src/core/wxc_common/src/logger.rs +++ b/src/core/wxc_common/src/logger.rs @@ -278,6 +278,19 @@ impl Logger { &self.buffer } + /// Switch to console logging, handing back whatever was buffered first. + /// + /// The mode has to be chosen before the request is parsed, but only the + /// parsed request says whether stdout is a free-form debug stream (one-shot) + /// or a strict JSON channel (state-aware). Starting buffered and promoting + /// once the answer is known keeps a debug run's diagnostics off stdout until + /// it is known to be safe to put them there; the returned buffer is what the + /// caller must emit so nothing logged before the switch is lost. + pub fn promote_to_console(&mut self) -> String { + self.mode = Mode::Console; + std::mem::take(&mut self.buffer) + } + // ----------------------------------------------------------------------- // Diagnostic sink internals // ----------------------------------------------------------------------- @@ -346,6 +359,24 @@ impl fmt::Write for Logger { mod tests { use super::*; + #[test] + fn promotion_to_console_hands_back_everything_buffered_first() { + // The mode is chosen before the request is parsed, so a debug run + // buffers until the phase is known. Whatever accumulated in the + // meantime is the caller's to emit -- dropping it would lose the + // parser's warnings, and leaving it would print them twice. + let mut logger = Logger::new(Mode::Buffer); + logger.log_line("a parser warning"); + + let carried = logger.promote_to_console(); + + assert_eq!(carried, "a parser warning\n"); + assert!( + logger.get_buffer().is_empty(), + "the buffer must be emptied so the switch cannot duplicate output" + ); + } + #[test] fn security_warnings_are_retained_outside_the_debug_buffer() { let mut logger = Logger::new(Mode::Buffer); diff --git a/src/core/wxc_common/src/models.rs b/src/core/wxc_common/src/models.rs index 02465f660..7d0a2746d 100644 --- a/src/core/wxc_common/src/models.rs +++ b/src/core/wxc_common/src/models.rs @@ -792,6 +792,13 @@ impl ContainerPolicy { &self.blocked_hosts, ) } + + pub fn requires_firewall(&self) -> bool { + self.default_network_policy == NetworkPolicy::Block + || !self.allowed_hosts.is_empty() + || !self.blocked_hosts.is_empty() + || self.network_proxy.is_enabled() + } } /// Windows denial-capture settings (from `processContainer.captureDenials`). diff --git a/src/core/wxc_common/src/state_aware_dispatch.rs b/src/core/wxc_common/src/state_aware_dispatch.rs index 521fa9202..605aa6c01 100644 --- a/src/core/wxc_common/src/state_aware_dispatch.rs +++ b/src/core/wxc_common/src/state_aware_dispatch.rs @@ -186,6 +186,7 @@ pub fn resolve_backend(parsed: &ParsedStateAwareRequest) -> Result Result { match prefix { "iso" => Ok(ContainmentBackend::IsolationSession), + "lxc" => Ok(ContainmentBackend::Lxc), "wsb" => Ok(ContainmentBackend::WindowsSandbox), "wslc" => Ok(ContainmentBackend::Wslc), // Future state-aware backends extend this list. @@ -1105,6 +1106,20 @@ mod tests { ); } + #[test] + fn resolve_backend_for_lxc_prefix_returns_lxc() { + let p = ParsedStateAwareRequest { + request: ExecutionRequest::default(), + phase: Phase::Start, + containment: None, + sandbox_id: Some("lxc:mxc-abcd1234".into()), + correlation_vector: None, + experimental_raw: None, + source_text: None, + }; + assert_eq!(resolve_backend(&p).unwrap(), ContainmentBackend::Lxc); + } + #[test] fn resolve_backend_for_wsb_prefix_returns_windows_sandbox() { let p = ParsedStateAwareRequest { diff --git a/src/core/wxc_common/src/unix_proxy_coordinator.rs b/src/core/wxc_common/src/unix_proxy_coordinator.rs index bf840db49..ef368c415 100644 --- a/src/core/wxc_common/src/unix_proxy_coordinator.rs +++ b/src/core/wxc_common/src/unix_proxy_coordinator.rs @@ -368,8 +368,8 @@ impl UnixProxyCoordinator { impl Drop for UnixProxyCoordinator { /// Silent best-effort cleanup if the coordinator is still active at /// drop time. **Never** writes to stderr or `Logger` because the drop - /// may run during panic unwinding and we must not corrupt the JSON - /// envelope on `lxc-exec`'s stderr. + /// may run during panic unwinding and we must not corrupt `lxc-exec`'s + /// JSON envelope. fn drop(&mut self) { if let Some(mut tp) = self.test_proxy.take() { let pid = tp.child.id(); diff --git a/src/core/wxc_common/src/wire.rs b/src/core/wxc_common/src/wire.rs index 64dd12c78..ce2b10e9d 100644 --- a/src/core/wxc_common/src/wire.rs +++ b/src/core/wxc_common/src/wire.rs @@ -586,6 +586,8 @@ pub struct Experimental { pub wslc: Option, /// IsolationSession backend config (Windows). pub isolation_session: Option, + /// LXC backend config (Linux). + pub lxc: Option, /// Seatbelt backend config (pre-promotion alias). #[serde(alias = "macos_sandbox")] pub seatbelt: Option, @@ -735,6 +737,36 @@ pub struct IsolationSessionProvisionPhase { pub app_id: Option, } +/// LXC backend config under the experimental surface. Carries only the +/// per-phase state-aware nesting for the phases that take config +/// (`provision`); the one-shot LXC surface is the stable top-level `lxc` +/// section, so this type is named apart from it rather than shared with it. +/// `start`, `exec`, `stop`, and `deprovision` take no per-phase config payload. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[cfg_attr(feature = "schema-gen", derive(schemars::JsonSchema))] +#[serde(rename_all = "camelCase")] +pub struct LxcExperimental { + /// State-aware provision-phase configuration. + pub provision: Option, +} + +/// Provision-phase LXC configuration (state-aware lifecycle), nested under +/// `experimental.lxc.provision`. Names the container image to create. +/// +/// Filesystem mounts and network policy derive from the top-level `filesystem` +/// and `network` sections, not from here. It is its own type rather than a +/// shared one because a shared type would advertise its fields on every phase +/// in the generated schema. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[cfg_attr(feature = "schema-gen", derive(schemars::JsonSchema))] +#[serde(rename_all = "camelCase")] +pub struct LxcProvisionPhase { + /// Distribution image (e.g. `alpine`). + pub distribution: Option, + /// Distribution release (e.g. `3.23`). + pub release: Option, +} + /// JSON Schema generation from the wire model, gated behind `schema-gen` so /// production builds don't carry `schemars`. The single public entry point is /// re-exported below as `generate_config_schema_json`. diff --git a/tests/configs/lxc_state_aware_provision.json b/tests/configs/lxc_state_aware_provision.json new file mode 100644 index 000000000..aa9ea0b48 --- /dev/null +++ b/tests/configs/lxc_state_aware_provision.json @@ -0,0 +1,9 @@ +{ + "phase": "provision", + "containment": "lxc", + "experimental": { + "lxc": { + "provision": { "distribution": "alpine", "release": "3.23" } + } + } +} diff --git a/tests/configs/lxc_state_aware_provision_conflicting_backend_rejected.json b/tests/configs/lxc_state_aware_provision_conflicting_backend_rejected.json new file mode 100644 index 000000000..8379cd0a3 --- /dev/null +++ b/tests/configs/lxc_state_aware_provision_conflicting_backend_rejected.json @@ -0,0 +1,12 @@ +{ + "phase": "provision", + "containment": "lxc", + "experimental": { + "lxc": { + "provision": { "distribution": "alpine", "release": "3.23" } + }, + "wslc": { + "provision": { "image": "ubuntu-24.04" } + } + } +} diff --git a/tests/configs/lxc_state_aware_provision_missing_fields_rejected.json b/tests/configs/lxc_state_aware_provision_missing_fields_rejected.json new file mode 100644 index 000000000..c299e8253 --- /dev/null +++ b/tests/configs/lxc_state_aware_provision_missing_fields_rejected.json @@ -0,0 +1,9 @@ +{ + "phase": "provision", + "containment": "lxc", + "experimental": { + "lxc": { + "provision": { "distribution": "alpine" } + } + } +} diff --git a/tests/configs/lxc_state_aware_start_allow_local_network.json b/tests/configs/lxc_state_aware_start_allow_local_network.json new file mode 100644 index 000000000..be21f8c4f --- /dev/null +++ b/tests/configs/lxc_state_aware_start_allow_local_network.json @@ -0,0 +1,8 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "allowLocalNetwork": true, + "enforcementMode": "firewall" + } +} diff --git a/tests/configs/lxc_state_aware_start_allowed_hosts_firewall.json b/tests/configs/lxc_state_aware_start_allowed_hosts_firewall.json new file mode 100644 index 000000000..6c8d46c58 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_allowed_hosts_firewall.json @@ -0,0 +1,8 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "allowedHosts": ["example.com"], + "enforcementMode": "firewall" + } +} diff --git a/tests/configs/lxc_state_aware_start_blocked_hosts_capabilities.json b/tests/configs/lxc_state_aware_start_blocked_hosts_capabilities.json new file mode 100644 index 000000000..37807914a --- /dev/null +++ b/tests/configs/lxc_state_aware_start_blocked_hosts_capabilities.json @@ -0,0 +1,7 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "blockedHosts": ["evil.example.com"] + } +} diff --git a/tests/configs/lxc_state_aware_start_default_allow_capabilities.json b/tests/configs/lxc_state_aware_start_default_allow_capabilities.json new file mode 100644 index 000000000..1ce23103b --- /dev/null +++ b/tests/configs/lxc_state_aware_start_default_allow_capabilities.json @@ -0,0 +1,7 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "defaultPolicy": "allow" + } +} diff --git a/tests/configs/lxc_state_aware_start_default_block_both.json b/tests/configs/lxc_state_aware_start_default_block_both.json new file mode 100644 index 000000000..fb0e8baf8 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_default_block_both.json @@ -0,0 +1,8 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "defaultPolicy": "block", + "enforcementMode": "both" + } +} diff --git a/tests/configs/lxc_state_aware_start_default_block_capabilities.json b/tests/configs/lxc_state_aware_start_default_block_capabilities.json new file mode 100644 index 000000000..3ec405799 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_default_block_capabilities.json @@ -0,0 +1,7 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "defaultPolicy": "block" + } +} diff --git a/tests/configs/lxc_state_aware_start_default_block_firewall.json b/tests/configs/lxc_state_aware_start_default_block_firewall.json new file mode 100644 index 000000000..6e220838d --- /dev/null +++ b/tests/configs/lxc_state_aware_start_default_block_firewall.json @@ -0,0 +1,8 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "defaultPolicy": "block", + "enforcementMode": "firewall" + } +} diff --git a/tests/configs/lxc_state_aware_start_empty_allowed_hosts.json b/tests/configs/lxc_state_aware_start_empty_allowed_hosts.json new file mode 100644 index 000000000..f6b8212c7 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_empty_allowed_hosts.json @@ -0,0 +1,7 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "allowedHosts": [] + } +} diff --git a/tests/configs/lxc_state_aware_start_filesystem_at_provision_rejected.json b/tests/configs/lxc_state_aware_start_filesystem_at_provision_rejected.json new file mode 100644 index 000000000..3f64e87e3 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_filesystem_at_provision_rejected.json @@ -0,0 +1,17 @@ +{ + "phase": "provision", + "containment": "lxc", + "filesystem": { + "readonlyPaths": [ + "/tmp/mxc_lxc_sa_ro" + ] + }, + "experimental": { + "lxc": { + "provision": { + "distribution": "alpine", + "release": "3.23" + } + } + } +} diff --git a/tests/configs/lxc_state_aware_start_filesystem_paths.json b/tests/configs/lxc_state_aware_start_filesystem_paths.json new file mode 100644 index 000000000..c9b890716 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_filesystem_paths.json @@ -0,0 +1,8 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "filesystem": { + "readonlyPaths": ["/mxc-e2e-ro"], + "deniedPaths": ["/mxc-e2e-denied"] + } +} diff --git a/tests/configs/lxc_state_aware_start_network_at_provision_rejected.json b/tests/configs/lxc_state_aware_start_network_at_provision_rejected.json new file mode 100644 index 000000000..a4efb7e26 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_network_at_provision_rejected.json @@ -0,0 +1,15 @@ +{ + "phase": "provision", + "containment": "lxc", + "network": { + "defaultPolicy": "block" + }, + "experimental": { + "lxc": { + "provision": { + "distribution": "alpine", + "release": "3.23" + } + } + } +} diff --git a/tests/configs/lxc_state_aware_start_no_network.json b/tests/configs/lxc_state_aware_start_no_network.json new file mode 100644 index 000000000..cbbdcea8a --- /dev/null +++ b/tests/configs/lxc_state_aware_start_no_network.json @@ -0,0 +1,4 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__" +} diff --git a/tests/configs/lxc_state_aware_start_nonempty_allowed_hosts_capabilities.json b/tests/configs/lxc_state_aware_start_nonempty_allowed_hosts_capabilities.json new file mode 100644 index 000000000..3ba36a773 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_nonempty_allowed_hosts_capabilities.json @@ -0,0 +1,7 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "allowedHosts": ["example.com"] + } +} diff --git a/tests/configs/lxc_state_aware_start_proxy.json b/tests/configs/lxc_state_aware_start_proxy.json new file mode 100644 index 000000000..35d121a92 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_proxy.json @@ -0,0 +1,7 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "proxy": { "builtinTestServer": true } + } +} \ No newline at end of file diff --git a/tests/configs/lxc_state_aware_start_proxy_external.json b/tests/configs/lxc_state_aware_start_proxy_external.json new file mode 100644 index 000000000..dc0a087d4 --- /dev/null +++ b/tests/configs/lxc_state_aware_start_proxy_external.json @@ -0,0 +1,7 @@ +{ + "phase": "start", + "sandboxId": "__SANDBOX_ID__", + "network": { + "proxy": { "url": "http://127.0.0.1:3128" } + } +} \ No newline at end of file diff --git a/tests/scripts/run_lxc_all_tests.sh b/tests/scripts/run_lxc_all_tests.sh index dfdd756bf..da6a9aa71 100644 --- a/tests/scripts/run_lxc_all_tests.sh +++ b/tests/scripts/run_lxc_all_tests.sh @@ -77,6 +77,10 @@ run_test "LXC Network Bridge Fail-Closed" "$SCRIPT_DIR/run_lxc_network_bridge_fa run_test "LXC Inbound Default-Deny" "$SCRIPT_DIR/run_lxc_inbound_deny_test.sh" run_test "LXC Timeout" "$SCRIPT_DIR/run_lxc_timeout_test.sh" run_test "LXC Env+Cwd" "$SCRIPT_DIR/run_lxc_env_cwd_test.sh" +run_test "LXC State-Aware Lifecycle" "$SCRIPT_DIR/run_lxc_state_aware_test.sh" +run_test "LXC State-Aware Network Matrix" "$SCRIPT_DIR/run_lxc_state_aware_network_test.sh" +run_test "LXC Conflicting Backends" "$SCRIPT_DIR/run_lxc_conflicting_backends_test.sh" +run_test "LXC Experimental Provision Fields" "$SCRIPT_DIR/run_lxc_experimental_provision_fields_test.sh" echo "================================" echo "Results: $PASSED passed, $FAILED failed, $SKIPPED skipped" diff --git a/tests/scripts/run_lxc_conflicting_backends_test.sh b/tests/scripts/run_lxc_conflicting_backends_test.sh new file mode 100644 index 000000000..40c1e0e68 --- /dev/null +++ b/tests/scripts/run_lxc_conflicting_backends_test.sh @@ -0,0 +1,148 @@ +#!/bin/bash +# LXC conflicting containment backends test. +# +# Proves the single-backend-section rule for LXC: a request that names +# containment "lxc" but carries a second experimental backend section is +# rejected, rather than silently honoring one section and ignoring the other. +# +# The rule is documented in docs/versioning.md ("a backend section requires +# `containment` to be set, and the value must be either the concrete backend +# name or any abstract intent that resolves to it") and is enforced by the +# parser, not by the JSON schema -- the generated schema intentionally omits +# the cross-field clauses. So this is only observable by running the binary, +# which is why it is an E2E test and not a schema fixture. +# +# The fixture deliberately carries a *complete and individually valid* +# experimental.lxc section alongside the foreign one. A fixture with only the +# foreign section would be rejected too, but for the weaker reason that no +# matching section was found; requiring both proves the request is refused +# because two backends were named, which is the scenario under test. +# +# Unlike every other LXC E2E test, this one needs no root, no LXC runtime and +# no iptables: the request is refused during config parsing, before any +# privileged work. Only a missing binary can skip it. Note that +# run_lxc_all_tests.sh still requires root before it dispatches anything, so +# under the suite this runs on the same hosts as everything else; the property +# matters when the script is invoked directly. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(dirname "$(dirname "$SCRIPT_DIR")")" +CONFIG="$REPO_DIR/tests/configs/lxc_state_aware_provision_conflicting_backend_rejected.json" +VALID_CONFIG="$REPO_DIR/tests/configs/lxc_state_aware_provision.json" + +LXC_EXEC="$REPO_DIR/src/target/release/lxc-exec" +if [ ! -f "$LXC_EXEC" ]; then + LXC_EXEC="$REPO_DIR/src/target/debug/lxc-exec" +fi + +# An honest skip for a missing prerequisite: exit 77 so run_lxc_all_tests.sh +# records SKIPPED rather than PASS. A suite that could not run must not look +# green. +SKIP_EXIT=77 +skip() { + echo "SKIP: $1" + exit "$SKIP_EXIT" +} + +fail() { + echo "FAIL: $1" + exit 1 +} + +[ -f "$LXC_EXEC" ] || skip "lxc-exec binary not built; run build.sh first." +[ -f "$CONFIG" ] || fail "fixture is missing: $CONFIG" +[ -f "$VALID_CONFIG" ] || fail "control fixture is missing: $VALID_CONFIG" + +# The positive control provisions a real container, so this test owns its +# removal on every exit path rather than leaving it for the runner or the next +# test to trip over. +CONTROL_CONTAINER="" +cleanup() { + if [ -n "$CONTROL_CONTAINER" ]; then + lxc-destroy -n "$CONTROL_CONTAINER" -f >/dev/null 2>&1 + fi + return 0 +} +trap cleanup EXIT +trap 'cleanup; exit 130' INT +trap 'cleanup; exit 143' TERM + +# Asserting on the message text makes it part of the observable contract. It is +# quoted here exactly as the binary emits it; a reword is a breaking change for +# anyone matching on it and must fail this test rather than pass silently. +EXPECTED_CODE='"code":"malformed_request"' +EXPECTED_MESSAGE='Multiple containment backends configured' +EXPECTED_SECTION='experimental.wslc' +EXPECTED_REMEDY='Only one backend section is allowed' + +# Drift guard: the fixture and these assertions rot apart the moment someone +# edits one of them. jq and python3 are not guaranteed on an LXC test host, so +# these are plain text checks against the fixture. The backend-section checks +# are scoped to the text from "experimental" onward, because an unscoped +# grep for '"lxc"' is also satisfied by the `"containment": "lxc"` line and +# would pass even if the experimental.lxc section were deleted outright. +experimental_block() { + sed -n '/"experimental"/,$p' "$1" +} + +grep -Fq '"containment": "lxc"' "$CONFIG" \ + || fail "fixture no longer selects lxc containment; the test would prove nothing." +experimental_block "$CONFIG" | grep -Fq '"lxc"' \ + || fail "fixture no longer carries an experimental.lxc section." +experimental_block "$CONFIG" | grep -Fq '"wslc"' \ + || fail "fixture no longer carries the conflicting experimental.wslc section." + +echo "Running LXC conflicting containment backends test..." + +# The error envelope goes to stdout and diagnostics to stderr, so stdout is +# captured on its own -- asserting against a merged stream would let a stderr +# diagnostic satisfy an assertion about the envelope. +set +e +OUTPUT="$("$LXC_EXEC" "$CONFIG" 2>/dev/null)" +RC=$? +set -e +echo "$OUTPUT" + +[ "$RC" -ne 0 ] || fail "conflicting backend sections were accepted (exit 0); the request should be refused." + +echo "$OUTPUT" | grep -Fq "$EXPECTED_CODE" \ + || fail "rejection did not carry $EXPECTED_CODE." +echo "$OUTPUT" | grep -Fq "$EXPECTED_MESSAGE" \ + || fail "rejection did not explain that multiple backends were configured." +echo "$OUTPUT" | grep -Fq "$EXPECTED_SECTION" \ + || fail "rejection did not name the conflicting section ($EXPECTED_SECTION)." +echo "$OUTPUT" | grep -Fq "$EXPECTED_REMEDY" \ + || fail "rejection did not tell the caller how to fix it." + +# Positive control: the same binary and the same phase must accept a request +# that names exactly one backend. Without this, an lxc-exec that rejected every +# config would pass every assertion above while verifying nothing. The control +# is allowed to fail for an environmental reason -- no LXC runtime on the host +# -- which is not what is under test; it must not fail as a malformed request. +set +e +CONTROL_OUTPUT="$("$LXC_EXEC" "$VALID_CONFIG" 2>/dev/null)" +CONTROL_RC=$? +set -e +echo "$CONTROL_OUTPUT" + +# Recorded from the control's own result envelope so the trap can remove it. +CONTROL_CONTAINER="$(printf '%s' "$CONTROL_OUTPUT" | sed -n 's/.*"containerName":"\([^"]*\)".*/\1/p')" + +if echo "$CONTROL_OUTPUT" | grep -Fq "$EXPECTED_MESSAGE"; then + fail "the single-backend control config was also rejected as multi-backend; the assertions above do not discriminate." +fi +if echo "$CONTROL_OUTPUT" | grep -Fq "$EXPECTED_SECTION"; then + fail "the single-backend control config named $EXPECTED_SECTION as a conflict; the assertions above do not discriminate." +fi +if echo "$CONTROL_OUTPUT" | grep -Fq "$EXPECTED_CODE"; then + fail "the single-backend control config was rejected as $EXPECTED_CODE; a well-formed request must get past parsing." +fi +if [ "$CONTROL_RC" -eq 0 ]; then + echo "Control accepted (exit 0)." +else + echo "NOTE: control exited $CONTROL_RC without a parse rejection, which is an environmental failure and not what this test covers." +fi + +echo "PASS: a request naming two containment backends is refused, naming the conflicting section." +echo "LXC conflicting containment backends test complete." diff --git a/tests/scripts/run_lxc_experimental_provision_fields_test.sh b/tests/scripts/run_lxc_experimental_provision_fields_test.sh new file mode 100644 index 000000000..21fae73f4 --- /dev/null +++ b/tests/scripts/run_lxc_experimental_provision_fields_test.sh @@ -0,0 +1,191 @@ +#!/bin/bash +# LXC experimental.lxc.provision field contract test. +# +# Covers the two fields the provision phase reads out of its own sub-object in +# the config, experimental.lxc.provision.distribution and .release: +# +# cause effect +# -------------------------------------- --------------------------------- +# provision names only one of the two refused, naming both as required +# distribution is not a string refused, naming the exact path +# both present and well typed not refused for either reason +# +# Scope note: this pins the *runtime* contract. Both fields are optional in the +# wire model, as they are in the stable top-level lxc section, so the schema +# accepts a provision section that omits them and the backend is what refuses +# it -- the same layer every other state-aware backend rejects provision config +# at. The non-string case is written to a temp file rather than committed under +# tests/configs, because a committed fixture that fails schema validation would +# have to be registered in scripts/versioning/config-validation-exemptions.json, +# and no other backend has an entry there. +# +# Like the conflicting-backends test, every assertion here lands during config +# parsing, so the script itself needs no root, no LXC runtime and no iptables. +# run_lxc_all_tests.sh still requires root before it dispatches anything, so +# that property matters when the script is invoked directly. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(dirname "$(dirname "$SCRIPT_DIR")")" +CONFIG_DIR="$REPO_DIR/tests/configs" + +MISSING_FIELD_CONFIG="$CONFIG_DIR/lxc_state_aware_provision_missing_fields_rejected.json" +VALID_CONFIG="$CONFIG_DIR/lxc_state_aware_provision.json" + +LXC_EXEC="$REPO_DIR/src/target/release/lxc-exec" +if [ ! -f "$LXC_EXEC" ]; then + LXC_EXEC="$REPO_DIR/src/target/debug/lxc-exec" +fi + +# An honest skip for a missing prerequisite: exit 77 so run_lxc_all_tests.sh +# records SKIPPED rather than PASS. +SKIP_EXIT=77 +skip() { + echo "SKIP: $1" + exit "$SKIP_EXIT" +} + +PASSED=0 +FAILED=0 + +fail() { + echo "FAIL: $1" + FAILED=$((FAILED + 1)) +} + +pass() { + echo "PASS: $1" + PASSED=$((PASSED + 1)) +} + +[ -f "$LXC_EXEC" ] || skip "lxc-exec binary not built; run build.sh first." +for c in "$MISSING_FIELD_CONFIG" "$VALID_CONFIG"; do + [ -f "$c" ] || { echo "FAIL: fixture is missing: $c"; exit 1; } +done + +# Written here rather than committed: see the scope note above. +INVALID_TYPE_CONFIG="$(mktemp)" +cat > "$INVALID_TYPE_CONFIG" <<'JSON' +{ + "phase": "provision", + "containment": "lxc", + "experimental": { + "lxc": { + "provision": { "distribution": 123, "release": "3.23" } + } + } +} +JSON + +# The positive control provisions a real container, so this test owns its +# removal on every exit path rather than leaving it for the runner or the next +# test to trip over. +CONTROL_CONTAINER="" +cleanup() { + if [ -n "$CONTROL_CONTAINER" ]; then + lxc-destroy -n "$CONTROL_CONTAINER" -f >/dev/null 2>&1 + fi + rm -f "$INVALID_TYPE_CONFIG" + return 0 +} +trap cleanup EXIT +trap 'cleanup; exit 130' INT +trap 'cleanup; exit 143' TERM + +# Quoted exactly as the binary emits them, which makes them part of the +# observable contract; a reword must fail here rather than pass silently. +REQUIRED_MESSAGE='LXC distribution and release are required' +INVALID_TYPE_PATH='experimental.lxc.provision.distribution' +INVALID_TYPE_DETAIL='expected a string' + +# Drift guard: these fixtures only prove what they claim if they still carry +# the shape the assertions assume. jq and python3 are not guaranteed on an LXC +# test host, so these are plain text checks. +grep -Fq '"distribution"' "$MISSING_FIELD_CONFIG" \ + || fail "missing-field fixture no longer supplies distribution; a rejection would no longer prove that the *missing* field is what was caught." +grep -Fq '"release"' "$MISSING_FIELD_CONFIG" \ + && fail "missing-field fixture now supplies release; it no longer tests a missing field." +grep -Fq '"distribution": 123' "$INVALID_TYPE_CONFIG" \ + || fail "the generated invalid-type config no longer carries a non-string distribution." +grep -Fq '"distribution": "alpine"' "$VALID_CONFIG" \ + || fail "control fixture no longer carries a well-typed distribution." + +# Run $1, leaving its stdout in OUT and its exit status in RUN_RC. The error +# envelope goes to stdout and diagnostics to stderr, so stdout is captured on +# its own. OUT is assigned here rather than echoed because a caller using +# command substitution would run this in a subshell and discard RUN_RC, which +# would let a rejection message printed alongside exit 0 satisfy the +# assertions below. +OUT="" +RUN_RC=0 +run_config() { + set +e + OUT="$("$LXC_EXEC" "$1" 2>/dev/null)" + RUN_RC=$? + set -e +} + +echo "Running LXC experimental.lxc.provision field contract test..." + +echo "=== provision naming only one of the two fields ===" +run_config "$MISSING_FIELD_CONFIG" +echo "$OUT" +if [ "$RUN_RC" -eq 0 ]; then + fail "a provision section missing 'release' was accepted (exit 0); the request should be refused." +elif echo "$OUT" | grep -Fq "$REQUIRED_MESSAGE"; then + pass "a provision section missing 'release' is refused, naming both fields as required" +else + fail "a provision section missing 'release' was not refused with '$REQUIRED_MESSAGE'" +fi + +echo "=== provision with a non-string distribution ===" +run_config "$INVALID_TYPE_CONFIG" +echo "$OUT" +if [ "$RUN_RC" -eq 0 ]; then + fail "a non-string distribution was accepted (exit 0); the request should be refused." +else + pass "a non-string distribution is refused with a non-zero exit status" +fi +if echo "$OUT" | grep -Fq "$INVALID_TYPE_PATH"; then + pass "a non-string distribution is refused, naming the exact path $INVALID_TYPE_PATH" +else + fail "a non-string distribution was not refused with the path $INVALID_TYPE_PATH" +fi +if echo "$OUT" | grep -Fq "$INVALID_TYPE_DETAIL"; then + pass "the type error says what was expected instead" +else + fail "the type error did not say '$INVALID_TYPE_DETAIL'" +fi + +# Positive control. Without it, an lxc-exec that refused every config would +# satisfy every assertion above while verifying nothing. This asserts only the +# absence of the two field diagnostics: the run is expected to fail later on a +# host with no LXC runtime, and that failure is not what is under test. +echo "=== control: both fields present and well typed ===" +run_config "$VALID_CONFIG" +echo "$OUT" +# Recorded from the control's own result envelope so the trap can remove it. +CONTROL_CONTAINER="$(printf '%s' "$OUT" | sed -n 's/.*"containerName":"\([^"]*\)".*/\1/p')" +if echo "$OUT" | grep -Fq "$REQUIRED_MESSAGE"; then + fail "the control config was also refused as missing fields; the assertions above do not discriminate." +elif echo "$OUT" | grep -Fq "$INVALID_TYPE_PATH"; then + fail "the control config also produced a distribution type error; the assertions above do not discriminate." +else + pass "a well-formed provision section produces neither field diagnostic" +fi +# The control may still fail for an environmental reason -- no LXC runtime on +# this host -- which is not what this test covers. It must not fail as a +# malformed request, because that is the class of failure under test. +if echo "$OUT" | grep -Fq '"code":"malformed_request"'; then + fail "the control config was rejected as malformed_request; a well-formed provision section must get past parsing." +fi +if [ "$RUN_RC" -ne 0 ]; then + echo "NOTE: control exited $RUN_RC without a malformed_request rejection, which is environmental and not what this test covers." +fi + +echo "================================" +echo "Results: $PASSED passed, $FAILED failed" +if [ "$FAILED" -gt 0 ]; then + exit 1 +fi +echo "LXC experimental.lxc.provision field contract test complete." diff --git a/tests/scripts/run_lxc_state_aware_network_test.sh b/tests/scripts/run_lxc_state_aware_network_test.sh new file mode 100644 index 000000000..efd124440 --- /dev/null +++ b/tests/scripts/run_lxc_state_aware_network_test.sh @@ -0,0 +1,855 @@ +#!/bin/bash +# LXC state-aware network policy matrix test. +# +# Proves that LXC start requests enforce policy-driven network rules, including +# the inherited default-deny policy when the request omits a network block. +# +# Case 3 proves the default-deny hook reaches a stock provisioned container, +# and reads the host's own iptables state to show the hook was applied rather +# than merely reported. +# +# The start fixtures keep a __SANDBOX_ID__ placeholder because the live LXC +# sandboxId is the container name returned by this test's own provision call. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(dirname "$(dirname "$SCRIPT_DIR")")" +CONFIG_DIR="$REPO_DIR/tests/configs" + +LXC_EXEC="$REPO_DIR/src/target/release/lxc-exec" +if [ ! -f "$LXC_EXEC" ]; then + LXC_EXEC="$REPO_DIR/src/target/debug/lxc-exec" +fi + +SKIP_EXIT=77 +skip() { + echo "SKIP: $1" + exit "$SKIP_EXIT" +} + +WORK_DIR="$REPO_DIR/tests/.lxc_state_aware_network_test.$$" +SANDBOX_ID="" +SANDBOX_STARTED=0 +PASSED=0 +FAILED=0 +QUARANTINED=0 +QUARANTINE_ACTIVE="" +QUARANTINE_NOTES="" +CLEANED_UP=0 +FS_FIXTURES_CREATED=0 + +cleanup() { + if [ "$CLEANED_UP" -ne 0 ]; then + return + fi + CLEANED_UP=1 + if [ -n "$SANDBOX_ID" ]; then + if [ "$SANDBOX_STARTED" -ne 0 ]; then + echo "--- cleanup: stop $SANDBOX_ID ---" + run_phase stop "$SANDBOX_ID" >/dev/null 2>&1 || true + SANDBOX_STARTED=0 + fi + echo "--- cleanup: deprovision $SANDBOX_ID ---" + run_phase deprovision "$SANDBOX_ID" >/dev/null 2>&1 || true + fi + if [ "$FS_FIXTURES_CREATED" -ne 0 ]; then + rm -rf "$FS_RO_DIR" "$FS_DENIED_DIR" + FS_FIXTURES_CREATED=0 + fi + rm -rf "$WORK_DIR" +} +trap cleanup EXIT +trap 'cleanup; exit 130' INT +trap 'cleanup; exit 143' TERM + +run_phase() { + local phase="$1" + local sandbox_id="${2:-}" + local extra="${3:-}" + local req="$WORK_DIR/$phase.json" + + { + printf '{\n "phase": "%s"' "$phase" + if [ "$phase" = "provision" ]; then + printf ',\n "containment": "lxc"' + fi + if [ -n "$sandbox_id" ]; then + printf ',\n "sandboxId": "%s"' "$sandbox_id" + fi + if [ -n "$extra" ]; then + printf ',\n %s' "$extra" + fi + printf '\n}\n' + } > "$req" + + "$LXC_EXEC" "$req" +} + +extract_sandbox_id() { + sed -n 's/.*"sandboxId"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' | head -n 1 +} + +check() { + local name="$1" + local ok="$2" + if [ "$ok" = "0" ]; then + echo "PASS: $name" + PASSED=$((PASSED + 1)) + elif [ -n "$QUARANTINE_ACTIVE" ]; then + # The assertion above is preserved byte for byte and still runs; only + # its tally changes, because it exposes a product gap that predates this + # change and is out of its scope. This must never read as a pass. + echo "QUARANTINED (BEHAVIOR NOT VERIFIED): $name" + echo " reason: $QUARANTINE_ACTIVE" + QUARANTINED=$((QUARANTINED + 1)) + QUARANTINE_NOTES="${QUARANTINE_NOTES} - ${name} + ${QUARANTINE_ACTIVE} +" + else + echo "FAIL: $name" + FAILED=$((FAILED + 1)) + fi +} + +fail_now() { + echo "FAIL: $1" + exit 1 +} + +record_result() { + local case_no="$1" + local config="$2" + local cause="$3" + local expected="$4" + local actual="$5" + local status="$6" + RESULTS="${RESULTS}${case_no}|${config}|${cause}|${expected}|${actual}|${status} +" +} + +CONFIG_NO_NETWORK="$CONFIG_DIR/lxc_state_aware_start_no_network.json" +CONFIG_BLOCK_CAPS="$CONFIG_DIR/lxc_state_aware_start_default_block_capabilities.json" +CONFIG_BLOCK_FIREWALL="$CONFIG_DIR/lxc_state_aware_start_default_block_firewall.json" +CONFIG_ALLOW_CAPS="$CONFIG_DIR/lxc_state_aware_start_default_allow_capabilities.json" +CONFIG_EMPTY_ALLOWED="$CONFIG_DIR/lxc_state_aware_start_empty_allowed_hosts.json" +CONFIG_NONEMPTY_ALLOWED="$CONFIG_DIR/lxc_state_aware_start_nonempty_allowed_hosts_capabilities.json" +CONFIG_PROVISION_NETWORK="$CONFIG_DIR/lxc_state_aware_start_network_at_provision_rejected.json" +CONFIG_PROVISION_FILESYSTEM="$CONFIG_DIR/lxc_state_aware_start_filesystem_at_provision_rejected.json" +CONFIG_BLOCK_BOTH="$CONFIG_DIR/lxc_state_aware_start_default_block_both.json" +CONFIG_ALLOWED_FIREWALL="$CONFIG_DIR/lxc_state_aware_start_allowed_hosts_firewall.json" +CONFIG_BLOCKED_CAPS="$CONFIG_DIR/lxc_state_aware_start_blocked_hosts_capabilities.json" +CONFIG_ALLOW_LOCAL="$CONFIG_DIR/lxc_state_aware_start_allow_local_network.json" +CONFIG_PROXY="$CONFIG_DIR/lxc_state_aware_start_proxy.json" +CONFIG_PROXY_EXTERNAL="$CONFIG_DIR/lxc_state_aware_start_proxy_external.json" +CONFIG_FILESYSTEM_PATHS="$CONFIG_DIR/lxc_state_aware_start_filesystem_paths.json" +RESULTS="" + +verify_fixture_contracts() { + for cfg in "$CONFIG_NO_NETWORK" "$CONFIG_BLOCK_CAPS" "$CONFIG_BLOCK_FIREWALL" \ + "$CONFIG_ALLOW_CAPS" "$CONFIG_EMPTY_ALLOWED" "$CONFIG_NONEMPTY_ALLOWED" \ + "$CONFIG_PROVISION_NETWORK" "$CONFIG_PROVISION_FILESYSTEM" \ + "$CONFIG_BLOCK_BOTH" "$CONFIG_ALLOWED_FIREWALL" "$CONFIG_BLOCKED_CAPS" \ + "$CONFIG_ALLOW_LOCAL" "$CONFIG_PROXY" "$CONFIG_PROXY_EXTERNAL" \ + "$CONFIG_FILESYSTEM_PATHS"; do + [ -f "$cfg" ] || fail_now "fixture not found: $cfg" + done + + # Case 8's fixture is guarded here rather than in the block below so the + # existing seven keep their positional indices. + grep -q '"phase"[[:space:]]*:[[:space:]]*"provision"' "$CONFIG_PROVISION_FILESYSTEM" \ + || fail_now "fixture drift in $CONFIG_PROVISION_FILESYSTEM: phase must be provision" + grep -q '"filesystem"' "$CONFIG_PROVISION_FILESYSTEM" \ + || fail_now "fixture drift in $CONFIG_PROVISION_FILESYSTEM: must carry a filesystem block" + grep -q '"sandboxId"' "$CONFIG_PROVISION_FILESYSTEM" \ + && fail_now "fixture drift in $CONFIG_PROVISION_FILESYSTEM: must not hard-code a sandboxId" + + if command -v python3 >/dev/null 2>&1; then + python3 - "$CONFIG_NO_NETWORK" "$CONFIG_BLOCK_CAPS" "$CONFIG_BLOCK_FIREWALL" \ + "$CONFIG_ALLOW_CAPS" "$CONFIG_EMPTY_ALLOWED" "$CONFIG_NONEMPTY_ALLOWED" \ + "$CONFIG_PROVISION_NETWORK" <<'PY' +import json +import sys + +cases = [json.load(open(path, encoding="utf-8")) for path in sys.argv[1:]] +paths = sys.argv[1:] + +def fail(index, message): + raise SystemExit(f"fixture drift in {paths[index]}: {message}") + +if cases[0].get("phase") != "start" or cases[0].get("sandboxId") != "__SANDBOX_ID__" or "network" in cases[0]: + fail(0, "case 1 must be a start request with no network block") +if cases[1].get("network") != {"defaultPolicy": "block"}: + fail(1, "case 2 must carry only defaultPolicy=block") +if cases[2].get("network") != {"defaultPolicy": "block", "enforcementMode": "firewall"}: + fail(2, "case 3 must carry defaultPolicy=block with enforcementMode=firewall") +if cases[3].get("network") != {"defaultPolicy": "allow"}: + fail(3, "case 4 must carry only defaultPolicy=allow") +if cases[4].get("network") != {"allowedHosts": []}: + fail(4, "case 5 must carry an empty allowedHosts list and no enforcementMode") +if cases[5].get("network") != {"allowedHosts": ["example.com"]}: + fail(5, "case 6 must carry one allowedHosts entry and no enforcementMode") +if cases[6].get("phase") != "provision" or cases[6].get("containment") != "lxc": + fail(6, "case 7 must be an LXC provision request") +if cases[6].get("network") != {"defaultPolicy": "block"}: + fail(6, "case 7 must carry provision-time network.defaultPolicy=block") +if "sandboxId" in cases[6]: + fail(6, "case 7 must not hard-code a sandboxId") +PY + [ $? -eq 0 ] || fail_now "fixture drift in the case 1-7 configs" + else + grep -q '"network"' "$CONFIG_NO_NETWORK" && fail_now "fixture drift in $CONFIG_NO_NETWORK: case 1 must have no network block" + grep -q '"defaultPolicy"[[:space:]]*:[[:space:]]*"block"' "$CONFIG_BLOCK_CAPS" || fail_now "fixture drift in $CONFIG_BLOCK_CAPS: missing defaultPolicy=block" + grep -q '"enforcementMode"' "$CONFIG_BLOCK_CAPS" && fail_now "fixture drift in $CONFIG_BLOCK_CAPS: enforcementMode must be omitted" + grep -q '"defaultPolicy"[[:space:]]*:[[:space:]]*"block"' "$CONFIG_BLOCK_FIREWALL" || fail_now "fixture drift in $CONFIG_BLOCK_FIREWALL: missing defaultPolicy=block" + grep -q '"enforcementMode"[[:space:]]*:[[:space:]]*"firewall"' "$CONFIG_BLOCK_FIREWALL" || fail_now "fixture drift in $CONFIG_BLOCK_FIREWALL: missing enforcementMode=firewall" + grep -q '"defaultPolicy"[[:space:]]*:[[:space:]]*"allow"' "$CONFIG_ALLOW_CAPS" || fail_now "fixture drift in $CONFIG_ALLOW_CAPS: missing defaultPolicy=allow" + grep -q '"enforcementMode"' "$CONFIG_ALLOW_CAPS" && fail_now "fixture drift in $CONFIG_ALLOW_CAPS: enforcementMode must be omitted" + grep -q '"allowedHosts"[[:space:]]*:[[:space:]]*\[\]' "$CONFIG_EMPTY_ALLOWED" || fail_now "fixture drift in $CONFIG_EMPTY_ALLOWED: allowedHosts must be empty" + grep -q '"allowedHosts"[[:space:]]*:[[:space:]]*\["example.com"\]' "$CONFIG_NONEMPTY_ALLOWED" || fail_now "fixture drift in $CONFIG_NONEMPTY_ALLOWED: allowedHosts must contain example.com" + grep -q '"phase"[[:space:]]*:[[:space:]]*"provision"' "$CONFIG_PROVISION_NETWORK" || fail_now "fixture drift in $CONFIG_PROVISION_NETWORK: phase must be provision" + grep -q '"defaultPolicy"[[:space:]]*:[[:space:]]*"block"' "$CONFIG_PROVISION_NETWORK" || fail_now "fixture drift in $CONFIG_PROVISION_NETWORK: missing defaultPolicy=block" + grep -q '"sandboxId"' "$CONFIG_PROVISION_NETWORK" && fail_now "fixture drift in $CONFIG_PROVISION_NETWORK: must not hard-code sandboxId" + fi + + # Guarded in its own block so the seven above keep their positional indices. + # Each of these fixtures isolates one field, and a field that drifts into a + # neighbouring fixture would let a case pass while pinning nothing. + if command -v python3 >/dev/null 2>&1; then + python3 - "$CONFIG_BLOCK_BOTH" "$CONFIG_ALLOWED_FIREWALL" "$CONFIG_BLOCKED_CAPS" \ + "$CONFIG_ALLOW_LOCAL" "$CONFIG_PROXY" "$CONFIG_FILESYSTEM_PATHS" \ + "$CONFIG_PROXY_EXTERNAL" <<'PY' +import json +import sys + +paths = sys.argv[1:] +cases = [json.load(open(path, encoding="utf-8")) for path in paths] + +def fail(index, message): + raise SystemExit(f"fixture drift in {paths[index]}: {message}") + +for i, case in enumerate(cases): + if case.get("phase") != "start" or case.get("sandboxId") != "__SANDBOX_ID__": + fail(i, "must be a start request carrying the sandbox id placeholder") + +if cases[0].get("network") != {"defaultPolicy": "block", "enforcementMode": "both"}: + fail(0, "case 11 must carry defaultPolicy=block with enforcementMode=both") +if cases[1].get("network") != {"allowedHosts": ["example.com"], "enforcementMode": "firewall"}: + fail(1, "case 12 must carry one allowedHosts entry with enforcementMode=firewall") +if cases[2].get("network") != {"blockedHosts": ["evil.example.com"]}: + fail(2, "case 13 must carry one blockedHosts entry and no enforcementMode") +if cases[3].get("network", {}).get("allowLocalNetwork") is not True: + fail(3, "case 14 must request allowLocalNetwork=true") +if cases[4].get("network") != {"proxy": {"builtinTestServer": True}}: + fail(4, "case 15 must carry exactly the builtin-test-server proxy and nothing else: an " + "external proxy url, a host list, or an enforcementMode is refused earlier by a " + "shared parse-time rule, so the case would pass without ever reaching the LXC " + "start verdict it exists to pin") +if cases[5].get("filesystem") != { + "readonlyPaths": ["/mxc-e2e-ro"], + "deniedPaths": ["/mxc-e2e-denied"], +}: + fail(5, "case 16 must carry exactly the readonly and denied paths the case creates on the host") +if "network" in cases[5]: + fail(5, "case 16 must carry no network block so it isolates the filesystem fields") +if not cases[6].get("network", {}).get("proxy", {}).get("url"): + fail(6, "case 15's second half must carry an external proxy url, which is the form a " + "production config uses") +if cases[6].get("network", {}).get("proxy", {}).get("builtinTestServer"): + fail(6, "case 15's second half must not be the builtin form; the two halves exist to cover " + "the two different refusal routes") +PY + [ $? -eq 0 ] || fail_now "fixture drift in the field-isolation configs" + fi + echo "Fixture drift guard passed for all LXC state-aware network configs." +} + +make_request_from_config() { + local config="$1" + local out="$2" + sed "s/__SANDBOX_ID__/$SANDBOX_ID/g" "$config" > "$out" +} + +expect_error_code() { + local output="$1" + local code="$2" + echo "$output" | grep -Eq '"code"[[:space:]]*:[[:space:]]*"'"$code"'"' +} + +start_fresh_sandbox() { + local label="$1" + local out rc + echo "=== provision for $label ===" + out="$($LXC_EXEC "$CONFIG_DIR/lxc_state_aware_provision.json")" + rc=$? + echo "$out" + if [ "$rc" -ne 0 ]; then + fail_now "$label: provision failed before the network input could be tested (config: $CONFIG_DIR/lxc_state_aware_provision.json, rc=$rc)." + fi + SANDBOX_ID="$(printf '%s' "$out" | extract_sandbox_id)" + if [ -z "$SANDBOX_ID" ]; then + fail_now "$label: provision did not return a sandboxId, so the start input cannot be tested." + fi + case "$SANDBOX_ID" in + lxc:mxc-*) ;; + *) fail_now "$label: provision returned unsafe-looking sandboxId '$SANDBOX_ID'." ;; + esac + SANDBOX_STARTED=0 +} + +finish_current_sandbox() { + if [ -n "$SANDBOX_ID" ]; then + if [ "$SANDBOX_STARTED" -ne 0 ]; then + echo "=== stop $SANDBOX_ID ===" + run_phase stop "$SANDBOX_ID" + check "stop after $1 exits 0 for input $2" $? + SANDBOX_STARTED=0 + fi + echo "=== deprovision $SANDBOX_ID ===" + run_phase deprovision "$SANDBOX_ID" + local deprovision_rc=$? + check "deprovision after $1 exits 0 for input $2" "$deprovision_rc" + # Release the ID only once the container is actually gone. Clearing it + # after a failed deprovision disarms the EXIT trap, so a container this + # case could not remove leaks into every later case in the matrix. + if [ "$deprovision_rc" -eq 0 ]; then + SANDBOX_ID="" + fi + fi +} + +container_host_veth() { + local name="$1" + local peer + + # Ask the container which host ifindex its own link is paired with, then + # resolve that index on the host. Nothing here trusts a name MXC chose, so + # a hook aimed at the wrong interface still fails. + # + # `lxc-info` reports a Link: line instead, but it walks lxc.net.N from 0 and + # stops at the first gap, so it reports nothing at all for a container whose + # interface is numbered above a hole -- which is exactly the topology case 10 + # builds. Reading it from the container's own kernel view works whatever the + # index. + peer="$(lxc-attach -n "$name" -- ip -o link 2>/dev/null \ + | sed -n 's/.*@if\([0-9][0-9]*\):.*/\1/p' | head -1)" + [ -n "$peer" ] || return 0 + ip -o link 2>/dev/null \ + | awk -v want="$peer" '{ idx = $1; sub(/:$/, "", idx); if (idx == want) { n = $2; sub(/[@:].*/, "", n); print n; exit } }' +} + +renumber_sole_interface() { + local index="$1" + local label="$2" + local name="${SANDBOX_ID#lxc:}" + local cfg="${LXC_PATH:-/var/lib/lxc}/$name/config" + + # A provisioned container is handed its interface by an include, always at + # lxc.net.0, so this is the only way to build the one topology that tells + # "MXC found the interface" apart from "MXC assumed index 0". Assigning an + # empty lxc.net clears what the include supplied; the numbered keys then + # declare the same interface somewhere else. + [ -f "$cfg" ] || fail_now "$label: no container config at $cfg to renumber." + { + echo "lxc.net =" + echo "lxc.net.$index.type = veth" + echo "lxc.net.$index.link = lxcbr0" + echo "lxc.net.$index.flags = up" + } >> "$cfg" + + echo "=== $label: renumbered sole interface to lxc.net.$index ===" + echo " lxc.net -> $(lxc-info -n "$name" -c lxc.net 2>&1 | tr '\n' ' ')" + echo " index 0 type -> $(lxc-info -n "$name" -c lxc.net.0.type 2>&1 | tr '\n' ' ')" + echo " index $index type -> $(lxc-info -n "$name" -c "lxc.net.$index.type" 2>&1 | tr '\n' ' ')" + + # Without these the case is vacuous: a renumber that silently failed leaves + # an ordinary index-0 container, every assertion below still passes, and the + # test reports success for a topology it never built. + if lxc-info -n "$name" -c lxc.net.0.type 2>/dev/null | grep -q .; then + fail_now "$label: renumber did not take -- liblxc still reports an interface at lxc.net.0, so this case would not exercise index independence." + fi + if ! lxc-info -n "$name" -c "lxc.net.$index.type" 2>/dev/null | grep -q "veth"; then + fail_now "$label: renumber did not take -- liblxc reports no veth at lxc.net.$index." + fi + if [ "$(lxc-info -n "$name" -c lxc.net 2>/dev/null | grep -c 'veth')" != "1" ]; then + fail_now "$label: renumber left the container with something other than exactly one interface." + fi +} + +run_start_case() { + local case_no="$1" + local config="$2" + local cause="$3" + local expected="$4" + local expect_success="$5" + local must_exec="$6" + local clause="$7" + local assert_default_deny="${8:-}" + local quarantine="${9:-}" + local renumber_to="${10:-}" + local req="$WORK_DIR/case_${case_no}.json" + local out rc actual status sentinel + + start_fresh_sandbox "case $case_no" + if [ -n "$renumber_to" ]; then + renumber_sole_interface "$renumber_to" "case $case_no" + fi + make_request_from_config "$config" "$req" + QUARANTINE_ACTIVE="$quarantine" + + echo "=== case $case_no start: $cause ===" + out="$($LXC_EXEC "$req" 2>&1)" + rc=$? + echo "$out" + + if [ "$expect_success" = "1" ]; then + if [ "$rc" -eq 0 ]; then + check "case $case_no start succeeds for input $config -- $clause" 0 + SANDBOX_STARTED=1 + actual="start exited 0" + status="PASS" + else + check "case $case_no start succeeds for input $config -- $clause" 1 + actual="start exited $rc: $(echo "$out" | tr '\n' ' ' | sed 's/|/ /g')" + status="FAIL" + fi + + if [ "$must_exec" = "1" ] && [ "$rc" -eq 0 ]; then + sentinel="MXC_STATE_AWARE_NETWORK_CASE_${case_no}_RAN" + echo "=== case $case_no exec: prove container is running ===" + out="$(run_phase exec "$SANDBOX_ID" '"process": { "commandLine": "echo '"$sentinel"'" }' 2>&1)" + rc=$? + echo "$out" + if [ "$rc" -eq 0 ] && echo "$out" | grep -Fq "$sentinel"; then + check "case $case_no exec observes running container for input $config -- $clause" 0 + actual="$actual; exec printed $sentinel" + else + check "case $case_no exec observes running container for input $config -- $clause" 1 + actual="$actual; exec rc=$rc output=$(echo "$out" | tr '\n' ' ' | sed 's/|/ /g')" + status="FAIL" + fi + fi + + if [ "$assert_default_deny" = "1" ] && [ "$rc" -eq 0 ]; then + local veth chain direct terminal + # The roadmap asks that the hook be applied, which a zero exit does + # not show: a run that skipped enforcement and still reported success + # would look identical here. Read the host instead, and read it + # through the interface the container actually ended up with rather + # than a name this test derived, so a hook aimed at the wrong + # interface cannot pass. + # + # The physdev form is the one that decides the case. lxcbr0 is a + # bridge, and on a bridged veth the plain `-i` rule installs cleanly + # and never matches, so asserting only that form would accept a + # container whose traffic walks past the chain untouched. + veth="$(container_host_veth "${SANDBOX_ID#lxc:}")" + chain="" + direct="" + if [ -n "$veth" ]; then + chain="$(iptables -S FORWARD 2>/dev/null \ + | awk -v v="$veth" 'index($0, "--physdev-in " v " -j ") { for (i = 1; i <= NF; i++) if ($i == "-j") { print $(i + 1); exit } }')" + direct="$(iptables -S FORWARD 2>/dev/null \ + | awk -v v="$veth" 'index($0, "-i " v " -j ") { for (i = 1; i <= NF; i++) if ($i == "-j") { print $(i + 1); exit } }')" + fi + + echo "=== case $case_no default-deny: live veth=${veth:-} physdev->${chain:-} direct->${direct:-} ===" + if [ -n "$veth" ] && [ -n "$chain" ] && [ "$direct" = "$chain" ]; then + check "case $case_no hooks live veth $veth into $chain on both the physdev and direct paths -- (N1) default-deny outbound" 0 + actual="$actual; FORWARD hooks $veth to $chain on both paths" + else + check "case $case_no hooks live veth ${veth:-} into a chain on both the physdev and direct paths -- (N1) default-deny outbound" 1 + actual="$actual; physdev hook=${chain:-} direct hook=${direct:-} for veth ${veth:-}" + status="FAIL" + fi + + terminal="$(iptables -S "$chain" 2>/dev/null | tail -1)" + echo "=== case $case_no terminal rule: ${terminal:-} ===" + if [ -n "$chain" ] && [ "${terminal##* }" = "DROP" ]; then + check "case $case_no hooked chain ends in DROP -- (N1) default-deny outbound" 0 + actual="$actual; chain ends in DROP" + else + check "case $case_no hooked chain ends in DROP -- (N1) default-deny outbound" 1 + actual="$actual; terminal rule was ${terminal:-}" + status="FAIL" + fi + fi + else + if [ "$rc" -ne 0 ] && expect_error_code "$out" "policy_validation"; then + check "case $case_no start rejects input $config with policy_validation -- $clause" 0 + actual="start exited $rc with policy_validation" + status="PASS" + else + check "case $case_no start rejects input $config with policy_validation -- $clause" 1 + actual="start rc=$rc output=$(echo "$out" | tr '\n' ' ' | sed 's/|/ /g')" + status="FAIL" + if [ "$rc" -eq 0 ]; then + SANDBOX_STARTED=1 + fi + fi + fi + + # Clear before teardown so a genuine stop/deprovision failure is still a + # real failure rather than being absorbed by this case's quarantine. + if [ -n "$QUARANTINE_ACTIVE" ] && [ "$status" = "FAIL" ]; then + status="QUARANTINED" + fi + QUARANTINE_ACTIVE="" + + finish_current_sandbox "case $case_no" "$config" + record_result "$case_no" "$config" "$cause" "$expected" "$actual" "$status" +} + +run_provision_rejection_case() { + local case_no="$1" + local config="$2" + local cause="$3" + local clause="$4" + local expected='rejected with policy_validation' + local out rc actual status + + echo "=== case $case_no provision rejection: $cause ===" + out="$($LXC_EXEC "$config" 2>&1)" + rc=$? + echo "$out" + + if [ "$rc" -ne 0 ] && expect_error_code "$out" "policy_validation"; then + check "case $case_no provision rejects input $config with policy_validation -- $clause" 0 + actual="provision exited $rc with policy_validation" + status="PASS" + else + check "case $case_no provision rejects input $config with policy_validation -- $clause" 1 + actual="provision rc=$rc output=$(echo "$out" | tr '\n' ' ' | sed 's/|/ /g')" + status="FAIL" + SANDBOX_ID="$(printf '%s' "$out" | extract_sandbox_id)" + if [ -n "$SANDBOX_ID" ]; then + case "$SANDBOX_ID" in + lxc:mxc-*) ;; + *) fail_now "case $case_no unexpectedly returned unsafe-looking sandboxId '$SANDBOX_ID'; refusing to deprovision it." ;; + esac + fi + fi + finish_current_sandbox "case $case_no" "$config" + record_result "$case_no" "$config" "$cause" "$expected" "$actual" "$status" +} + +FS_RO_DIR="/mxc-e2e-ro" +FS_DENIED_DIR="/mxc-e2e-denied" +FS_SENTINEL="MXC_E2E_FS_SENTINEL" + +# The filesystem lists are the one part of the start policy whose effect is not +# visible in iptables, so this case reads it where it does show: from inside the +# container. Host directories are created first because the parser refuses paths +# that do not exist (roadmap item 8), and each carries a sentinel file so a mount +# that silently did not happen cannot be mistaken for one that did. +run_filesystem_start_case() { + local case_no="$1" + local config="$2" + local cause="$3" + local clause="$4" + local expected='start succeeds, the readonly path is readable but not writable, and the denied path is masked' + local req="$WORK_DIR/case_${case_no}.json" + local out rc actual status probe + + # These are absolute paths at the host root, so anything already there + # belongs to something else and deleting it would destroy data this test + # never created. + for fixture_dir in "$FS_RO_DIR" "$FS_DENIED_DIR"; do + if [ -e "$fixture_dir" ]; then + fail_now "case $case_no found a pre-existing host path '$fixture_dir'; refusing to delete a directory this test did not create. Remove it by hand if an interrupted run left it behind." + fi + done + mkdir -p "$FS_RO_DIR" "$FS_DENIED_DIR" || fail_now "case $case_no could not create its host fixture directories" + FS_FIXTURES_CREATED=1 + echo "$FS_SENTINEL" > "$FS_RO_DIR/sentinel" + echo "$FS_SENTINEL" > "$FS_DENIED_DIR/sentinel" + + start_fresh_sandbox "case $case_no" + make_request_from_config "$config" "$req" + + echo "=== case $case_no start: $cause ===" + out="$($LXC_EXEC "$req" 2>&1)" + rc=$? + echo "$out" + + if [ "$rc" -eq 0 ]; then + check "case $case_no start succeeds for input $config -- $clause" 0 + SANDBOX_STARTED=1 + actual="start exited 0" + status="PASS" + else + check "case $case_no start succeeds for input $config -- $clause" 1 + actual="start exited $rc: $(echo "$out" | tr '\n' ' ' | sed 's/|/ /g')" + status="FAIL" + fi + + if [ "$rc" -eq 0 ]; then + echo "=== case $case_no probe: read the mounts from inside the container ===" + probe="cat $FS_RO_DIR/sentinel 2>/dev/null; touch $FS_RO_DIR/probe 2>/dev/null && echo RO_WRITABLE || echo RO_READONLY; cat $FS_DENIED_DIR/sentinel 2>/dev/null && echo DENIED_VISIBLE || echo DENIED_HIDDEN" + out="$(run_phase exec "$SANDBOX_ID" '"process": { "commandLine": "'"$probe"'" }' 2>&1)" + rc=$? + echo "$out" + + if [ "$rc" -eq 0 ] && echo "$out" | grep -Fq "$FS_SENTINEL"; then + check "case $case_no readonlyPaths mounts the host directory into the container -- $clause" 0 + actual="$actual; readonly path carries the host sentinel" + else + check "case $case_no readonlyPaths mounts the host directory into the container -- $clause" 1 + actual="$actual; readonly path did not carry the host sentinel (exec rc=$rc)" + status="FAIL" + fi + + if echo "$out" | grep -Fq "RO_READONLY"; then + check "case $case_no readonlyPaths is mounted read-only -- $clause" 0 + actual="$actual; readonly path refused a write" + else + check "case $case_no readonlyPaths is mounted read-only -- $clause" 1 + actual="$actual; readonly path accepted a write" + status="FAIL" + fi + + if echo "$out" | grep -Fq "DENIED_HIDDEN"; then + check "case $case_no deniedPaths masks the host directory -- $clause" 0 + actual="$actual; denied path is masked" + else + check "case $case_no deniedPaths masks the host directory -- $clause" 1 + actual="$actual; denied path still exposed the host sentinel" + status="FAIL" + fi + fi + + finish_current_sandbox "case $case_no" "$config" + rm -rf "$FS_RO_DIR" "$FS_DENIED_DIR" + FS_FIXTURES_CREATED=0 + record_result "$case_no" "$config" "$cause" "$expected" "$actual" "$status" +} + +LXC_PROXY_REFUSAL='LXC state-aware start does not support network.proxy' + +# The proxy field needs its own case because neither wire form reaches the LXC +# verdict by the route the other cases use. +# +# A state-aware start may not carry `containment` -- the backend is fixed at +# provision and later phases route by sandboxId -- so the shared parser applies +# its backend-specific rules under the default backend. An external proxy url is +# refused there, before any LXC code runs. The builtin-test-server form skips +# that rule but is testing-only scaffolding gated centrally for every backend, +# so it needs --allow-testing-features to get past the gate. +# +# Both halves are asserted. The first opens the testing gate and requires LXC's +# own refusal *by its message*, because a case that accepted any non-zero exit +# would pass on the central gate alone and would still pass with LXC's refusal +# deleted. The second sends the production-shaped external form and requires only +# that it is refused -- which layer refuses it is not LXC's contract to state, +# but silently accepting it would leave the container talking past a proxy the +# caller believed was in force. +run_proxy_start_case() { + local case_no="$1" + local cause="$2" + local clause="$3" + local expected='start is refused, and the builtin form is refused by LXC itself' + local req="$WORK_DIR/case_${case_no}.json" + local out rc actual status + + start_fresh_sandbox "case $case_no" + make_request_from_config "$CONFIG_PROXY" "$req" + + echo "=== case $case_no start: $cause ===" + out="$($LXC_EXEC --allow-testing-features "$req" 2>&1)" + rc=$? + echo "$out" + + if [ "$rc" -ne 0 ] && expect_error_code "$out" "policy_validation" \ + && echo "$out" | grep -Fq "$LXC_PROXY_REFUSAL"; then + check "case $case_no start refuses the builtin proxy with LXC's own verdict -- $clause" 0 + actual="start exited $rc with LXC's policy_validation refusal" + status="PASS" + else + check "case $case_no start refuses the builtin proxy with LXC's own verdict -- $clause" 1 + actual="start rc=$rc output=$(echo "$out" | tr '\n' ' ' | sed 's/|/ /g')" + status="FAIL" + if [ "$rc" -eq 0 ]; then + SANDBOX_STARTED=1 + fi + fi + + if [ "$rc" -ne 0 ]; then + make_request_from_config "$CONFIG_PROXY_EXTERNAL" "$req" + echo "=== case $case_no start: external proxy url ===" + out="$($LXC_EXEC "$req" 2>&1)" + rc=$? + echo "$out" + if [ "$rc" -ne 0 ]; then + check "case $case_no start refuses an external proxy url rather than accepting it -- $clause" 0 + actual="$actual; external url refused" + else + check "case $case_no start refuses an external proxy url rather than accepting it -- $clause" 1 + actual="$actual; external url started the container" + status="FAIL" + SANDBOX_STARTED=1 + fi + fi + + finish_current_sandbox "case $case_no" "$CONFIG_PROXY" + record_result "$case_no" "$CONFIG_PROXY" "$cause" "$expected" "$actual" "$status" +} + +print_case_table() { + echo "| Case | Config file | Cause | Expected effect | Actual result | Status |" + echo "|---|---|---|---|---|---|" + printf '%s' "$RESULTS" | while IFS='|' read -r case_no config cause expected actual status; do + [ -n "$case_no" ] || continue + echo "| $case_no | $config | $cause | $expected | $actual | $status |" + done +} + +verify_fixture_contracts +mkdir -p "$WORK_DIR" || fail_now "could not create work directory $WORK_DIR" + +[ "$(id -u)" -eq 0 ] || skip "LXC state-aware network matrix UNVERIFIED — requires root for LXC." +command -v iptables >/dev/null 2>&1 || skip "LXC state-aware network matrix UNVERIFIED — iptables is not installed." +command -v ip6tables >/dev/null 2>&1 || skip "LXC state-aware network matrix UNVERIFIED — ip6tables is not installed." +command -v lxc-create >/dev/null 2>&1 || skip "LXC state-aware network matrix UNVERIFIED — LXC (lxc-create) is not installed." +[ -f "$LXC_EXEC" ] || skip "LXC state-aware network matrix UNVERIFIED — lxc-exec binary not built; run build.sh first." + +echo "Running LXC state-aware network policy matrix test..." + +run_start_case "1" "$CONFIG_NO_NETWORK" \ + 'no network block at start' \ + 'start succeeds, exec proves the container runs, and FORWARD drops by default' \ + "1" "1" \ + 'an absent network block inherits deny-by-default and is enforced' \ + "1" + +run_start_case "2" "$CONFIG_BLOCK_CAPS" \ + 'defaultPolicy=block with enforcementMode omitted at start' \ + 'start succeeds, exec proves the container runs, and FORWARD drops by default' \ + "1" "1" \ + 'defaultPolicy=block is enforced when enforcementMode is omitted' \ + "1" + +# +# This is the direct observable proof of roadmap item 13's "ensure hook is +# always applied": a stock provisioned container carries +# `lxc.include = /usr/share/lxc/config/common.conf`, and the default-deny hook +# has to reach it anyway. The assertion was previously quarantined because +# enumeration read the container's own config file, where an include hides +# whatever it pulls in; it is live now that enumeration asks liblxc, which has +# already resolved the include. +run_start_case "3" "$CONFIG_BLOCK_FIREWALL" \ + 'defaultPolicy=block with enforcementMode=firewall at start' \ + 'start succeeds, exec proves the container runs, and FORWARD drops by default' \ + "1" "1" \ + 'an explicit firewall mode remains accepted, but is not required for enforcement' \ + "1" + +run_start_case "4" "$CONFIG_ALLOW_CAPS" \ + 'defaultPolicy=allow with enforcementMode omitted at start' \ + 'start succeeds' \ + "1" "0" \ + 'a permissive default needs no iptables rule and does not require firewall enforcement' + +run_start_case "5" "$CONFIG_EMPTY_ALLOWED" \ + 'allowedHosts empty list with enforcementMode omitted at start' \ + 'start succeeds, exec proves the container runs, and FORWARD drops by default' \ + "1" "1" \ + 'an empty list does not relax the inherited default-deny policy' \ + "1" + +run_start_case "6" "$CONFIG_NONEMPTY_ALLOWED" \ + 'allowedHosts non-empty with enforcementMode omitted at start' \ + 'start succeeds, exec proves the container runs, and FORWARD drops by default' \ + "1" "1" \ + 'a non-empty allowedHosts policy is enforced when enforcementMode is omitted' \ + "1" + +# Clause: roadmap item 13 (N1) asks that default-deny outbound be enforced. It +# says nothing about how the container numbers its interfaces, and an interface +# at lxc.net.3 is exactly as filterable as one at lxc.net.0 -- enforcement must +# not read the index at all. This is the only case that separates "MXC enforced +# whichever interface the container has" from "MXC assumed index 0", so it is +# the one that fails if any index assumption is ever reintroduced. +run_start_case "10" "$CONFIG_BLOCK_FIREWALL" \ + 'defaultPolicy=block with enforcementMode=firewall on a container whose only interface is at lxc.net.3' \ + 'start succeeds, exec proves the container runs, and FORWARD drops by default' \ + "1" "1" \ + '(N1) default-deny outbound does not depend on the interface index' \ + "1" "" "3" + +# Clause: the LXC matrix marks network as rejected at provision. +run_provision_rejection_case "7" "$CONFIG_PROVISION_NETWORK" \ + 'network.defaultPolicy=block sent at provision' \ + 'matrix marks network as rejected at provision' + +# Clause: the LXC matrix marks a non-empty filesystem path list as rejected at +# provision. +run_provision_rejection_case "8" "$CONFIG_PROVISION_FILESYSTEM" \ + 'filesystem block with a populated path list sent at provision' \ + 'matrix marks a non-empty filesystem path list as rejected at provision' + +# Clause: roadmap item 13 (N1) names firewall enforcement, and the schema offers +# `both` alongside `firewall`. Case 3 pins `firewall`; nothing pinned `both`, so +# a mode that parsed but enforced nothing would have gone unnoticed. This asserts +# the same host-visible default-deny for it. +run_start_case "11" "$CONFIG_BLOCK_BOTH" \ + 'defaultPolicy=block with enforcementMode=both at start' \ + 'start succeeds, exec proves the container runs, and FORWARD drops by default' \ + "1" "1" \ + '(N1) default-deny outbound is enforced under enforcementMode=both' \ + "1" + +# Case 12 preserves compatibility for callers that explicitly request firewall enforcement. +run_start_case "12" "$CONFIG_ALLOWED_FIREWALL" \ + 'allowedHosts non-empty with enforcementMode=firewall at start' \ + 'start succeeds, exec proves the container runs, and FORWARD drops by default' \ + "1" "1" \ + '(N3) a non-empty allowedHosts policy remains enforceable with an explicit firewall mode' \ + "1" + +run_start_case "13" "$CONFIG_BLOCKED_CAPS" \ + 'blockedHosts non-empty with enforcementMode omitted at start' \ + 'start succeeds, exec proves the container runs, and FORWARD drops by default' \ + "1" "1" \ + '(N4) a non-empty blockedHosts policy is enforced when enforcementMode is omitted' \ + "1" + +# Clause: roadmap item 14 (N2) is explicit -- "reject `hostLoopback: "allow"` +# rather than guessing ports or exposing the container IP." The roadmap records +# `allowLocalNetwork` as "parsed but silently ignored", which is the behavior the +# item asks to replace. A silent ignore and a refusal both exit non-zero on +# nothing, so only asserting the error code separates them. +run_start_case "14" "$CONFIG_ALLOW_LOCAL" \ + 'allowLocalNetwork=true at start' \ + 'start fails with policy_validation' \ + "0" "0" \ + '(N2) allowLocalNetwork is rejected rather than silently ignored' + +# Clause: roadmap item 17 (N5) records the proxy field as one the backend +# ignores, and asks for env-var injection plus egress restriction to the proxy +# port. That work is not in this PR. What is in scope is item 13 (N1)'s posture: +# fail fast rather than silently skip. A start that accepted a proxy it cannot +# enforce would report success while the container talked past it, so it is +# refused until the enforcement item 17 describes exists. See +# run_proxy_start_case for why this case needs both wire forms. +run_proxy_start_case "15" \ + 'network.proxy set at start' \ + '(N5) an unenforceable proxy is refused rather than silently ignored' + +# Clause: the start phase accepts the filesystem lists it rejects at provision +# (case 8), and D1/D4 ask that readonly be readable-not-writable and that +# denied be masked. Case 8 only pins the provision-time refusal; without this, +# no case shows the lists ever take effect anywhere. +run_filesystem_start_case "16" "$CONFIG_FILESYSTEM_PATHS" \ + 'readonlyPaths and deniedPaths sent at start' \ + 'the start phase applies the filesystem lists it refuses at provision' + +echo "================================" +echo "Results: $PASSED passed, $FAILED failed, $QUARANTINED quarantined" +print_case_table +if [ "$QUARANTINED" -gt 0 ]; then + echo "" + echo "!!! $QUARANTINED assertion(s) QUARANTINED -- these behaviors are NOT verified !!!" + printf '%s' "$QUARANTINE_NOTES" +fi +if [ "$FAILED" -gt 0 ]; then + exit 1 +fi diff --git a/tests/scripts/run_lxc_state_aware_test.sh b/tests/scripts/run_lxc_state_aware_test.sh new file mode 100755 index 000000000..d11527134 --- /dev/null +++ b/tests/scripts/run_lxc_state_aware_test.sh @@ -0,0 +1,181 @@ +#!/bin/bash +# LXC state-aware lifecycle test. +# +# Drives the full provision -> start -> exec -> stop -> deprovision sequence +# against lxc-exec, relaying the sandboxId the way a real client does. This is +# the Linux counterpart to run_isolation_session_state_aware_tests.ps1 and +# run_windows_sandbox_state_aware_tests.ps1. +# +# Until this existed, tests/configs/lxc_state_aware_provision.json was checked +# in but never executed by anything, so no test covered the LXC lifecycle end to +# end: the unit tests stub the container, and run_lxc_all_tests.sh only exercised +# the one-shot path. A phase that failed on a real host would ship green. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_DIR="$(dirname "$(dirname "$SCRIPT_DIR")")" +CONFIG_DIR="$REPO_DIR/tests/configs" + +LXC_EXEC="$REPO_DIR/src/target/release/lxc-exec" +if [ ! -f "$LXC_EXEC" ]; then + LXC_EXEC="$REPO_DIR/src/target/debug/lxc-exec" +fi + +# An honest skip for a missing prerequisite: exit 77 so run_lxc_all_tests.sh +# records SKIPPED rather than FAILED. Without this the lifecycle runs anyway and +# dies at provision on any host without LXC, which reads as a broken test rather +# than an unmet prerequisite. The provision config asks for no network, so this +# probes only root, the LXC runtime, and the binary. +SKIP_EXIT=77 +skip() { + echo "SKIP: $1" + exit "$SKIP_EXIT" +} + +[ "$(id -u)" -eq 0 ] || skip "requires root for LXC." +command -v lxc-create >/dev/null 2>&1 || skip "LXC (lxc-create) is not installed." +[ -f "$LXC_EXEC" ] || skip "lxc-exec binary not built; run build.sh first." + +WORK_DIR="$(mktemp -d)" +SANDBOX_ID="" +PASSED=0 +FAILED=0 +CLEANED_UP=0 + +# Always attempt to tear the container down, including on an early failure or a +# signal. A leaked container outlives the test run and breaks the next one, so +# this is best-effort and deliberately ignores its own exit status. +# +# Bash resumes the script after an INT or TERM handler returns, so those two +# need handlers that clean up and then exit rather than letting the run carry +# on with the work directory already gone. Each exit fires the EXIT trap as +# well, so the guard keeps the deprovision to one run rather than two. +cleanup() { + if [ "$CLEANED_UP" -ne 0 ]; then + return + fi + CLEANED_UP=1 + if [ -n "$SANDBOX_ID" ]; then + echo "--- cleanup: deprovision $SANDBOX_ID ---" + run_phase deprovision "$SANDBOX_ID" >/dev/null 2>&1 || true + fi + rm -rf "$WORK_DIR" +} +trap cleanup EXIT +trap 'cleanup; exit 130' INT +trap 'cleanup; exit 143' TERM + +# Emit a state-aware request for $1 (phase) with sandboxId $2, plus any extra +# JSON members in $3, then run it. Envelope goes to stdout, diagnostics to +# stderr, so the caller can parse stdout directly. +run_phase() { + local phase="$1" + local sandbox_id="${2:-}" + local extra="${3:-}" + local req="$WORK_DIR/$phase.json" + + { + printf '{\n "phase": "%s"' "$phase" + # The parser rejects a non-provision envelope that carries containment; + # the backend is fixed at provision and later phases route by sandboxId. + if [ "$phase" = "provision" ]; then + printf ',\n "containment": "lxc"' + fi + if [ -n "$sandbox_id" ]; then + printf ',\n "sandboxId": "%s"' "$sandbox_id" + fi + if [ -n "$extra" ]; then + printf ',\n %s' "$extra" + fi + printf '\n}\n' + } > "$req" + + "$LXC_EXEC" "$req" +} + +# Pull sandboxId out of a result envelope without depending on jq or python, +# neither of which is guaranteed on an LXC test host. +extract_sandbox_id() { + sed -n 's/.*"sandboxId"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' | head -n 1 +} + +check() { + local name="$1" + local ok="$2" + if [ "$ok" = "0" ]; then + echo "PASS: $name" + PASSED=$((PASSED + 1)) + else + echo "FAIL: $name" + FAILED=$((FAILED + 1)) + fi +} + +echo "Running LXC state-aware lifecycle test..." + +# --- provision ------------------------------------------------------------- +# Uses the checked-in config so the distribution/release stay in one place. +echo "=== provision ===" +PROVISION_OUT="$("$LXC_EXEC" "$CONFIG_DIR/lxc_state_aware_provision.json")" +PROVISION_RC=$? +check "provision exits 0" "$PROVISION_RC" +echo "$PROVISION_OUT" + +SANDBOX_ID="$(printf '%s' "$PROVISION_OUT" | extract_sandbox_id)" +if [ -n "$SANDBOX_ID" ]; then + check "provision returns a sandboxId" 0 +else + check "provision returns a sandboxId" 1 + echo "Cannot continue without a sandboxId." + echo "Results: $PASSED passed, $FAILED failed" + exit 1 +fi + +case "$SANDBOX_ID" in + lxc:mxc-*) check "sandboxId has the lxc:mxc- prefix ($SANDBOX_ID)" 0 ;; + *) check "sandboxId has the lxc:mxc- prefix ($SANDBOX_ID)" 1 ;; +esac + +# --- start ----------------------------------------------------------------- +echo "=== start ===" +run_phase start "$SANDBOX_ID" +check "start exits 0" $? + +# --- exec ------------------------------------------------------------------ +# Exec relays the script's own exit code rather than an envelope, so these two +# assert the code directly: a successful command and a deliberate failure. +echo "=== exec (success) ===" +run_phase exec "$SANDBOX_ID" '"process": { "commandLine": "echo hello-from-lxc" }' +check "exec relays exit code 0" $? + +echo "=== exec (nonzero) ===" +run_phase exec "$SANDBOX_ID" '"process": { "commandLine": "exit 7" }' +EXEC_RC=$? +if [ "$EXEC_RC" = "7" ]; then + check "exec relays a nonzero exit code (got 7)" 0 +else + check "exec relays a nonzero exit code (got $EXEC_RC, want 7)" 1 +fi + +# --- stop ------------------------------------------------------------------ +echo "=== stop ===" +run_phase stop "$SANDBOX_ID" +check "stop exits 0" $? + +# --- deprovision ----------------------------------------------------------- +echo "=== deprovision ===" +run_phase deprovision "$SANDBOX_ID" +DEPROVISION_RC=$? +check "deprovision exits 0" "$DEPROVISION_RC" +# Release the ID only once the container is actually gone. Clearing it after a +# failed deprovision disarms the trap's retry, so the container the test could +# not remove leaks into the next run. +if [ "$DEPROVISION_RC" -eq 0 ]; then + SANDBOX_ID="" +fi + +echo "================================" +echo "Results: $PASSED passed, $FAILED failed" +if [ "$FAILED" -gt 0 ]; then + exit 1 +fi