[LXC] State-aware sandbox lifecycle

[Darren Hoehna (dhoehna)]

📖 Description

Implements StatefulSandboxBackend for LXC, so a container can be driven through provision, start, exec, stop, and deprovision rather than only the one-shot spawn path.

• Adds the five lifecycle phases, with start serialized per container under an advisory flock.
• Adds network policy enforcement in both directions: the egress FORWARD chain in the host netns before the container starts, and the inbound INPUT chain inside the container's own netns once it is running.
• Adds ContainerPolicy::requires_firewall(), so enforcement follows the policy — a block default, a non-empty host list, or a proxy — and a start with no network section denies by default.
• Adds lxc to the Node SDK as a state-aware backend, with per-phase config types and LxcNetworkConfig narrowed to the three fields LXC enforces.
• Adds two E2E scripts run against a live LXC host: the provision-through-deprovision lifecycle, and a network matrix over 15 cases and 16 checked-in configs.

🔍 Validation

• cargo test --workspace --no-fail-fast — 2,785 passed, 2 failed. Both failures are pre-existing wslc_common tests that require an unlocked D: drive and reproduce at the base commit.
• npm test (Node SDK) — 283 tests, 277 passed, 0 failed, 6 skipped.
• cargo fmt --all -- --check and cargo clippy --workspace --all-targets — clean.
• run_lxc_state_aware_network_test.sh on a live LXC host — 70 passed, 0 failed, 0 quarantined.
• run_lxc_state_aware_test.sh on a live LXC host — 8 passed, 0 failed.

✅ Checklist

• Signed the Contributor License Agreement
• Linked to an issue
• Updated documentation (if applicable)
• Updated Copilot instructions (if build, architecture, or conventions changed)
• If this PR changes Cargo.lock, the dependency-feed-check check passes

📋 Issue Type

• Bug fix
• Feature
• Task

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

tests/scripts/run_lxc_state_aware_network_test.sh:743

• This case never verifies that allowedHosts produces an ACCEPT rule: it only checks that the chain is hooked and ends in DROP, which is identical to default-deny with the allow list ignored. Add a positive connectivity check to a controlled allowed endpoint (plus a denied control), or inspect the destination-specific ACCEPT rule; case 12 has the same gap.
  tests/scripts/run_lxc_state_aware_network_test.sh:305
• A failed deprovision still clears SANDBOX_ID, so the EXIT trap cannot retry cleanup and the container leaks into later cases/runs. Preserve the ID unless deprovision succeeds, as the lifecycle test already does in tests/scripts/run_lxc_state_aware_test.sh:168-175.
  tests/scripts/run_lxc_state_aware_network_test.sh:793
• This blocklist assertion is vacuous because omitting defaultPolicy inherits default block, so evil.example.com is denied even if blockedHosts is completely ignored. Exercise the list under defaultPolicy: allow and verify both the blocked destination and an allowed control, or inspect the explicit destination DROP rule.

[Copilot]

Pull request overview

Copilot reviewed 56 out of 57 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk/node/src/state-aware-types.ts:140

• FilesystemConfig also exposes the SDK-only clearPolicyOnExit field. buildStateAwareEnvelope forwards this object unchanged, while Rust's wire::Filesystem uses deny_unknown_fields and has no such member, so a configuration accepted by this new public type fails at runtime as malformed. Narrow the filesystem type just as the network type is narrowed.

  filesystem?: FilesystemConfig;

tests/scripts/run_lxc_state_aware_network_test.sh:307

• The sandbox ID is cleared even when deprovision fails, which disarms the EXIT trap and can leave a container behind. The same issue occurs for SANDBOX_STARTED: a failed stop is recorded as stopped, so cleanup will not retry it. Retain both states until their corresponding phase succeeds.

[Copilot]

Pull request overview

Copilot reviewed 63 out of 65 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 63 out of 65 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk/node/src/state-aware-helper.ts:40

• The SDK adds LXC to the state-aware version/prefix registries and envelope shape, but sdk/node/tests/unit/state-aware.test.ts has runtime envelope/routing coverage for the other backends and none for LXC. Add tests that provision emits top-level containerId plus nested experimental.lxc.provision, and that subsequent lxc: phases route correctly; otherwise a registry or lifting regression can pass the current compile-time-only checks.

  lxc: STATE_AWARE_VERSION,

tests/scripts/run_lxc_state_aware_network_test.sh:305

• A failed deprovision is still followed by clearing SANDBOX_ID, which disarms the EXIT trap's retry and can leak the container/firewall state into subsequent cases. Preserve the ID unless deprovision succeeds, as the lifecycle script already does.

[Copilot]

Pull request overview

Copilot reviewed 65 out of 67 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 65 out of 67 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

src/backends/lxc/common/src/signal_cleanup.rs:405

• A signal can arrive after the egress chain is created but before container.start(). In that state lxc-stop -k reports failure because the container is already stopped, so execute_rollback aborts before RemoveFirewall and strands the chain; subsequent starts then fail on the existing chain. Treat a positively stopped container as a successful stop step, while still failing closed when the state probe is unknown.

                RollbackStep::StopContainer => LxcContainer::new(&name, None).kill().is_ok(),

tests/scripts/run_lxc_state_aware_network_test.sh:305

• The sandbox ID is cleared even when deprovision fails, so the EXIT trap cannot retry cleanup and the failed case leaks its container/firewall state onto the test host. Keep the ID until deprovision succeeds, matching the lifecycle test's cleanup behavior.

[Copilot]

Pull request overview

Copilot reviewed 66 out of 68 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/backends/lxc/common/src/state_aware.rs:948

• A concurrent exec can pass is_running() immediately after container.start() and run untrusted code before this ingress policy is installed (or while installation is about to fail). The lifecycle lock excludes stop/deprovision but exec does not participate, so successful start is not an atomic state transition from an exec caller's perspective. Use shared locking for exec and exclusive locking for lifecycle transitions, or persist/check a fully-started marker before allowing attach.

            // Inbound enforcement lands only once the container's network
            // namespace exists, so unlike the egress chain it comes after start.
            if let Err(e) = apply_ingress_policy(&container, container_name, request, &mut logger) {

src/backends/lxc/common/src/lxc_runner.rs:379

• On ingress setup failure, an adopted container that this invocation started with destroyOnExit=false is left running because these branches only destroy newly owned/disposable containers. It therefore survives a failed run without the required INPUT policy (the missing-PID branch below has the same issue). Track whether this invocation started the container and stop/kill it before returning any post-start policy error; preservation should avoid destroying the container, not leave a failed start running.

                match mgr.apply_firewall_rules(&request.policy, logger) {
                    Ok(true) => {}
                    Ok(false) => {
                        if self.destroy_on_exit || container_created {
                            let _ = container.destroy();
                        }
                        return ScriptResponse::error(
                            "Failed to apply inbound network firewall rules.",
                        );
                    }
                    Err(e) => {
                        if self.destroy_on_exit || container_created {
                            let _ = container.destroy();

src/core/wxc_common/src/wire.rs:590

• Adding this typed field makes one-shot lxc-exec accept experimental.lxc.provision and silently ignore its distribution/release. The CLI now parses both paths through load_mxc_request; its one-shot conversion consumes the other experimental fields but neither maps nor rejects raw_exp.lxc. The adapter-only rejection test does not exercise that path. Explicitly reject this state-aware-only section during one-shot conversion and add a load_mxc_request regression test.

[Copilot]

Pull request overview

Copilot reviewed 66 out of 68 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 66 out of 68 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

src/backends/lxc/common/src/lxc_runner.rs:299

• pinnable == false still starts the container before applying the firewall. An adopted container with one macvlan/phys interface takes this branch, can transmit during startup, and only afterward fails because live_veth_interface recognizes veth names only. Reject any non-pinnable topology before container.start() whenever the policy requires a firewall, as the state-aware path already does.

        if !running && request.policy.requires_firewall() {
            let pinnable = net_config
                .as_ref()
                .map(Self::pinnable_before_start)
                .unwrap_or(false);
            if pinnable {

src/backends/lxc/common/src/lxc_runner.rs:339

• Now that egress may already be installed, a nonzero lxc-start result is not safe to handle by immediately returning. As the state-aware path notes, start can fail after the container is already live; if destroy is skipped or the ignored destroy fails, fw_manager then drops and removes the chain, leaving that container running unfiltered. Probe the resulting state and kill/stop it before allowing firewall cleanup; preserve the rules if shutdown cannot be confirmed.

            if let Err(e) = container.start() {
                if self.destroy_on_exit || container_created {
                    let _ = container.destroy();
                }
                return ScriptResponse::error(&format!("Failed to start container: {}", e));

src/core/wxc_common/src/wire.rs:590

• Because wire::MxcConfig is shared by both request shapes, this new typed field also makes experimental.lxc valid on one-shot requests. The production one-shot path deserializes directly into MxcConfig (config_parser.rs:246-252) and convert_wire_config never consumes or rejects raw_exp.lxc (config_parser.rs:1249-1340), so a caller's requested image is silently ignored. The added adapter test does not cover production parsing; explicitly reject this section for one-shot requests or use request-specific wire types.
  tests/scripts/run_lxc_state_aware_network_test.sh:310
• This clears SANDBOX_ID even when deprovision fails, which disarms the EXIT trap's retry and leaks the container/firewall into later matrix cases. Capture the return code and only release the ID after successful deprovision, matching run_lxc_state_aware_test.sh:168-175.

[Copilot]

Pull request overview

Copilot reviewed 66 out of 68 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 66 out of 68 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 65 out of 67 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

tests/scripts/run_lxc_state_aware_network_test.sh:316

• Retaining the ID on failure is not enough because this function returns and the next case's start_fresh_sandbox overwrites SANDBOX_ID. The EXIT trap then loses the only reference to this sandbox and cannot retry its cleanup. Abort immediately on failed deprovision so the trap retries the same ID (or track all outstanding IDs).

src/core/wxc_common/src/wire.rs:590

• Adding this field makes experimental.lxc deserialize successfully on the production one-shot path, but convert_wire_config neither consumes nor rejects raw_exp.lxc, and present_backend_sections does not count it. A one-shot request can therefore specify a provision image here, be accepted, and then silently run with the top-level LXC defaults. The new contract-adapter test does not exercise the production parser; explicitly reject this field during one-shot conversion and add a production-parser regression test.

[Copilot]

Pull request overview

Copilot reviewed 65 out of 67 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/backends/lxc/common/src/state_aware.rs:665

• Returning an unlocked guard when the LXC root is absent breaks the provision critical section. provision calls this same helper before its is_defined/create sequence, so two first provisions on a host with no root can both observe an unused name and concurrently run lxc-create. Keep stop/deprovision idempotent, but ensure provision locks in a durable parent before probing or creating the container.

                    Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
                        return Ok(LifecycleLock { _guard: None })
                    }