Report a timeout and a refused kill through the state-aware exec handle - #874
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Updates state-aware exec to report timeouts and termination refusals while preventing unsupported library execs from running workloads.
Changes:
- Adds
ExecOutcomeand fallible termination. - Improves IsolationSession timeout handling.
- Rejects unsupported in-process exec before execution.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
state_aware_dispatch.rs |
Adapts executor relay to new outcomes. |
state_aware_backend.rs |
Defines updated exec contracts. |
exec_stream.rs |
Propagates timeout and kill results. |
wslc/common/src/state_aware.rs |
Rejects library exec early. |
windows_sandbox/lifecycle/src/state_aware.rs |
Rejects library exec early. |
isolation_session/common/src/state_aware.rs |
Reports timeout and termination results. |
process_options.rs |
Adds service timeout grace. |
manager.rs |
Classifies timeout versus exit. |
mxc-state-aware-sandbox-api.md |
Documents the new API contract. |
state-aware-rust.md |
Documents IsolationSession behavior. |
Suppressed comments (2)
src/core/wxc_common/src/exec_stream.rs:215
- The waiter-thread spawn failure path also ignores a reported kill refusal and then invokes the waiter inline. With an infinite process wait, the API call never returns—the exact refusal-induced teardown hang this PR is intended to prevent. Only reap after an accepted termination, and handle ownership explicitly when no waiter thread can be created.
let _ = terminator();
if let Some(waiter) = waiter_slot.lock().ok().and_then(|mut w| w.take()) {
let _ = waiter();
src/core/wxc_common/src/exec_stream.rs:328
- If the discard thread cannot start and the terminator refuses the kill,
join_waiter()can block forever (especially whenscriptTimeoutis zero). Check the terminator result here; on refusal, retain that state and detach the existing waiter instead of joining it, matching the bounded behavior implemented inDrop.
if let Some(terminator) = self.terminator.take() {
// Discarded: the drain-spawn failure is the error being returned.
let _ = terminator();
}
let _ = self.join_waiter();
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
`ExecHandle`'s two closures could not express either answer. The waiter returned a bare exit code, so a workload the backend killed at its deadline was indistinguishable from one that exited with whatever code the kill produced. The terminator returned nothing, so a platform that refused to terminate a process had no way to say so, and the caller could only assume it had worked. The waiter now yields `ExecOutcome::Exited(i32) | TimedOut`. A timeout is an outcome rather than an error: the deadline was spent while the process ran and the process is no longer running, whereas `Err` means the exit could not be determined at all. Those are different problems with different responses, and collapsing them costs the caller the ability to tell them apart. The terminator now returns `Result<(), MxcError>`. `TimedOut` deliberately does not claim the backend killed the process. A workload that overruns its deadline and then exits on its own has still missed it, and reporting the exit code it happened to produce would hide that. How far "gone" reaches is left to the backend: one that owns a process tree kills the tree, one whose only primitive is the foreground process confirms that process. Only an in-process caller can observe `TimedOut`. `ScriptResponse` carries an exit code and no timeout field, so the executor has nowhere to put one; giving the CLI a timeout channel is a change to its output contract and belongs with that work rather than here. Also states the contract for a backend that cannot serve an in-process caller at all: it must refuse before running anything, since the workload is arbitrary and may not be idempotent, and a refusal issued afterwards describes side effects that have already happened. Types only. This does not compile on its own -- the implementations and the two consumers follow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
Both backends relay the workload's output to this process's own stdout and stderr and return sentinel handles, so neither can hand an in-process caller real streams. Both previously ignored the consumer entirely: they ran the workload to completion, wrote its output wherever the backend chose, and returned a handle the adapter then rejected. The caller was told the operation was unsupported about a command that had already run and whose output had already gone somewhere it never asked for -- and a caller that retried ran it twice. Both now refuse an in-process caller as the first statement of `exec`, before touching a daemon or validating an id. The refusal states that nothing has been run, which is the part that makes it actionable. Behaviour for the executor is unchanged: the waiter reports `Exited` with the code the relay captured, and the terminator succeeds because there is nothing left to terminate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
This backend can now answer both questions the handle asks, which took undoing two ways the answers were previously unobtainable. The timeout was never observable. The service arms its own per-process timer at process creation, from the same `process.timeout` the caller supplied, and enforces it by killing the process with an ordinary exit code -- the host suite pins it as exit code 1. Our own wait starts later with an equal duration, so the service always won the race and a genuine timeout arrived looking exactly like a normal exit. The streaming path now arms the service timer with a margin, making the caller's deadline the one that fires first and leaving the service timer as a watchdog for the case this process dies. A deadline too large to move behind disarms that watchdog rather than saturating into equality with the caller's deadline, since `process.timeout` is an unconstrained `u32` on the wire and saturating would silently shrink the margin back to nothing. The margin is an ordering heuristic that holds while process start-up stays under it, documented as such: nothing in the interface distinguishes a service-enforced kill from an ordinary exit. Neither remaining signal proves a timeout alone. `WaitForExit` answers `-1` on timeout and `ExitCode()` reads `STILL_ACTIVE` (259) for a process that has not exited, and both are legal exit codes for untrusted code. Their conjunction establishes whether the process was still running when it was sampled, which is what the shutdown decision needs. A zero timeout means INFINITE and short-circuits the test entirely -- a wait with no deadline cannot have missed one, and that is the default. A spent deadline is sticky. If the wait returns its sentinel and the process is then observed to have exited with some other code, that later code cannot be what the wait reported -- so the sentinel was real, the deadline elapsed while the process ran, and reporting the exit code would hide a missed deadline from a caller who asked for one. The sibling WSLc backend already draws this line, tracking `deadline_elapsed` separately from `timed_out` precisely so a timeout is reported stickily rather than as a later-observed exit code. The `-1`/`-1` collision stays an exit: nothing there distinguishes a timeout from a workload that exited with `-1`. An exited process is never routed through the three-tier shutdown ladder. The ladder reads only `ExitCode()`, so it cannot tell an exit of 259 from a live process; running it over an exit that had already been established made a clean exit of 259 indistinguishable from a survivor. It now runs only once a survivor has been established independently, and the decision is a value rather than a branch so it can be tested at all -- the function holding it needs a live process object this host cannot provide. Both timeout paths confirm the foreground process only, because that is the only thing this API exposes. A descendant the workload backgrounded outlives a reported timeout and is reclaimed when the session is stopped and deprovisioned; stated rather than implied. The terminator reports whether the platform accepted the kill. What it still cannot say is whether the process actually died, which would need the bounded post-kill wait's result that the handle type does not carry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
The streaming adapter maps a timeout onto `io::ErrorKind::TimedOut`, which is the convention five one-shot backends already use and which `mxc_sdk::Sandbox::wait` reads as `WaitOutcome::TimedOut`. Translating at this boundary is what lets the state-aware path reach that outcome without changing anything above it. The outcome is cached rather than the exit code, because a timeout has no code to cache and a repeat wait must still be idempotent. A refused kill now reaches the caller, and is remembered. The terminator is consumed whether it succeeded or failed, so its absence alone cannot say which happened -- and reading that absence as "already killed" made `Drop` join a waiter parked on a live process, blocking forever in a destructor the caller cannot opt out of. A retried kill keeps reporting the refusal rather than claiming success, since nothing has changed. Every teardown path makes the same choice, through one helper rather than four open-coded copies: terminate, and reap only if the kill was accepted. Reaping after a refused kill is what parks the caller, and `script_timeout` defaults to zero -- INFINITE -- so that is the default configuration. Two of those paths run during construction, before the caller holds a handle it could interrupt; one runs inside `wait`, where blocking reintroduces the exact stall it exists to avoid. The fourth is the all-null backstop, which cannot assume the workload has finished: the IsolationSession library path starts a process and passes through whatever handles the service returned, so an all-zero set can name a live one. Abandoning the waiter leaks the thread, its share of the process object and a COM reference. That is the better trade against an unbounded hang in a case that already means the sandbox is not responding to termination. A confirmed exit retires all of it. Once the waiter has reported an outcome there is nothing to kill, so `kill` succeeds without running the terminator and an earlier refusal no longer applies -- otherwise a process that refused a kill and then exited on its own kept reporting a refusal to terminate something demonstrably gone. A handle with nothing on any stream is refused rather than wrapped, as a backstop behind the up-front refusal: by the time a handle exists the workload may have run, so this cannot undo its side effects, and reaching it means a backend returned an executor-shaped handle for an in-process caller anyway. The executor relay refuses a timeout instead of inventing an exit code for it. That is unreachable by construction -- every backend serving the executor has finished the workload before returning -- but a typed error is the right answer if it ever becomes reachable, rather than a fabricated exit status. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
The IsolationSession design note described the closure shapes that no longer exist, said making the terminator fallible was future work, and stated that teardown joins the waiter unconditionally. It also described the waiter as reporting a timeout only while the process was still running, and the service-side timer as a plain per-process reaper -- both superseded by the sticky deadline and the watchdog margin. The state-aware API document carried the same superseded closure signatures and the same "the backend killed the process" reading of a timeout. Before this change a state-aware exec could not report a timeout at all: `ErrorKind::TimedOut` appears nowhere in the adapter on the base commit, so every "the tree is killed" statement was true, satisfied by the process-spawning backends alone -- those lead their own process group on Unix and get a job object on Windows. Making the outcome reachable from a backend whose only primitive is the foreground process turned four of those statements false, in the `SandboxProcess` trait and in the public Rust SDK. They now say the workload is terminated and leave the reach to the backend, which is the only thing true of both, and still name the tree kill where it is real. The narrowed note also pointed at `exec_sandbox` as though a caller could take that route today. It cannot: the parser hardcodes `experimental_enabled` to false, so an experimental backend is refused before dispatch. The distinction still belongs in the doc -- it is what implementors build against, and it becomes observable the moment the opt-in lands -- but it is now stated as gated rather than as available. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
b416bb1 to
2ddfff1
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Suppressed comments (5)
src/core/wxc_common/src/exec_stream.rs:549
- This message contradicts the new outcome contract:
TimedOutAfterDeadlinereturnsTimedOutwhen the process exited on its own after the deadline, so the sandbox did not necessarily terminate it. Preserve the distinction in the observable error text.
"the exec timed out and the sandbox terminated it",
src/core/wxc_common/src/exec_stream.rs:472
- A process can finish and its waiter thread can return before the caller invokes
wait/try_wait; in that stateself.exitis stillNone, sokill()invokes the fallible terminator against an already-dead process and can report a spurious refusal. The previous infallible terminator hid this race. Join an already-finished waiter before terminating, and ensure the terminator treats a concurrent natural exit as success.
if self.exit.is_some() {
self.terminator = None;
self.kill_refused = false;
return Ok(());
}
src/core/wxc_common/src/exec_stream.rs:90
- This field comment still promises a process-tree kill, but the newly supported IsolationSession terminator only reaches the foreground process. Describe this as backend-defined termination so the internal contract matches
ExecOutcomeand the implementation.
terminator: Option<Box<dyn FnOnce() -> Result<(), MxcError> + Send>>,
src/core/wxc_common/src/sandbox_process.rs:129
- This caveat only qualifies timeout handling;
SandboxProcess::killabove still unconditionally promises termination of the process and all descendants.ExecSandboxProcess::killnow invokes an IsolationSession terminator that can only target the foreground process, so the trait's kill contract remains false and should carry the same backend-specific reach qualification.
/// **How far that termination reaches is the implementation's to state.**
/// The process-spawning implementations kill the whole tree — the child
/// leads its own process group on Unix and is assigned a job object on
/// Windows. An implementation whose only primitive is the foreground
/// process (the state-aware exec adapter, over a backend with no tree API)
src/core/mxc-sdk/src/sandbox.rs:37
- The new backend-specific qualification conflicts with this public type's existing kill promises: the
Sandboxrustdoc still says “and its whole tree” andSandbox::killstill says “Kill the child and its process tree.” The sameSandboxwraps state-aware exec handles, so update both statements to describe backend-defined termination reach.
/// How far "gone" reaches depends on the backend. A process-spawning
/// backend kills the whole tree. A backend whose only primitive is the
/// foreground process — the state-aware `exec` path over IsolationSession —
/// confirms that process, and a descendant the workload backgrounded is
/// reclaimed when the sandbox is stopped and deprovisioned rather than here.
Resolves the collision with microsoft#798, which rewrote the same LXC network and attach paths this branch changes, and adapts to the exec-handle contract microsoft#874 introduced. network_iptables.rs -- FORWARD teardown now branches on whether the veth interface is known. microsoft#798 deletes by replaying the full remembered rule spec, which needs the interface name; this branch enumerates FORWARD for jumps to the container chain, which does not. Both are kept, because neither covers the other's case. With the veth the specs are replayed for all four rule classes, which is the only route that reaches return_rule and physdev_return -- those jump to ACCEPT rather than to the chain, so no search for jumps into the chain can see them and the chain flush cannot reclaim them. Without the veth, signal-time force_cleanup enumerates instead of matching nothing and leaking the chain and its hooks permanently. Enumeration is deliberately not used when the veth is known: an absent tool answers that no hooks remain, and that verdict gates the -F, so it would flush a still-hooked chain into a fail-open. lxc_bindings.rs -- build_attach_args_with_env_control now carries both sides' additions, force_clear_env from microsoft#798 and the exec marker from this branch. Both test suites are kept. state_aware.rs -- adapted to microsoft#874. This file does not exist on main and had no conflict markers, but main changed ExecHandle underneath it. The waiter now reports ExecOutcome::Exited(exit_code) and the terminator Ok(()), matching wslc, windows_sandbox, and isolation_session, each of which blocks on a backend that already holds the exit code. lxc_runner.rs -- resolved from line-ending-normalized blobs. microsoft#798 committed this file with CRLF while every other .rs in the tree is LF, which turned the whole file into a single conflict; .gitattributes pins only *.sh. The merged file is LF. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acf3853b-fbe3-41ab-870a-9274a698cc0c
microsoft#874 added the consumer argument to the state-aware exec contract, and the merge adapted the signature without honoring it. attach_run relays the workload's output to this process's stdio and returns no pipes, so LXC is one of the backends that cannot serve an in-process caller. Returning a handle with three null pipes is a contract violation: by the time the caller sees it the script has already run and its output has already gone somewhere it never asked for. Refuses through the shared unsupported_library_exec, ahead of the id parse and both container probes, matching WSLc and Windows Sandbox. The test passes an id that would fail as MalformedId, so it fails if the check ever moves after it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: acf3853b-fbe3-41ab-870a-9274a698cc0c
📖 Description
ExecHandle's two closures could not express two things a backend knows. The waiter returned a bare exit code, so a workload killed at its deadline was indistinguishable from one that exited with whatever code the kill produced. The terminator returned nothing, so a platform that refused to terminate had no way to say so and the caller could only assume it worked.The waiter now yields
ExecOutcome::Exited(i32) | TimedOut, and the terminator returnsResult<(), MxcError>. A timeout is an outcome — the deadline was spent while the process ran and the process is no longer running — whereasErrmeans the exit could not be determined; collapsing them costs the caller the ability to tell them apart.TimedOutdeliberately does not claim the backend killed the process: a workload that overruns its deadline and then exits on its own has still missed it, and reporting the code it happened to produce would hide that. Only an in-process caller can observeTimedOut:ScriptResponsehas no timeout field, so the executor keeps reportingExitedand the CLI is unchanged.Two defects surfaced while making the IsolationSession backend answer honestly, both of which would have shipped silently:
The timeout could never be reported. The service arms its own per-process timer at creation from the same
process.timeoutand enforces it by killing with an ordinary exit code — the host suite pins it as exit code 1. Our wait started later with an equal duration, so the service always won and a genuine timeout arrived looking exactly like a normal exit. The streaming path now arms the service timer with a margin, making the caller's deadline fire first and leaving the service timer as a watchdog. A deadline too large to move behind disarms that watchdog rather than saturating back into equality. The margin is documented as an ordering heuristic, not a guarantee: nothing in the interface distinguishes a service-enforced kill from an ordinary exit.Exit code 259 was ambiguous twice.
ExitCode()readsSTILL_ACTIVE(259) for a live process, but 259 is also a legal exit code, so neither that norWaitForExit's-1proves a timeout alone. Their conjunction establishes whether the process was live when sampled. The same ambiguity lived one function away in the shutdown ladder, which reads onlyExitCode()— an exited process is no longer routed through it. A spent deadline is also sticky: if the wait returns its sentinel and the process is then seen to have exited with some other code, that code cannot be what the wait reported, so the deadline provably elapsed and reporting the exit would hide it. The sibling WSLc backend already draws this line, trackingdeadline_elapsedseparately fromtimed_out.Backends that cannot serve an in-process caller now refuse before running anything. WSLc and Windows Sandbox previously ignored the consumer, ran the workload, wrote its output to the host process's own stdout, and only then returned a handle the adapter rejected — so the caller was told "unsupported" about side effects that had already happened, and a retry ran the command twice.
Every teardown path now terminates and reaps only if the kill was accepted, through one helper rather than four open-coded copies. Reaping after a refused kill blocks on a waiter that may never return, and
script_timeoutdefaults to zero — INFINITE — so that is the default configuration. Two of those paths run during construction, before the caller holds a handle it could interrupt; one runs insidewait, where blocking reintroduces the stall it exists to avoid; the fourth is the all-null backstop, which cannot assume the workload finished, since the IsolationSession library path passes through whatever handles the service returned. Abandoning the waiter leaks a thread and a COM reference — the better trade against an unbounded hang in a case that already means the sandbox is not responding. A confirmed exit retires the terminator entirely.Closes three of the four gaps #839 documented: the terminator now reports refusal,
WaitOutcome::TimedOutis reachable, and teardown is bounded against a reported refusal. Untaken stdin is still only dropped. Still unbounded: an accepted kill that never took effect, which needs the backend to confirm the process died.Before this change a state-aware exec could not report a timeout at all, so four statements in the
SandboxProcesstrait and the public Rust SDK promising a whole-tree kill were true, satisfied by the process-spawning backends alone. Making the outcome reachable from a backend whose only primitive is the foreground process turned them false; they now leave the reach to the backend and still name the tree kill where it is real.🔗 References
Continues #829 and #839.
Related Issues
execnow returns a typed refusal forExecConsumer::Libraryinstead of silently returning null pipes and blocking. That is not the real-pipe streaming the issue asks for, but it does loosen its "do not land #1 without Adding Microsoft SECURITY.MD #2" coupling — if the experimental opt-in lands first, the FFI path gets a clean error rather than dead pipes. Defects 1 and 3 are untouched.🔍 Validation
Full local gate green on this tree:
fmt,clippy --workspace --all-targets -D warnings, build and test withisolation_sessionon and off, arm64 build, versioning scripts, C# SDK, Node SDK unit + integration, and the elevatedwxc_host_preptests.Every behavioural claim is mutation-verified: 25 mutations, 23 detected. The two survivors are recorded rather than quietly dropped — the shutdown ladder's post-kill survivor check and the waiter-thread-spawn site, each needing something a unit test cannot arrange (a live process object; an OS refusing a thread). Mutations are checked for compiling, because one that fails to build exits non-zero and would otherwise read as a caught regression.
On a host with the OS-side isolation service: end-to-end suites 90 passed / 0 failed / 0 skipped across all three harnesses, an empty account-leak delta corroborated by
IsoSessionClireporting zero agent users, and the interactive terminal tests passed at the console.✅ Checklist
📋 Issue Type
Microsoft Reviewers: Open in CodeFlow