Skip to content

Report a timeout and a refused kill through the state-aware exec handle - #874

Merged
adpa-ms merged 5 commits into
mainfrom
user/adibpa/copilot-exechandle-library-contract
Aug 14, 2026
Merged

Report a timeout and a refused kill through the state-aware exec handle#874
adpa-ms merged 5 commits into
mainfrom
user/adibpa/copilot-exechandle-library-contract

Conversation

@adpa-ms

@adpa-ms adpa-ms commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

📖 Description

ExecHandle's two closures could not express two things a backend knows. The waiter returned a bare exit code, so a workload killed at its deadline was indistinguishable from one that exited with whatever code the kill produced. The terminator returned nothing, so a platform that refused to terminate had no way to say so and the caller could only assume it worked.

The waiter now yields ExecOutcome::Exited(i32) | TimedOut, and the terminator returns Result<(), MxcError>. A timeout is an outcome — the deadline was spent while the process ran and the process is no longer running — whereas Err means the exit could not be determined; collapsing them costs the caller the ability to tell them apart. TimedOut deliberately does not claim the backend killed the process: a workload that overruns its deadline and then exits on its own has still missed it, and reporting the code it happened to produce would hide that. Only an in-process caller can observe TimedOut: ScriptResponse has no timeout field, so the executor keeps reporting Exited and the CLI is unchanged.

Two defects surfaced while making the IsolationSession backend answer honestly, both of which would have shipped silently:

The timeout could never be reported. The service arms its own per-process timer at creation from the same process.timeout and enforces it by killing with an ordinary exit code — the host suite pins it as exit code 1. Our wait started later with an equal duration, so the service always won and a genuine timeout arrived looking exactly like a normal exit. The streaming path now arms the service timer with a margin, making the caller's deadline fire first and leaving the service timer as a watchdog. A deadline too large to move behind disarms that watchdog rather than saturating back into equality. The margin is documented as an ordering heuristic, not a guarantee: nothing in the interface distinguishes a service-enforced kill from an ordinary exit.

Exit code 259 was ambiguous twice. ExitCode() reads STILL_ACTIVE (259) for a live process, but 259 is also a legal exit code, so neither that nor WaitForExit's -1 proves a timeout alone. Their conjunction establishes whether the process was live when sampled. The same ambiguity lived one function away in the shutdown ladder, which reads only ExitCode() — an exited process is no longer routed through it. A spent deadline is also sticky: if the wait returns its sentinel and the process is then seen to have exited with some other code, that code cannot be what the wait reported, so the deadline provably elapsed and reporting the exit would hide it. The sibling WSLc backend already draws this line, tracking deadline_elapsed separately from timed_out.

Backends that cannot serve an in-process caller now refuse before running anything. WSLc and Windows Sandbox previously ignored the consumer, ran the workload, wrote its output to the host process's own stdout, and only then returned a handle the adapter rejected — so the caller was told "unsupported" about side effects that had already happened, and a retry ran the command twice.

Every teardown path now terminates and reaps only if the kill was accepted, through one helper rather than four open-coded copies. Reaping after a refused kill blocks on a waiter that may never return, and script_timeout defaults to zero — INFINITE — so that is the default configuration. Two of those paths run during construction, before the caller holds a handle it could interrupt; one runs inside wait, where blocking reintroduces the stall it exists to avoid; the fourth is the all-null backstop, which cannot assume the workload finished, since the IsolationSession library path passes through whatever handles the service returned. Abandoning the waiter leaks a thread and a COM reference — the better trade against an unbounded hang in a case that already means the sandbox is not responding. A confirmed exit retires the terminator entirely.

Closes three of the four gaps #839 documented: the terminator now reports refusal, WaitOutcome::TimedOut is reachable, and teardown is bounded against a reported refusal. Untaken stdin is still only dropped. Still unbounded: an accepted kill that never took effect, which needs the backend to confirm the process died.

Before this change a state-aware exec could not report a timeout at all, so four statements in the SandboxProcess trait and the public Rust SDK promising a whole-tree kill were true, satisfied by the process-spawning backends alone. Making the outcome reachable from a backend whose only primitive is the foreground process turned them false; they now leave the reach to the backend and still name the tree kill where it is real.

🔗 References

Continues #829 and #839.

Related Issues

🔍 Validation

Full local gate green on this tree: fmt, clippy --workspace --all-targets -D warnings, build and test with isolation_session on and off, arm64 build, versioning scripts, C# SDK, Node SDK unit + integration, and the elevated wxc_host_prep tests.

Every behavioural claim is mutation-verified: 25 mutations, 23 detected. The two survivors are recorded rather than quietly dropped — the shutdown ladder's post-kill survivor check and the waiter-thread-spawn site, each needing something a unit test cannot arrange (a live process object; an OS refusing a thread). Mutations are checked for compiling, because one that fails to build exits non-zero and would otherwise read as a caught regression.

On a host with the OS-side isolation service: end-to-end suites 90 passed / 0 failed / 0 skipped across all three harnesses, an empty account-leak delta corroborated by IsoSessionCli reporting zero agent users, and the interactive terminal tests passed at the console.

✅ Checklist

  • Signed the Contributor License Agreement
  • Linked to an issue
  • Updated documentation (if applicable)

📋 Issue Type

  • Task
Microsoft Reviewers: Open in CodeFlow

Copilot AI balanced review requested due to automatic review settings August 14, 2026 16:33
@adpa-ms
adpa-ms requested a review from a team as a code owner August 14, 2026 16:33
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates state-aware exec to report timeouts and termination refusals while preventing unsupported library execs from running workloads.

Changes:

  • Adds ExecOutcome and fallible termination.
  • Improves IsolationSession timeout handling.
  • Rejects unsupported in-process exec before execution.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
state_aware_dispatch.rs Adapts executor relay to new outcomes.
state_aware_backend.rs Defines updated exec contracts.
exec_stream.rs Propagates timeout and kill results.
wslc/common/src/state_aware.rs Rejects library exec early.
windows_sandbox/lifecycle/src/state_aware.rs Rejects library exec early.
isolation_session/common/src/state_aware.rs Reports timeout and termination results.
process_options.rs Adds service timeout grace.
manager.rs Classifies timeout versus exit.
mxc-state-aware-sandbox-api.md Documents the new API contract.
state-aware-rust.md Documents IsolationSession behavior.
Suppressed comments (2)

src/core/wxc_common/src/exec_stream.rs:215

  • The waiter-thread spawn failure path also ignores a reported kill refusal and then invokes the waiter inline. With an infinite process wait, the API call never returns—the exact refusal-induced teardown hang this PR is intended to prevent. Only reap after an accepted termination, and handle ownership explicitly when no waiter thread can be created.
                let _ = terminator();
                if let Some(waiter) = waiter_slot.lock().ok().and_then(|mut w| w.take()) {
                    let _ = waiter();

src/core/wxc_common/src/exec_stream.rs:328

  • If the discard thread cannot start and the terminator refuses the kill, join_waiter() can block forever (especially when scriptTimeout is zero). Check the terminator result here; on refusal, retain that state and detach the existing waiter instead of joining it, matching the bounded behavior implemented in Drop.
        if let Some(terminator) = self.terminator.take() {
            // Discarded: the drain-spawn failure is the error being returned.
            let _ = terminator();
        }
        let _ = self.join_waiter();

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/core/wxc_common/src/exec_stream.rs Outdated
Comment thread src/backends/isolation_session/common/src/manager.rs Outdated
adpa-ms and others added 5 commits August 14, 2026 11:55
`ExecHandle`'s two closures could not express either answer. The waiter returned
a bare exit code, so a workload the backend killed at its deadline was
indistinguishable from one that exited with whatever code the kill produced. The
terminator returned nothing, so a platform that refused to terminate a process
had no way to say so, and the caller could only assume it had worked.

The waiter now yields `ExecOutcome::Exited(i32) | TimedOut`. A timeout is an
outcome rather than an error: the deadline was spent while the process ran and
the process is no longer running, whereas `Err` means the exit could not be
determined at all. Those are different problems with different responses, and
collapsing them costs the caller the ability to tell them apart. The terminator
now returns `Result<(), MxcError>`.

`TimedOut` deliberately does not claim the backend killed the process. A
workload that overruns its deadline and then exits on its own has still missed
it, and reporting the exit code it happened to produce would hide that. How far
"gone" reaches is left to the backend: one that owns a process tree kills the
tree, one whose only primitive is the foreground process confirms that process.

Only an in-process caller can observe `TimedOut`. `ScriptResponse` carries an
exit code and no timeout field, so the executor has nowhere to put one; giving
the CLI a timeout channel is a change to its output contract and belongs with
that work rather than here.

Also states the contract for a backend that cannot serve an in-process caller at
all: it must refuse before running anything, since the workload is arbitrary and
may not be idempotent, and a refusal issued afterwards describes side effects
that have already happened.

Types only. This does not compile on its own -- the implementations and the two
consumers follow.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
Both backends relay the workload's output to this process's own stdout and
stderr and return sentinel handles, so neither can hand an in-process caller
real streams. Both previously ignored the consumer entirely: they ran the
workload to completion, wrote its output wherever the backend chose, and
returned a handle the adapter then rejected. The caller was told the operation
was unsupported about a command that had already run and whose output had
already gone somewhere it never asked for -- and a caller that retried ran it
twice.

Both now refuse an in-process caller as the first statement of `exec`, before
touching a daemon or validating an id. The refusal states that nothing has been
run, which is the part that makes it actionable.

Behaviour for the executor is unchanged: the waiter reports `Exited` with the
code the relay captured, and the terminator succeeds because there is nothing
left to terminate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
This backend can now answer both questions the handle asks, which took undoing
two ways the answers were previously unobtainable.

The timeout was never observable. The service arms its own per-process timer at
process creation, from the same `process.timeout` the caller supplied, and
enforces it by killing the process with an ordinary exit code -- the host suite
pins it as exit code 1. Our own wait starts later with an equal duration, so the
service always won the race and a genuine timeout arrived looking exactly like a
normal exit. The streaming path now arms the service timer with a margin, making
the caller's deadline the one that fires first and leaving the service timer as
a watchdog for the case this process dies. A deadline too large to move behind
disarms that watchdog rather than saturating into equality with the caller's
deadline, since `process.timeout` is an unconstrained `u32` on the wire and
saturating would silently shrink the margin back to nothing. The margin is an
ordering heuristic that holds while process start-up stays under it, documented
as such: nothing in the interface distinguishes a service-enforced kill from an
ordinary exit.

Neither remaining signal proves a timeout alone. `WaitForExit` answers `-1` on
timeout and `ExitCode()` reads `STILL_ACTIVE` (259) for a process that has not
exited, and both are legal exit codes for untrusted code. Their conjunction
establishes whether the process was still running when it was sampled, which is
what the shutdown decision needs. A zero timeout means INFINITE and
short-circuits the test entirely -- a wait with no deadline cannot have missed
one, and that is the default.

A spent deadline is sticky. If the wait returns its sentinel and the process is
then observed to have exited with some other code, that later code cannot be
what the wait reported -- so the sentinel was real, the deadline elapsed while
the process ran, and reporting the exit code would hide a missed deadline from a
caller who asked for one. The sibling WSLc backend already draws this line,
tracking `deadline_elapsed` separately from `timed_out` precisely so a timeout
is reported stickily rather than as a later-observed exit code. The `-1`/`-1`
collision stays an exit: nothing there distinguishes a timeout from a workload
that exited with `-1`.

An exited process is never routed through the three-tier shutdown ladder. The
ladder reads only `ExitCode()`, so it cannot tell an exit of 259 from a live
process; running it over an exit that had already been established made a clean
exit of 259 indistinguishable from a survivor. It now runs only once a survivor
has been established independently, and the decision is a value rather than a
branch so it can be tested at all -- the function holding it needs a live
process object this host cannot provide.

Both timeout paths confirm the foreground process only, because that is the
only thing this API exposes. A descendant the workload backgrounded outlives a
reported timeout and is reclaimed when the session is stopped and deprovisioned;
stated rather than implied.

The terminator reports whether the platform accepted the kill. What it still
cannot say is whether the process actually died, which would need the bounded
post-kill wait's result that the handle type does not carry.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
The streaming adapter maps a timeout onto `io::ErrorKind::TimedOut`, which is
the convention five one-shot backends already use and which
`mxc_sdk::Sandbox::wait` reads as `WaitOutcome::TimedOut`. Translating at this
boundary is what lets the state-aware path reach that outcome without changing
anything above it. The outcome is cached rather than the exit code, because a
timeout has no code to cache and a repeat wait must still be idempotent.

A refused kill now reaches the caller, and is remembered. The terminator is
consumed whether it succeeded or failed, so its absence alone cannot say which
happened -- and reading that absence as "already killed" made `Drop` join a
waiter parked on a live process, blocking forever in a destructor the caller
cannot opt out of. A retried kill keeps reporting the refusal rather than
claiming success, since nothing has changed.

Every teardown path makes the same choice, through one helper rather than four
open-coded copies: terminate, and reap only if the kill was accepted. Reaping
after a refused kill is what parks the caller, and `script_timeout` defaults to
zero -- INFINITE -- so that is the default configuration. Two of those paths run
during construction, before the caller holds a handle it could interrupt; one
runs inside `wait`, where blocking reintroduces the exact stall it exists to
avoid. The fourth is the all-null backstop, which cannot assume the workload has
finished: the IsolationSession library path starts a process and passes through
whatever handles the service returned, so an all-zero set can name a live one.

Abandoning the waiter leaks the thread, its share of the process object and a
COM reference. That is the better trade against an unbounded hang in a case that
already means the sandbox is not responding to termination.

A confirmed exit retires all of it. Once the waiter has reported an outcome
there is nothing to kill, so `kill` succeeds without running the terminator and
an earlier refusal no longer applies -- otherwise a process that refused a kill
and then exited on its own kept reporting a refusal to terminate something
demonstrably gone.

A handle with nothing on any stream is refused rather than wrapped, as a
backstop behind the up-front refusal: by the time a handle exists the workload
may have run, so this cannot undo its side effects, and reaching it means a
backend returned an executor-shaped handle for an in-process caller anyway.

The executor relay refuses a timeout instead of inventing an exit code for it.
That is unreachable by construction -- every backend serving the executor has
finished the workload before returning -- but a typed error is the right answer
if it ever becomes reachable, rather than a fabricated exit status.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
The IsolationSession design note described the closure shapes that no longer
exist, said making the terminator fallible was future work, and stated that
teardown joins the waiter unconditionally. It also described the waiter as
reporting a timeout only while the process was still running, and the
service-side timer as a plain per-process reaper -- both superseded by the
sticky deadline and the watchdog margin.

The state-aware API document carried the same superseded closure signatures and
the same "the backend killed the process" reading of a timeout.

Before this change a state-aware exec could not report a timeout at all:
`ErrorKind::TimedOut` appears nowhere in the adapter on the base commit, so
every "the tree is killed" statement was true, satisfied by the process-spawning
backends alone -- those lead their own process group on Unix and get a job
object on Windows. Making the outcome reachable from a backend whose only
primitive is the foreground process turned four of those statements false, in
the `SandboxProcess` trait and in the public Rust SDK. They now say the workload
is terminated and leave the reach to the backend, which is the only thing true
of both, and still name the tree kill where it is real.

The narrowed note also pointed at `exec_sandbox` as though a caller could take
that route today. It cannot: the parser hardcodes `experimental_enabled` to
false, so an experimental backend is refused before dispatch. The distinction
still belongs in the doc -- it is what implementors build against, and it
becomes observable the moment the opt-in lands -- but it is now stated as gated
rather than as available.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02c36d36-fb3d-446f-9108-4e4a01b01af1
Copilot AI review requested due to automatic review settings August 14, 2026 19:30
@adpa-ms
adpa-ms force-pushed the user/adibpa/copilot-exechandle-library-contract branch from b416bb1 to 2ddfff1 Compare August 14, 2026 19:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (5)

src/core/wxc_common/src/exec_stream.rs:549

  • This message contradicts the new outcome contract: TimedOutAfterDeadline returns TimedOut when the process exited on its own after the deadline, so the sandbox did not necessarily terminate it. Preserve the distinction in the observable error text.
            "the exec timed out and the sandbox terminated it",

src/core/wxc_common/src/exec_stream.rs:472

  • A process can finish and its waiter thread can return before the caller invokes wait/try_wait; in that state self.exit is still None, so kill() invokes the fallible terminator against an already-dead process and can report a spurious refusal. The previous infallible terminator hid this race. Join an already-finished waiter before terminating, and ensure the terminator treats a concurrent natural exit as success.
        if self.exit.is_some() {
            self.terminator = None;
            self.kill_refused = false;
            return Ok(());
        }

src/core/wxc_common/src/exec_stream.rs:90

  • This field comment still promises a process-tree kill, but the newly supported IsolationSession terminator only reaches the foreground process. Describe this as backend-defined termination so the internal contract matches ExecOutcome and the implementation.
    terminator: Option<Box<dyn FnOnce() -> Result<(), MxcError> + Send>>,

src/core/wxc_common/src/sandbox_process.rs:129

  • This caveat only qualifies timeout handling; SandboxProcess::kill above still unconditionally promises termination of the process and all descendants. ExecSandboxProcess::kill now invokes an IsolationSession terminator that can only target the foreground process, so the trait's kill contract remains false and should carry the same backend-specific reach qualification.
    /// **How far that termination reaches is the implementation's to state.**
    /// The process-spawning implementations kill the whole tree — the child
    /// leads its own process group on Unix and is assigned a job object on
    /// Windows. An implementation whose only primitive is the foreground
    /// process (the state-aware exec adapter, over a backend with no tree API)

src/core/mxc-sdk/src/sandbox.rs:37

  • The new backend-specific qualification conflicts with this public type's existing kill promises: the Sandbox rustdoc still says “and its whole tree” and Sandbox::kill still says “Kill the child and its process tree.” The same Sandbox wraps state-aware exec handles, so update both statements to describe backend-defined termination reach.
    /// How far "gone" reaches depends on the backend. A process-spawning
    /// backend kills the whole tree. A backend whose only primitive is the
    /// foreground process — the state-aware `exec` path over IsolationSession —
    /// confirms that process, and a descendant the workload backgrounded is
    /// reclaimed when the sandbox is stopped and deprovisioned rather than here.

@adpa-ms
adpa-ms merged commit 20a66c4 into main Aug 14, 2026
23 checks passed
@adpa-ms
adpa-ms deleted the user/adibpa/copilot-exechandle-library-contract branch August 14, 2026 21:44
Darren Hoehna (dhoehna) added a commit to dhoehna/mxc that referenced this pull request Aug 14, 2026
Resolves the collision with microsoft#798, which rewrote the same LXC network and
attach paths this branch changes, and adapts to the exec-handle contract
microsoft#874 introduced.

network_iptables.rs -- FORWARD teardown now branches on whether the veth
interface is known.  microsoft#798 deletes by replaying the full remembered rule
spec, which needs the interface name; this branch enumerates FORWARD for
jumps to the container chain, which does not.  Both are kept, because
neither covers the other's case.  With the veth the specs are replayed for
all four rule classes, which is the only route that reaches return_rule and
physdev_return -- those jump to ACCEPT rather than to the chain, so no
search for jumps into the chain can see them and the chain flush cannot
reclaim them.  Without the veth, signal-time force_cleanup enumerates
instead of matching nothing and leaking the chain and its hooks
permanently.  Enumeration is deliberately not used when the veth is known:
an absent tool answers that no hooks remain, and that verdict gates the -F,
so it would flush a still-hooked chain into a fail-open.

lxc_bindings.rs -- build_attach_args_with_env_control now carries both
sides' additions, force_clear_env from microsoft#798 and the exec marker from this
branch.  Both test suites are kept.

state_aware.rs -- adapted to microsoft#874.  This file does not exist on main and
had no conflict markers, but main changed ExecHandle underneath it.  The
waiter now reports ExecOutcome::Exited(exit_code) and the terminator Ok(()),
matching wslc, windows_sandbox, and isolation_session, each of which blocks
on a backend that already holds the exit code.

lxc_runner.rs -- resolved from line-ending-normalized blobs.  microsoft#798 committed
this file with CRLF while every other .rs in the tree is LF, which turned
the whole file into a single conflict; .gitattributes pins only *.sh.  The
merged file is LF.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: acf3853b-fbe3-41ab-870a-9274a698cc0c
Darren Hoehna (dhoehna) added a commit to dhoehna/mxc that referenced this pull request Aug 14, 2026
microsoft#874 added the consumer argument to the state-aware exec contract, and the
merge adapted the signature without honoring it.  attach_run relays the
workload's output to this process's stdio and returns no pipes, so LXC is
one of the backends that cannot serve an in-process caller.  Returning a
handle with three null pipes is a contract violation: by the time the caller
sees it the script has already run and its output has already gone somewhere
it never asked for.

Refuses through the shared unsupported_library_exec, ahead of the id parse
and both container probes, matching WSLc and Windows Sandbox.  The test
passes an id that would fail as MalformedId, so it fails if the check ever
moves after it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: acf3853b-fbe3-41ab-870a-9274a698cc0c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants