ci: stop shipping tier2_bfs in ADO-built release binaries#443
Merged
Conversation
The `1ES.Build.Rust.Run@1` task defaults to `--all-features` when no `features:` input is supplied. That was harmless until commit 8c57211 ("feat(wxc_common): gate Tier 2 (BFS) behind tier2_bfs Cargo feature") landed via #410 on 2026-05-26, which introduced `tier2_bfs` with the explicit constraint that production binaries on Win 11 25H2 must NOT carry it; otherwise the embedded `bfscfg.exe` invocation risks an OS hang. The GH Actions workflow already respects this — its build line in `.github/workflows/build.yml` does not pass `--all-features` — but the ADO pipeline silently enables every feature including `tier2_bfs`, and ships that binary via `release-binaries.zip` and the npm SDK. Confirmed by inspecting the actual `Cargo build (1ES PT)` step log of build 147990337 (PR #434): Running: cargo build --locked --offline --target x86_64-pc-windows-msvc --profile release --all-features Override the task default by setting `allFeatures: false` and listing the production feature set explicitly. This keeps every runtime backend that IS intended for production (`hyperlight`, `isolation_session`, `microvm`, `wslc`) compiled in while dropping `tier2_bfs`. Lint.Job.yml is intentionally left alone; clippy with `--all-features` gives broader coverage of conditionally compiled code and does not produce a shipping binary. Closes #442 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Azure DevOps (1ES) Rust build template to avoid implicitly building shipping Windows binaries with --all-features, which currently pulls in the tier2_bfs feature that must not be present in production binaries on Win 11 25H2.
Changes:
- Adds explicit
allFeatures: falseplus an explicit Cargo feature list intended for production (hyperlight isolation_session microvm wslc) in the1ES.Build.Rust.Run@1build step.
Show a summary per file
| File | Description |
|---|---|
.azure-pipelines/templates/Rust.Build.Job.yml |
Overrides the 1ES Rust task’s default feature selection to prevent tier2_bfs from being compiled into ADO-built Windows release binaries. |
Copilot's findings
- Files reviewed: 1/1 changed files
- Comments generated: 1
PR-build 148026004 confirmed `allFeatures: false` correctly drops the task's `--all-features` default, but `features: '...'` was silently ignored — the resulting cargo line had no `--features` at all, so the binary lost hyperlight/isolation_session/microvm/wslc backends. Shotgun several candidate input names in one push so the next CI run narrows down which spelling the task actually consumes. Unknown inputs are silently ignored by the 1ES PT task layer (already proven), so this is safe. Once we identify the canonical name we will collapse to a single key. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bbonaby
reviewed
May 28, 2026
Comment on lines
+125
to
+138
| # IMPORTANT: do not let the 1ES task default to `--all-features`. The | ||
| # `tier2_bfs` Cargo feature gates Tier 2 (BFS) and must NOT be | ||
| # compiled into production binaries on Win 11 25H2 (the embedded | ||
| # `bfscfg.exe` invocation risks an OS hang). See | ||
| # `.github/workflows/build.yml`, `test_scripts/T3-Workloads.ps1`, | ||
| # and `test_scripts/Win25H2Safe-Tests.ps1` for the safety model. | ||
| # Enumerate the production feature set explicitly so the shipped | ||
| # binary still carries every runtime backend except `tier2_bfs`. | ||
| # | ||
| # Shotgun across candidate input names while we identify which one | ||
| # the 1ES Rust task actually consumes (the task's source is internal | ||
| # and not browsable). Unknown inputs are silently ignored by the | ||
| # 1ES PT task layer, so listing several candidates is safe; once we | ||
| # know the canonical name we'll collapse this to a single key. |
Collaborator
There was a problem hiding this comment.
Collaborator
There was a problem hiding this comment.
Also AI related but you should be able to get your AI agent to access this info by installing the EngHub mcp server: https://eng.ms/docs/help/announcements/mcp-server#setup-in-github-copilot-cli . That should help with all the 1ES stuff that isn't public.
Member
Author
There was a problem hiding this comment.
Thanks, the MCP was very helpful.
Per the 1ES Rust virtual task schema (eng.ms doc), the inputs at
`1ES.Build.Rust.Run@1` level are:
allFeatures: boolean # default: true
activatedFeatures: string[]
Build 148028047 confirmed `allFeatures: false` correctly drops
`--all-features` from the cargo line, but the previous commit's
`features: '...'` (and four other guessed names) were silently ignored,
so the binary lost hyperlight/isolation_session/microvm/wslc backends.
Replace the shotgun with the documented `activatedFeatures` YAML array
spelling. This should produce a cargo line of the form:
cargo build --locked --offline --target ... --profile release \\
--features hyperlight,isolation_session,microvm,wslc
Schema: https://eng.ms/docs/coreai/devdiv/one-engineering-system-1es/1es-docs/1es-pipeline-templates/features/buildworkflows/rust-virtual-task
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build 148041192 rejected the previous commit at YAML validation:
/.azure-pipelines/templates/Rust.Build.Job.yml (Line: 136, Col: 13):
A sequence was not expected
Even though the 1ES Rust virtual task documents `activatedFeatures:
string[]`, AzDO task `inputs:` only accept scalar values at the YAML
schema layer; the 1ES task itself splits the scalar internally. Use a
whitespace-separated string instead.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build 148041225's WXC jobs got the correct cargo line:
cargo build ... --features hyperlight isolation_session microvm wslc
But the LXC jobs failed because `lxc/Cargo.toml` only defines the
`hyperlight` feature (no microvm/isolation_session/wslc). Previously
this was hidden because `--all-features` enables only the features
defined on the package being built, and lxc only has `hyperlight`.
Split activatedFeatures by `item.os`:
- windows (WXC) -> hyperlight isolation_session microvm wslc
- linux (LXC) -> hyperlight
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MGudgin
marked this pull request as ready for review
May 28, 2026 05:55
bbonaby
approved these changes
May 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The ADO build was implicitly passing
--all-featuresto1ES.Build.Rust.Run@1, which has been the default behaviour of that task ever since we adopted it (commit 8875d40, 2026-03-25). That stayed harmless until commit 8c57211 (2026-05-25, merged via #410) introduced a new Cargo feature calledtier2_bfs, which gates an OS-hang risk on Win 11 25H2 viabfscfg.exeand explicitly must not ship in production. Since then, every ADO-built release binary has shipped withtier2_bfscompiled in — exactly the configuration that the preflight scripts intest_scripts/T3-Workloads.ps1andtest_scripts/Win25H2Safe-Tests.ps1are designed to refuse to run.This PR pins the feature set explicitly on the
1ES.Build.Rust.Run@1build task in.azure-pipelines/templates/Rust.Build.Job.ymlso the task no longer falls through to its--all-featuresdefault. The feature lists are scoped by OS to match the actual feature definitions in each crate;hyperlight isolation_session microvm wslchyperlightThe Cargo task input names took some iteration to nail down. The 1ES Rust virtual task documents
features.allFeaturesandfeatures.activatedFeaturesinfeatures.Xnotation, but at the actual taskinputs:level they are flat keysallFeaturesandactivatedFeatures, andactivatedFeaturesis a scalar string at the YAML schema layer (despite being documentedstring[]) — the 1ES task splits it internally. Reference; https://eng.ms/docs/coreai/devdiv/one-engineering-system-1es/1es-docs/1es-pipeline-templates/features/buildworkflows/rust-virtual-taskTesting
Verified on ADO build 148042154 (PR check run on this branch): all 16 jobs green, total pipeline 28.7 min. The four cargo build invocations now report exactly the expected feature sets;
cargo build ... --features hyperlight isolation_session microvm wslccargo build ... --features hyperlight isolation_session microvm wslccargo build ... --features hyperlightcargo build ... --features hyperlightNo
--all-featuresand notier2_bfsin any of them. Critical-path jobBuild (x64 WXC)ran in 13.4 min.Closes #442