Skip to content

Publish MXC crates through ESRP - #715

Closed
Darren Hoehna (dhoehna) wants to merge 10 commits into
microsoft:mainfrom
dhoehna:user/dahoehna/crate-release-pipeline
Closed

Publish MXC crates through ESRP#715
Darren Hoehna (dhoehna) wants to merge 10 commits into
microsoft:mainfrom
dhoehna:user/dahoehna/crate-release-pipeline

Conversation

@dhoehna

@dhoehna Darren Hoehna (dhoehna) commented Jul 30, 2026 •
edited by microsoft-github-policy-service Bot
Loading

Copy link
Copy Markdown
Contributor

Summary

  • makes the complete 18-crate mxc-sdk dependency closure publishable
  • packages versioned .crate artifacts in the official-build pipeline
  • publishes crates leaf-first through ESRP with checksum and propagation validation
  • supports dry runs and safe retries without storing a crates.io token in the repository

Current limitation

Multi-package cargo package with the private-feed replace-with configuration is blocked by rust-lang/cargo#17196. The crate-packaging stage is therefore explicitly opt-in and disabled by default, so ordinary official builds are unaffected while a workaround is selected.

Validation

  • packaged and validated all 18 crates locally
  • verified the generated dependency order
  • parsed the modified pipeline YAML
  • verified crates.io sparse-index probing and checksum logic

No pipeline was queued and no crates were published.

Microsoft Reviewers: Open in CodeFlow

Darren Hoehna (dhoehna) and others added 10 commits July 13, 2026 20:41
@dhoehna
Make the mxc-sdk crate closure publishable to crates.io
b2e6f36 
Add crates.io-required metadata so mxc-sdk and its first-party dependency
closure (10 crates) can be published:

- workspace: add repository URL; give the 8 first-party path-deps in
  [workspace.dependencies] explicit versions (cargo publish requires a version
  on every path dependency).
- wxc_common: version its optional `nanvix_common` path-dep. Although it is
  gated by the off-by-default `microvm` feature, cargo requires every
  dependency in a published manifest -- optional ones included -- to carry a
  version and exist on crates.io, so nanvix_common must ship too. It is a leaf
  (serde/serde_json only).
- add description to wxc_common, mxc_pty, and the appcontainer / bubblewrap /
  lxc / seatbelt common crates.
- add license + repository to mxc_telemetry; repository to nanvix_common.
- inherit repository on each published crate.
- make sandbox_spec publishable (remove publish = false).

Validated with `cargo metadata` and per-crate `cargo package --list` (plus
`cargo publish --dry-run --no-verify` for leaf crates) against real crates.io.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
@dhoehna
Add one-click crates.io release pipeline for mxc-sdk
bf82cd1 
New ADO pipeline publishes the mxc-sdk crate and its 10-crate first-party
closure to crates.io from a single "Run pipeline" click, with a dryRun
parameter (default true) that previews every crate before anything uploads.

- .azure-pipelines/scripts/publish_crates_to_cratesio.py: leaf-first ordered
  publish (mxc_telemetry, nanvix_common, mxc_pty, sandbox_spec, wxc_common,
  lxc_common, seatbelt_common, appcontainer_common, bwrap_common, mxc-sdk),
  per-crate version resolved via `cargo metadata`, idempotent version-exists
  skip via the crates.io sparse index, `cargo publish --no-verify` with a
  post-publish index-propagation wait, and a --dry-run mode that runs
  `cargo package --list` (validates each manifest without needing first-party
  deps on the index yet). Modeled on scripts/seed_feed.py conventions.
- .azure-pipelines/1ES.Crate.Release.yml: trigger:none, extends the 1ES
  Official template; DryRun stage always previews; Publish stage is compiled
  in only when dryRun=false and gates on manual approval via the
  MXC-CratesIo-Production environment. Reuses Rust.Toolchain.Public.yml and
  deliberately skips Cargo.Setup.* so cargo talks to the real crates.io.

Secrets stay out of the repo: CARGO_REGISTRY_TOKEN comes from the
MXC-CratesIo-Publish variable group and is mapped to env only in the publish
step. One-time ADO/crates.io setup is documented in the pipeline header.

Validated locally against real crates.io: the full --dry-run passes for all
10 crates; all 10 target names are currently unregistered on crates.io.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
@dhoehna
Merge microsoft/mxc main (47a172c) into crate-release-pipeline
d27c0f0 
Resolved Cargo.toml conflicts:
- src/Cargo.toml [workspace.dependencies]: kept version pins on wxc_common and
  appcontainer_common; kept upstream's new mxc_engine and mxc-sdk path deps
  (left unversioned - not part of the current publish closure).
- src/mxc_telemetry/Cargo.toml: kept repository.workspace = true.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
@dhoehna
ci(crates): release crates.io through ESRP in 1ES.Release.yml
a39e647 
Rework the crates.io release per PR microsoft#647 review feedback (Brandon Bonaby):
publish through the ESRP OSS release system instead of a bespoke
CARGO_REGISTRY_TOKEN, folded into the existing 1ES.Release.yml so one
pipeline releases npm, crates.io, or both (modeled on azure-sdk-for-rust).

- scripts/crates_release.py: package/verify-order/stage/wait subcommands;
  cargo-packages the closure leaf-first, no token, no publish endpoint.
- templates/Package.Crates.Job.yml: official-build job -> mxc-crates-package.
- templates/Publish.CratesIo.Job.yml: releaseJob, EsrpRelease@12 contenttype
  Rust, one task per crate leaf-first, verify each on the crates.io index
  before the next.
- 1ES.Release.yml: publishNpm/publishCrates params; gate NPM stage; add
  Publish_to_CratesIo stage; ESRP one-time-setup notes in the header.
- 1ES.Build.Stages.yml: add Package_Crates stage.
- Delete 1ES.Crate.Release.yml + publish_crates_to_cratesio.py (token approach).

publishCrates defaults to false: enabling actual publishing still needs the
crate closure metadata reconciled (7 path deps need versions;
isolation_session_bindings is publish=false) plus OSPO OSS-release setup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
@dhoehna
fix(crates): package the whole closure in one cargo invocation
d5bbcb3 
cmd_package looped `cargo package -p <crate>` one crate at a time, which
fails on the first non-leaf crate: packaging a crate alone strips the path
from each dependency and cargo then looks for the sibling on the crates.io
index, where it is not published yet (e.g. wxc_common -> mxc_telemetry ->
"no matching package named `mxc_telemetry` found").

Package the entire closure in a single `cargo package` invocation with every
crate passed via -p, so cargo resolves intra-closure deps from target/package/
instead of crates.io (the azure-sdk-for-rust Pack-Crates.ps1 approach). Verified
locally: all 9 packageable crates now tar in one run; the only remaining failure
is the known mxc-sdk -> mxc_engine missing-version blocker. Also corrected two
docstring claims that wrongly implied per-crate `--no-verify` packaging works.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
@dhoehna
fix(crates): check the private Azure feed index, not public crates.io
cfaaf18 
The 1ES official build runs network-isolated and cannot reach the public
crates.io sparse index (https://index.crates.io), so crates_release.py's
post-publish version check would fail at runtime. Query the private
Mxc-Azure-Feed sparse index instead (reachable in-pipeline), authenticated
with the build identity's $(System.AccessToken).

- crates_release.py: replace CRATES_IO_SPARSE_INDEX with AZURE_FEED_SPARSE_INDEX
  (matches .azure-pipelines/.cargo/config.toml); add _feed_request() Bearer-auth
  helper reading SYSTEM_ACCESSTOKEN; _published_versions() now hits the feed;
  relabel poll/index wording to "Azure Artifacts feed".
- Publish.CratesIo.Job.yml: map $(System.AccessToken) -> SYSTEM_ACCESSTOKEN on
  the wait step so the feed request can authenticate.

Addresses Brandon Bonaby's review comment on PR microsoft#647.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
@dhoehna
ci(crates): address review feedback (gate, defaults, dry-run, trim)
287305b 
Applies Brandon Bonaby's PR microsoft#647 review comments:

- 1ES.Build.Stages.yml: gate the Package_Crates stage on isOfficialBuild so
  crate packaging only runs on the official signed build, not PR/CI.
- 1ES.Release.yml: default publishNpm to false too, so the operator must
  explicitly select npm and/or crates in the run dialog.
- 1ES.Release.yml: reduce the oversized header comment (and drop the stale
  index.crates.io egress note superseded by the private-feed fix).
- Add a cratesDryRun parameter (1ES.Release.yml -> Publish.CratesIo.Job.yml)
  that runs stage + verify-order but skips the EsrpRelease publish and the feed
  wait, for testing the pipeline without publishing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
@dhoehna
Merge microsoft/mxc main into user/dahoehna/crate-release-pipeline
237abe6 
Resolve src/Cargo.toml conflict in [workspace.dependencies]: keep main's
roxmltree -> quick-xml migration and the new windows_sandbox_lifecycle
dependency, alongside the version = "0.7.0" pins added for crate publishing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
@dhoehna
fix(crates): pass --registry crates-io to cargo package
daa619e 
The pipeline's .cargo/config.toml replaces the crates-io source with the
internal Mxc-Azure-Feed, so `cargo package` aborts with "crates-io is
replaced with remote registry Mxc-Azure-Feed; include --registry
Mxc-Azure-Feed or --registry crates-io". Package for crates-io -- the real
publish target -- so intra-closure sibling deps are recorded as plain
crates.io deps (packaging for the feed would embed a registry-index that
crates.io rejects). The source replacement still serves index reads in the
network-isolated build.

Found via MXC-Official-Build run 153175941 (Package Crates stage failed at
cargo package exit 101).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
@dhoehna
feat(crates): publish releases through ESRP
b25d86e 
Package the complete mxc-sdk crate closure in official builds and add a guarded ESRP release flow with dependency ordering, checksum verification, propagation waits, and safe retries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: f199a77f-51ec-4441-baa5-b87df8177b83
@dhoehna
Darren Hoehna (dhoehna) requested a review from a team as a code owner July 30, 2026 21:38
Copilot AI review requested due to automatic review settings July 30, 2026 21:38
Copilot AI reviewed Jul 30, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in ESRP workflow for publishing the 18-crate mxc-sdk dependency closure to crates.io.

Changes:

  • Makes internal crates publishable with repository metadata and versioned path dependencies.
  • Packages crates leaf-first into official-build artifacts.
  • Adds dry-run, retry, dependency, and checksum validation to ESRP publishing.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/Cargo.toml Adds repository metadata and dependency versions.
src/mxc_telemetry/Cargo.toml Adds publish metadata.
src/core/wxc_common/Cargo.toml Adds metadata and versioned NanVix dependency.
src/core/mxc-sdk/Cargo.toml Adds repository metadata.
src/core/mxc_pty/Cargo.toml Adds publish metadata.
src/core/mxc_engine/Cargo.toml Adds repository metadata.
src/core/generated/base_container_specification/Cargo.toml Enables publishing generated bindings.
src/backends/wslc/common/Cargo.toml Adds publish metadata.
src/backends/windows_sandbox/lifecycle/Cargo.toml Adds publish metadata.
src/backends/windows_sandbox/common/Cargo.toml Adds publish metadata.
src/backends/seatbelt/common/Cargo.toml Adds publish metadata.
src/backends/nanvix/runner/Cargo.toml Adds metadata and dependency version.
src/backends/nanvix/common/Cargo.toml Adds repository metadata.
src/backends/lxc/common/Cargo.toml Adds publish metadata.
src/backends/isolation_session/common/Cargo.toml Adds publish metadata.
src/backends/isolation_session/bindings/Cargo.toml Enables publishing bindings.
src/backends/hyperlight/common/Cargo.toml Adds publish metadata.
src/backends/bubblewrap/common/Cargo.toml Adds publish metadata.
src/backends/appcontainer/common/Cargo.toml Adds publish metadata.
.azure-pipelines/templates/Publish.CratesIo.Job.yml Defines leaf-first ESRP publication.
.azure-pipelines/templates/Package.Crates.Job.yml Produces the crate artifact.
.azure-pipelines/templates/1ES.Build.Stages.yml Adds the opt-in packaging stage.
.azure-pipelines/scripts/crates_release.py Implements packaging and publication validation.
.azure-pipelines/1ES.Release.yml Adds selectable npm/crates release stages.
.azure-pipelines/1ES.Build.Official.yml Exposes the crate-packaging option.

Comment thread .azure-pipelines/1ES.Release.yml
Comment on lines +40 to +44
default: false
- name: publishCrates
displayName: "Publish the Rust crate closure to crates.io"
type: boolean
default: false
Comment thread .azure-pipelines/scripts/crates_release.py
package_args = [
"cargo",
"package",
"--no-verify",
@dhoehna

Darren Hoehna (dhoehna) commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #716, which is branched from current main and merges cleanly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dhoehna