Enable one-click crates.io release for mxc-sdk

[dhoehna]

Summary

Adds a one-click Azure DevOps pipeline that publishes the mxc-sdk crate and its first-party dependency closure to crates.io, plus the crate-metadata changes required to make that closure publishable.

This branch contains two logical commits:

1. Make the mxc-sdk crate closure publishable (11 Cargo.toml files) — registry-agnostic metadata: workspace repository, per-crate description/license/repository, dual { path, version } on internal path deps, and making sandbox_spec publishable. Required to publish from anywhere.
2. Add the release pipeline (2 files) — .azure-pipelines/1ES.Crate.Release.yml and .azure-pipelines/scripts/publish_crates_to_cratesio.py.

How it works

• Run pipeline with dryRun = true (the default) previews every crate via cargo package --list and publishes nothing.
• With dryRun = false, a Publish stage waits for manual approval on the MXC-CratesIo-Production environment, then publishes leaf-first, waiting for index propagation between crates. Re-runs are idempotent (a version already on crates.io is skipped).

Secrets stay out of the repo

CARGO_REGISTRY_TOKEN comes from the ADO variable group MXC-CratesIo-Publish and is mapped to the publish step's env only. Nothing secret is committed.

One-time setup (not codeable in the repo)

Variable group, environment + approval, crates.io egress allowlist, crate-name reservation / OSPO sign-off, and registering the pipeline. See the checklist at the bottom of 1ES.Crate.Release.yml.

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull request overview

This PR enables releasing the Rust mxc-sdk crate to crates.io by (1) making the crate’s first-party dependency closure publishable via Cargo metadata/manifest adjustments, and (2) adding an Azure DevOps “one-click” pipeline plus a helper script to publish the closure leaf-first.

Changes:

• Add crates.io publish metadata to mxc-sdk and its first-party dependency closure (license/repository/description) and make sandbox_spec publishable.
• Convert internal path dependencies used by the publishable closure to dual { path, version } so cargo publish can strip path and publish clean manifests.
• Add an ADO pipeline (1ES.Crate.Release.yml) and a Python script to dry-run/package or publish the crate closure with index-propagation waits.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/Cargo.toml	Adds workspace repository and sets { path, version } for publishable internal workspace deps.
src/core/wxc_common/Cargo.toml	Adds publish metadata and ensures optional nanvix_common path dep includes a version for publishing.
src/core/mxc-sdk/Cargo.toml	Adds workspace repository metadata needed for crates.io publishing.
src/core/mxc_pty/Cargo.toml	Adds publish metadata (repository/description) for the closure.
src/mxc_telemetry/Cargo.toml	Adds workspace license/repository metadata for publishing.
src/core/generated/base_container_specification/Cargo.toml	Removes publish = false by making sandbox_spec publishable and adds repository metadata.
src/backends/appcontainer/common/Cargo.toml	Adds publish metadata for publishing appcontainer_common.
src/backends/bubblewrap/common/Cargo.toml	Adds publish metadata for publishing bwrap_common.
src/backends/lxc/common/Cargo.toml	Adds publish metadata for publishing lxc_common.
src/backends/seatbelt/common/Cargo.toml	Adds publish metadata for publishing seatbelt_common.
src/backends/nanvix/common/Cargo.toml	Adds workspace repository metadata so nanvix_common is publishable.
.azure-pipelines/1ES.Crate.Release.yml	New ADO pipeline that runs dry-run preview and gated publish to crates.io.
.azure-pipelines/scripts/publish_crates_to_cratesio.py	New helper script to dry-run package or publish crates leaf-first and wait for sparse-index propagation.

[bbonaby]

issue: To publish to crates.io using the official Microsoft open source account we will need to publish it through the ESRP release system. See: https://eng.ms/docs/microsoft-security/identity/trust-and-security-services/tss-release-distribute/tss-release-esrp-parent/oss-publishing/releasing-open-source/cratesio . You will need to follow that and do something similar to what we do in the 1ES.Release.yml. To be honest, we should be able to reuse that same yaml file to release to either Npm, Crates.io or both.

At the very least that one should be the entry point to all releases, but we can still have template yml files that we point to to keep it clean. Everything in terms of set up on the back end for ESRP should already be set up for us. So, let me know if you have any issues with that.

In the ESRP docs they say the azure rust sdk folks have this working, where they publish all their crate dependencies first then publish their main crate. I believe the yml file they do this in is this one https://github.com/Azure/azure-sdk-for-rust/blob/95eb8c3878cb2c7d3239b5f584b8179ee0483c31/eng/pipelines/templates/stages/archetype-rust-release.yml . We can probably do something similar.

[dhoehna]

Thanks Brandon -- followed this. Reworked the crate release to go through ESRP and folded it into 1ES.Release.yml so one pipeline releases npm, crates.io, or both:

• 1ES.Release.yml now takes publishNpm / publishCrates boolean params. The existing NPM stage is gated on publishNpm, and a new Publish_to_CratesIo stage runs the ESRP path.
• Following the azure-sdk-for-rust approach, the official build packages the closure leaf-first in a single cargo package invocation (all --package flags together, so intra-closure deps resolve locally) into the mxc-crates-package artifact. The release job then runs one EsrpRelease@12 task per crate (contenttype: Rust, account microsoft-oss-releases), leaf-first, verifying each on the crates.io sparse index before the next.
• Deleted the old token-based 1ES.Crate.Release.yml + publish_crates_to_cratesio.py -- no more CARGO_REGISTRY_TOKEN. New templates: templates/Package.Crates.Job.yml (packaging) and templates/Publish.CratesIo.Job.yml (ESRP release loop).

This also obsoletes the two Copilot comments above (they were on the now-deleted script/pipeline).

One heads-up before we flip publishing on -- publishCrates defaults to false for now. Packaging the true --all-features closure of mxc-sdk surfaced that it now routes through the mxc_engine hub crate and expands to 17 first-party crates. Before a first real publish we still need to: (1) add a version to the 7 path deps in src/Cargo.toml that currently only carry path; (2) resolve isolation_session_bindings being publish = false; and (3) do the OSPO OSS-release registration + reserve the crate names. Happy to take that as a follow-up -- let me know if you'd rather have it in this PR or a separate one.

[bbonaby]

issue: we should probably add a conditional here to only do this packaging when the isOfficialBuild parameter is true.

[dhoehna]

Done in 287305b -- gated the Package_Crates stage on isOfficialBuild in 1ES.Build.Stages.yml, so crate packaging only runs on the official signed build (not PR/CI).

[bbonaby]

note: the ESRP task above will actually keep polling until the package is published so we probably don't need this. It's done when the waitforreleasecompletion input to the task is set to true. That said, I'm not against the extra wait task.

[dhoehna]

Good point -- waitforreleasecompletion: true already blocks until each ESRP publish finishes. I kept the wait step as a lightweight guard that confirms the crate is resolvable on the feed index before the next dependent publishes, and it only emits a warning (never fails the release) on timeout. It is also skipped in the new cratesDryRun mode. Happy to drop it entirely if you would prefer -- your call. (Also fixed it to query the private feed per your other comment.)

[bbonaby]

issue: I like the parameters, lets make both false so it's clear in the UI that we expect the person initiating the pipeline run to select one or both.

[dhoehna]

Done in 287305b -- publishNpm now defaults to false too, so the run dialog forces the operator to pick npm and/or crates.

[bbonaby]

issue: So this won't worked in the 1ES pipeline due to network isolation. You'd need to use the private feed's sparse index. Should be this one

mxc/.azure-pipelines/.cargo/config.toml

Line 8 in 3fd07f5

Mxc-Azure-Feed = { index = "sparse+https://microsoft.pkgs.visualstudio.com/Dart/_packaging/Mxc-Azure-Feed/Cargo/index/" }

So it would look something like this:

AZURE_FEED_REGISTRY = "Mxc-Azure-Feed"
AZURE_FEED_SPARSE_INDEX = (
    "https://microsoft.pkgs.visualstudio.com/"
    "Dart/_packaging/Mxc-Azure-Feed/Cargo/index"
)
SYSTEM_ACCESS_TOKEN_ENV = "SYSTEM_ACCESSTOKEN"

then you can have a helper function in here somewhere:

def _feed_request(url: str) -> urllib.request.Request:
    """Create an Azure Artifacts request authenticated by the pipeline token."""
    token = os.environ.get(SYSTEM_ACCESS_TOKEN_ENV)
    if not token:
        raise RuntimeError(
            f"{SYSTEM_ACCESS_TOKEN_ENV} is not set. Map $(System.AccessToken) "
            f"to that environment variable in the pipeline task."
        )

    return urllib.request.Request(
        url,
        headers={"Authorization": f"Bearer {token}"},
    )

Then your _published_versions function below can look similar to:

def _published_versions(crate: str) -> set[str]:
    """Return the versions of `crate` already in the Azure Artifacts feed."""
    url = f"{AZURE_FEED_SPARSE_INDEX}/{_sparse_index_path(crate)}"
    with urllib.request.urlopen(_feed_request(url), timeout=30) as response:
        body = response.read().decode("utf-8")
    versions: set[str] = set()
    for line in body.splitlines():
        line = line.strip()
        if line:
            versions.add(json.loads(line)["vers"])
    return versions

You also have have references to crates.io in the comments of this file as well. You'd want to update those to just say Azure Artifacts Feed or something like that.

[dhoehna]

Done in cfaaf18 -- switched the version check from public index.crates.io to the private Mxc-Azure-Feed sparse index (AZURE_FEED_SPARSE_INDEX, matching .azure-pipelines/.cargo/config.toml), added a _feed_request() Bearer-auth helper that reads SYSTEM_ACCESSTOKEN, and rewrote _published_versions() to hit the feed. Mapped $(System.AccessToken) -> SYSTEM_ACCESSTOKEN on the wait step and relabeled the crates.io comments to "Azure Artifacts feed". One assumption: the build identity has read access to Mxc-Azure-Feed -- let me know if that needs onboarding.

[bbonaby]

issue: We should update this to include all the crates that will be published as part of the mxc-sdk. That said, The crate names themselves should actually be prefixed with mxc- so they are easily discoverable in crates.io.

I think we also need a follow up PR to add a GitHub PR workflow/action to check that all crates are publishable.

[dhoehna]

Agreed on all three -- two need a decision/setup before I wire them in, so flagging rather than guessing:

• Full closure: the real mxc-sdk --all-features closure is 17 crates (the mxc_engine hub was added by a later refactor). Before expanding CRATES/crateOrder, two blockers need resolving -- 7 path deps in src/Cargo.toml lack a version, and isolation_session_bindings is publish = false. Want those in this PR, or a follow-up?
• mxc- prefix: renaming published crate names touches each [package] name and needs the names reserved on crates.io. Can you confirm the scheme (e.g. wxc_common -> mxc-wxc-common)? I will apply it once we agree.
• Publishability check action: agree this is the follow-up PR you mentioned; I can open it separately.

The list is a clearly-marked placeholder and publishCrates defaults false, so nothing ships until this is settled.

[bbonaby]

issue: most of this comment can be removed or reduced now I believe.

[dhoehna]

Done in 287305b -- trimmed the header to the essentials and dropped the stale "egress to index.crates.io" note (superseded by the private-feed fix).

[bbonaby]

Note: for your testing, you can add a dry-run boolean in here too, that when set won't go into the EsrpRelease@10 tasks. You can actually test this out if you create an in repo branch (noticed this PR is coming from your fork). But yea if you create an in repo branch e.g user/dhoehna/test-rust-publishing with these same changes, you should be able to use the branch to run the official pipeline via https://microsoft.visualstudio.com/Dart/_build?definitionId=190018 and then see any errors that pop up so you can fix them in this PR. Let me know if you don't have access.

[dhoehna]

Added a cratesDryRun boolean in 1ES.Release.yml (passed through to Publish.CratesIo.Job.yml) in 287305b: when set, each crate still runs stage + verify-order but skips the EsrpRelease task and the feed wait, so the plumbing runs without publishing.

On the in-repo branch: I do not have push access to microsoft/mxc from here (this PR is from my fork), so I cannot create user/dhoehna/test-rust-publishing myself. Could you create it from these changes (or grant access)? Then I will run pipeline 190018 against it and fix whatever surfaces.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

.azure-pipelines/scripts/crates_release.py:99

• The packaged list is not the mxc-sdk dependency closure: mxc-sdk directly depends on mxc_engine (src/core/mxc-sdk/Cargo.toml:15), but that crate and its seven first-party dependencies are absent here, their workspace dependencies still lack versions, and isolation_session_bindings remains non-publishable. Cargo rejects a packaged crate whose registry manifest contains a versionless path dependency, so the new unconditional Package_Crates stage will fail every official build instead of producing the release artifact. Complete the 17-crate closure and its publish metadata before enabling this stage.

    "mxc-sdk",

.azure-pipelines/1ES.Release.yml:45

• These defaults do not provide the advertised safe preview: with both publish flags false the template expands to no release stages, while enabling publishCrates without also changing the secondary option immediately takes the live ESRP branch because cratesDryRun is false. Default the crate action to a dry run (or provide an equivalent explicit no-op/validation design) so a default run cannot accidentally publish.

    default: false
  - name: cratesDryRun
    displayName: "Crates dry run: stage + verify only, skip the ESRP publish"
    type: boolean
    default: false

.azure-pipelines/templates/Publish.CratesIo.Job.yml:76

• Every rerun invokes ESRP for every crate unconditionally. The only _published_versions call occurs later in wait, so an existing crate version is never skipped and crates.io's duplicate-version rejection will fail a resumed release. Add a pre-publish check that bypasses the ESRP task and wait for versions already present, as promised by the PR's idempotency contract.

                  - task: EsrpRelease@12

.azure-pipelines/scripts/crates_release.py:308

• Returning success after propagation times out defeats the leaf-first ordering guarantee: the next ESRP upload can run while this dependency is still unresolved and fail after earlier crates have already been released. waitforreleasecompletion confirms the ESRP operation, not index visibility, so stop the release when visibility cannot be confirmed.

    print(f"##vso[task.logissue type=warning]{crate} {version} not confirmed on the "
          f"Azure Artifacts feed within {args.timeout}s; continuing (ESRP reported the "
          f"publish complete).")
    return 0

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

.azure-pipelines/scripts/crates_release.py:89

• CRATES is not the mxc-sdk dependency closure: mxc-sdk directly depends on mxc_engine, which is omitted and still has no registry version in src/Cargo.toml; its additional first-party dependencies are likewise unresolved. Because the new Package_Crates stage invokes this command on every official build, cargo package will reject the SDK manifest and no release artifact will be produced. Complete and version the full closure (including resolving the non-publishable bindings crate) before enabling this stage.

CRATES: list[str] = [

.azure-pipelines/1ES.Release.yml:45

• Selecting publishCrates while leaving this option untouched immediately enters the live ESRP branch, contrary to the PR's documented default-on, non-publishing preview. Make dry-run the safe default so an operator must explicitly opt into publication.

    default: false

.azure-pipelines/templates/Publish.CratesIo.Job.yml:34

• This deployment is not bound to the MXC-CratesIo-Production environment, so that environment's manual approval checks will not run; templateContext.isProduction only classifies the 1ES job. Explicitly target the environment described by the release contract before any publish steps execute.

  - deployment: publish_crates
    displayName: Publish crates to crates.io (ESRP)

.azure-pipelines/templates/Publish.CratesIo.Job.yml:76

• Every retry invokes ESRP for every crate unconditionally; the existing-version lookup only runs afterward in wait. Since crates.io rejects an already-published name/version, a retry after a partial release fails on the first completed crate instead of resuming as the PR promises. Add a pre-publish existence check and condition both this task and its wait step on the version being absent.

                  - task: EsrpRelease@12

[dhoehna]

@bbonaby quick process question. I now have write access to microsoft/mxc, so in-repo branches are possible.

As you noted earlier, this fork-based PR can't actually exercise the release pipeline end-to-end: the official build and 1ES.Release are both trigger: none and depend on the MXC-ESRP-Signing variable group / ESRP service connection, none of which is exposed to a fork PR (and a manual queue can't target a fork ref).

Would you like me to abandon this PR and re-create it from an in-repo branch (e.g. user/dhoehna/test-rust-publishing) so we can queue MXC-Official-Build and then 1ES.Release with publishCrates: true + cratesDryRun: true (stage/verify only, nothing published)? Or would you rather keep this PR for the code review and I open a separate in-repo branch purely for the test run? Either works for me — whichever you prefer.

[bbonaby]

@bbonaby quick process question. I now have write access to microsoft/mxc, so in-repo branches are possible.

As you noted earlier, this fork-based PR can't actually exercise the release pipeline end-to-end: the official build and 1ES.Release are both trigger: none and depend on the MXC-ESRP-Signing variable group / ESRP service connection, none of which is exposed to a fork PR (and a manual queue can't target a fork ref).

Would you like me to abandon this PR and re-create it from an in-repo branch (e.g. user/dhoehna/test-rust-publishing) so we can queue MXC-Official-Build and then 1ES.Release with publishCrates: true + cratesDryRun: true (stage/verify only, nothing published)? Or would you rather keep this PR for the code review and I open a separate in-repo branch purely for the test run? Either works for me — whichever you prefer.

Yes let's abandon this one and make a new in repo branch for it. You can do the testing for your branch before you create a PR too. That's what I usually do so I can try things out fast without having Github send PR emails to reviewers about new commits.

[dhoehna]

Closing in favor of an in-repo branch. Per discussion with @bbonaby above, I've moved these changes to user/dhoehna/test-rust-publishing on microsoft/mxc (head 237abe6, full history + all review feedback preserved).

A fork PR can't exercise the release pipeline (the official build and 1ES.Release are trigger: none and depend on the MXC-ESRP-Signing variable group / ESRP service connection, none of which is exposed to forks). From the in-repo branch I'll queue MXC-Official-Build and then 1ES.Release with publishCrates: true + cratesDryRun: true (stage/verify only, nothing published), and open a fresh PR from that branch once the dry run is green. Thanks for the review here!