ci: seed public cargo feed from Cargo.lock on lockfile changes#557
Draft
MGudgin wants to merge 1 commit into
Draft
ci: seed public cargo feed from Cargo.lock on lockfile changes#557MGudgin wants to merge 1 commit into
MGudgin wants to merge 1 commit into
Conversation
Fork PRs lose System.AccessToken and only run the GitHub Actions gates, which build against real crates.io. They never exercise the network-isolated ADO build that redirects crates.io to the anonymous-read MxcDependencies feed, so a fork-PR lockfile bump can introduce a brand-new transitive crate that was never cached in the public feed. The next in-repo PR or main push then fails cargo fetch with a 401 (as seen for futures-task/slab after #534). Add Seed.Cargo.Feed.yml + scripts/ci/seed-cargo-feed.ps1 to authenticated- download every locked crate's .crate file into the feed (which persists each version) whenever Cargo.lock changes, plus a daily safety-net run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an Azure DevOps pipeline + helper script to proactively “seed” the public anonymous-read MxcDependencies Cargo feed by authenticated-downloading every crates.io package pinned in src/Cargo.lock, preventing future cargo fetch 401s in the network-isolated ADO lane after lockfile bumps.
Changes:
- Add
scripts/ci/seed-cargo-feed.ps1to parsesrc/Cargo.lockand download each locked crates.io.crateusing an authenticated Azure Artifacts URL template from the sparse indexconfig.json. - Add
.azure-pipelines/Seed.Cargo.Feed.ymlto run seeding onmainwhen relevant files change and on a daily schedule, using a secret PAT from a variable group. - Document the public feed behavior and the new seeding pipeline in
.azure-pipelines/README.md.
Show a summary per file
| File | Description |
|---|---|
| scripts/ci/seed-cargo-feed.ps1 | New PowerShell seeding script that discovers the feed download template and downloads all locked crates with authentication. |
| .azure-pipelines/Seed.Cargo.Feed.yml | New scheduled/lockfile-triggered ADO pipeline to run the seeding script with a secret PAT. |
| .azure-pipelines/README.md | Adds documentation explaining why seeding is needed and how the new pipeline/script works. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 4
Comment on lines
+84
to
+85
| $IndexUrl = $IndexUrl.TrimEnd('/') | ||
| Write-Host "Public feed index: $IndexUrl" |
Comment on lines
+91
to
+96
| $config = Invoke-RestMethod -Uri "$IndexUrl/config.json" -Headers @{ Authorization = $auth } | ||
| $dlTemplate = $config.dl | ||
| if ([string]::IsNullOrWhiteSpace($dlTemplate)) { | ||
| Write-Host "##[error]Feed config.json did not return a 'dl' download template." | ||
| exit 1 | ||
| } |
Comment on lines
+42
to
+44
| [`Seed.Cargo.Feed.yml`](Seed.Cargo.Feed.yml) closes that gap. It runs on `main` | ||
| whenever `src/Cargo.lock` changes (and on a daily schedule), and authenticated- | ||
| downloads every locked crate's `.crate` file via |
Comment on lines
+19
to
+20
| not enough). This script walks src/Cargo.lock and authenticated-downloads | ||
| every crates.io `.crate` file, which permanently saves each version so the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fork PRs lose System.AccessToken and only run the GitHub Actions gates, which build against real crates.io. They never exercise the network-isolated ADO build that redirects crates.io to the anonymous-read MxcDependencies feed, so a fork-PR lockfile bump can introduce a brand-new transitive crate that was never cached in the public feed. The next in-repo PR or main push then fails cargo fetch with a 401 (as seen for futures-task/slab after #534).
Add Seed.Cargo.Feed.yml + scripts/ci/seed-cargo-feed.ps1 to authenticated- download every locked crate's .crate file into the feed (which persists each version) whenever Cargo.lock changes, plus a daily safety-net run.
📖 Description
🔗 References
🔍 Validation
✅ Checklist
📋 Issue Type
GitHub Actions runs the PR validation build automatically. The ADO pipeline
(
MXC-PR-Build) is the official build pipeline that signs the binaries; itruns on merge to
mainand nightly, and Microsoft reviewers can trigger iton a PR with
/azp run. See docs/pull-requests.md.Microsoft Reviewers: Open in CodeFlow