From 08810f98703d558832fa354c87d0451e9b5b159c Mon Sep 17 00:00:00 2001 From: ripark Date: Thu, 18 Jun 2026 18:39:06 -0700 Subject: [PATCH] Adding in first pass with copilot at doing nightly builds. - Run the individual release pipelines in a loop, from one orchestration job (release-ext-foundry-nightly.yml), There's a staged suffix variable that only gets set when we're doing this kind of aggregate job, so the stages all have unique names. (normally it's blank) - Use a simple versioning standard - whatever the for the plugin + .nightly.buildnumber. - No signing, just FYI. This is similar to what we get for dev builds today - there's also no docs about it, which makes me wonder if signing isn't needed, or if it's annoying and we need to include instructions about how to allow our unsigned binaries. Also, added in some tests just to make sure that our nightly versions, beta versions and normal stable versions all compare okay against each other. --- .../docs/extensions/extension-framework.md | 20 ++ .../extension-resolution-and-versioning.md | 85 +++++++ cli/azd/pkg/extensions/update_checker_test.go | 136 +++++++++++ docs/architecture/extension-framework.md | 1 + eng/pipelines/release-ext-foundry-nightly.yml | 217 ++++++++++++++++++ .../stages/build-and-test-azd-extension.yml | 15 +- .../templates/stages/publish-extension.yml | 189 +++++++++++---- .../stages/release-azd-extension.yml | 18 +- .../templates/stages/sign-extension.yml | 29 ++- .../extension-set-metadata-variables.yml | 9 +- eng/scripts/Set-ExtensionVersionVariable.ps1 | 86 ++++++- 11 files changed, 743 insertions(+), 62 deletions(-) create mode 100644 eng/pipelines/release-ext-foundry-nightly.yml diff --git a/cli/azd/docs/extensions/extension-framework.md b/cli/azd/docs/extensions/extension-framework.md index 70ffe5b178c..db4605ef49e 100644 --- a/cli/azd/docs/extensions/extension-framework.md +++ b/cli/azd/docs/extensions/extension-framework.md @@ -84,6 +84,26 @@ azd extension source add -n dev -t url -l "https://aka.ms/azd/extensions/registr Extensions installed from the dev registry are automatically promoted to the main registry when a newer version becomes available there. See the [Dev/Experimental Extension Registry](./extension-resolution-and-versioning.md#devexperimental-extension-registry) section for full details on stability expectations, submission guidelines, promotion behavior, and troubleshooting. +#### Nightly Registry + +> [!CAUTION] +> The nightly registry contains automated pre-release builds. They come with **no stability guarantees** and are **not covered by Azure support**. Expect breaking changes between nights. + +The nightly registry is an opt-in source containing daily pre-release builds of first-party extensions, published every night from the latest `main`. It lives on the `nightly-registry` branch of the [`azd` GitHub repo](https://github.com/Azure/azure-dev/blob/nightly-registry/cli/azd/extensions/registry.nightly.json) and is **not** configured by default. + +To opt-in for the nightly registry run the following command: + +```bash +# Add a new extension source named 'nightly' to your `azd` configuration. +azd extension source add -n nightly -t url \ + -l "https://raw.githubusercontent.com/Azure/azure-dev/nightly-registry/cli/azd/extensions/registry.nightly.json" + +# Install the nightly build of an extension. +azd extension install --source nightly +``` + +Nightly builds use a `-nightly..` version format, so the matching stable release always supersedes them — running `azd extension upgrade` moves you back to the `azd` source automatically once that stable version ships. See [Choosing Stable vs. Nightly](./extension-resolution-and-versioning.md#choosing-stable-vs-nightly) for the full workflow, including how to switch channels and opt back out. + #### `azd extension source list` Displays a list of installed extension sources. diff --git a/cli/azd/docs/extensions/extension-resolution-and-versioning.md b/cli/azd/docs/extensions/extension-resolution-and-versioning.md index 79120cec210..851d82c91c8 100644 --- a/cli/azd/docs/extensions/extension-resolution-and-versioning.md +++ b/cli/azd/docs/extensions/extension-resolution-and-versioning.md @@ -547,6 +547,91 @@ If the dev registry URL is unreachable (network issue, DNS failure), operations azd extension source remove dev ``` +## Nightly Extension Registry + +The nightly registry is a separate, opt-in extension source that contains automated daily +pre-release builds of first-party extensions. A scheduled pipeline builds each extension from +the latest `main` and publishes a new prerelease version to the registry every night, so the +nightly source always tracks the most recent in-development code. + +Like `dev`, the nightly source is **not** configured by default and is **not** a replacement for +the stable `azd` registry. It is a parallel pre-release channel for early adopters who want to +validate upcoming changes before they ship. + +| Property | Main Registry | Nightly Registry | +|----------|---------------|------------------| +| Source file | `cli/azd/extensions/registry.json` | `cli/azd/extensions/registry.nightly.json` | +| Location | `https://aka.ms/azd/extensions/registry` | `https://raw.githubusercontent.com/Azure/azure-dev/nightly-registry/cli/azd/extensions/registry.nightly.json` | +| Source name | `azd` (built-in default) | `nightly` (opt-in) | +| Cadence | On stable release | Every night from `main` | +| Stability | Stable releases | **Pre-release; may change or break nightly** | +| Support | Covered by Azure support | **Not covered** | + +### Nightly Version Format + +Nightly builds are stamped by [`eng/scripts/Set-ExtensionVersionVariable.ps1`](../../../../eng/scripts/Set-ExtensionVersionVariable.ps1) +into a semver-valid prerelease derived from the extension's `version.txt`: + +```text +-nightly.. e.g. 0.1.0-nightly.20260618.1234 +.nightly.. (when already has a prerelease label) +``` + +Because these are valid prereleases, semver precedence orders them the way you would expect: + +- A stable release always outranks any nightly built from the same base + (`0.1.0-nightly.20260618.1234` < `0.1.0`), so the stable version supersedes the nightly once it ships. +- A later nightly outranks an earlier one, comparing the date and then the build id numerically + (`...20260618.1234` < `...20260618.5678` and `...20260618.9999` < `...20260619.1000`). + +### Choosing Stable vs. Nightly + +The two channels coexist; you decide per extension which one to install from. + +**Stay on stable (default):** Do nothing. Without the `nightly` source configured, every +`azd extension install` and `azd extension upgrade` resolves against the official `azd` registry only. + +**Opt in to nightly:** Add the source, then install with `--source nightly`: + +```bash +# Add the nightly source once. +azd extension source add -n nightly -t url \ + -l "https://raw.githubusercontent.com/Azure/azure-dev/nightly-registry/cli/azd/extensions/registry.nightly.json" + +# Install the nightly build of an extension. +azd extension install --source nightly + +# Confirm both sources are configured. +azd extension source list +``` + +When an extension exists in both `azd` and `nightly` and you omit `--source`, `azd` prompts you to +choose (interactive) or errors with `"found in multiple sources"` (non-interactive). See +[Troubleshooting Multi-Registry Scenarios](#troubleshooting-multi-registry-scenarios). + +**Switch back to stable:** The nightly source uses the same one-way promotion logic as the dev +registry — see [Upgrade and Dev→Main Promotion](#upgrade-and-devmain-promotion). Because a stable +release always outranks the nightly built from the same base, running `azd extension upgrade` moves +you from `nightly` back to the `azd` source automatically once the corresponding stable version is +published. The extension's stored source flips from `nightly` to `azd`, and it stays on stable +from then on. To force the switch immediately (for example, to leave a nightly that is ahead of the +current stable), reinstall explicitly: + +```bash +# --force is required only when the stable version is lower than the installed nightly. +azd extension install --source azd --force +``` + +**Leave nightly entirely:** Remove the source to stop pulling nightly builds: + +```bash +azd extension source remove nightly +``` + +> [!CAUTION] +> Nightly builds come with **no stability guarantees** and are **not covered by Azure support**. +> Expect breaking changes between nights. Use a stable source for production workflows. + ## Related Documentation | Document | Description | diff --git a/cli/azd/pkg/extensions/update_checker_test.go b/cli/azd/pkg/extensions/update_checker_test.go index 3b66f1d8603..f85799bb529 100644 --- a/cli/azd/pkg/extensions/update_checker_test.go +++ b/cli/azd/pkg/extensions/update_checker_test.go @@ -249,3 +249,139 @@ func Test_UpdateChecker_InvalidVersions(t *testing.T) { require.NoError(t, err) require.False(t, result.HasUpdate) // Should gracefully handle invalid version } + +// Test_UpdateChecker_NightlyVersions verifies that the update checker correctly +// orders the nightly prerelease versions produced by Set-ExtensionVersionVariable.ps1. +// +// Nightly versions built from a base without an existing prerelease take the shape: +// +// -nightly.. e.g. 0.1.0-nightly.20260618.1234 +// +// Several transitions must hold for the upgrade experience to work: +// - nightly -> stable: once the stable ships it must supersede any nightly +// built from that base (semver: a release outranks its prereleases). +// - nightly -> next nightly: a later nightly (newer date, or same date with a +// higher build id) must supersede an earlier one, and an earlier nightly must +// never be offered as an update over a newer one. +// - nightly vs. milestone prereleases: for a clean base, semver orders +// alpha < beta < nightly < rc < stable, so a nightly supersedes alpha/beta of the +// same base but is itself superseded by rc and the final release. When the base +// already carries a prerelease label (e.g. 0.1.0-beta), the appended ".nightly" +// segment outranks every numbered beta of that base, so only the final release +// (or a higher base) supersedes it. +func Test_UpdateChecker_NightlyVersions(t *testing.T) { + tests := []struct { + name string + installed string + available []string + wantUpdate bool + }{ + { + name: "nightly to stable", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234", "0.1.0"}, + wantUpdate: true, + }, + { + name: "nightly to next nightly newer date", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234", "0.1.0-nightly.20260619.1234"}, + wantUpdate: true, + }, + { + name: "nightly to next nightly same date higher build", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234", "0.1.0-nightly.20260618.5678"}, + wantUpdate: true, + }, + { + name: "same nightly is not an update", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234"}, + wantUpdate: false, + }, + { + name: "older nightly is not offered over newer installed", + installed: "0.1.0-nightly.20260619.1234", + available: []string{"0.1.0-nightly.20260618.1234"}, + wantUpdate: false, + }, + { + // Clean base: nightly outranks beta of the same base (b < n). + name: "nightly supersedes beta of same base", + installed: "0.1.0-beta.1", + available: []string{"0.1.0-beta.1", "0.1.0-nightly.20260618.1234"}, + wantUpdate: true, + }, + { + // Clean base: an older beta must never be offered over an installed nightly. + name: "beta is not offered over installed nightly", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-beta.5", "0.1.0-nightly.20260618.1234"}, + wantUpdate: false, + }, + { + // Clean base: rc outranks nightly of the same base (n < r). + name: "rc supersedes nightly of same base", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234", "0.1.0-rc.1"}, + wantUpdate: true, + }, + { + // Prerelease base gotcha: "0.1.0-beta.nightly..." outranks every numbered + // beta of that base because numeric identifiers lose to alphanumeric ones, + // so a later beta.N is NOT offered as an update. + name: "numbered beta does not supersede beta-base nightly", + installed: "0.1.0-beta.nightly.20260618.1234", + available: []string{"0.1.0-beta.2", "0.1.0-beta.nightly.20260618.1234"}, + wantUpdate: false, + }, + { + // Prerelease base: only the final stable release supersedes the beta-base nightly. + name: "stable supersedes beta-base nightly", + installed: "0.1.0-beta.nightly.20260618.1234", + available: []string{"0.1.0-beta.nightly.20260618.1234", "0.1.0"}, + wantUpdate: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + tempDir := t.TempDir() + t.Setenv("AZD_CONFIG_DIR", tempDir) + + cacheManager, err := NewRegistryCacheManager() + require.NoError(t, err) + + ctx := t.Context() + sourceName := "test-source" + + versions := make([]ExtensionVersion, 0, len(tt.available)) + for _, v := range tt.available { + versions = append(versions, ExtensionVersion{Version: v}) + } + + err = cacheManager.Set(ctx, sourceName, []*ExtensionMetadata{ + { + Id: "test.extension", + DisplayName: "Test Extension", + Versions: versions, + }, + }) + require.NoError(t, err) + + updateChecker := NewUpdateChecker(cacheManager) + + extension := &Extension{ + Id: "test.extension", + DisplayName: "Test Extension", + Version: tt.installed, + Source: sourceName, + } + + result, err := updateChecker.CheckForUpdate(ctx, extension) + require.NoError(t, err) + require.Equal(t, tt.wantUpdate, result.HasUpdate) + }) + } +} diff --git a/docs/architecture/extension-framework.md b/docs/architecture/extension-framework.md index 6f961f5430a..7c8d4943383 100644 --- a/docs/architecture/extension-framework.md +++ b/docs/architecture/extension-framework.md @@ -25,6 +25,7 @@ Extensions are discovered from registries — JSON manifests that list available - **Official registry:** `https://aka.ms/azd/extensions/registry` — Stable, signed, production-ready extensions vetted by the azd team. - **Dev registry:** `https://aka.ms/azd/extensions/registry/dev` — Experimental and pre-release extensions (unsigned builds, backed by `cli/azd/extensions/registry.dev.json`). Not configured by default; users opt in with `azd extension source add`. +- **Nightly registry:** `https://raw.githubusercontent.com/Azure/azure-dev/nightly-registry/cli/azd/extensions/registry.nightly.json` — Automated daily pre-release builds of first-party extensions, published from the latest `main` and backed by `registry.nightly.json` on the `nightly-registry` branch. Not configured by default; users opt in with `azd extension source add -n nightly`. See [Choosing Stable vs. Nightly](../../cli/azd/docs/extensions/extension-resolution-and-versioning.md#choosing-stable-vs-nightly). - **Local sources:** File-based manifests for development The dev registry serves as a staging area for extensions before they graduate to the main registry. Extensions installed from the dev registry are automatically promoted to the main registry when a newer stable version becomes available there. See the [Extension Resolution and Versioning](../../cli/azd/docs/extensions/extension-resolution-and-versioning.md#devexperimental-extension-registry) guide for detailed criteria, stability expectations, and submission guidelines. diff --git a/eng/pipelines/release-ext-foundry-nightly.yml b/eng/pipelines/release-ext-foundry-nightly.yml new file mode 100644 index 00000000000..29dc945231c --- /dev/null +++ b/eng/pipelines/release-ext-foundry-nightly.yml @@ -0,0 +1,217 @@ +# Single nightly publishing pipeline for the Microsoft Foundry bundle. +# +# One pipeline builds every Foundry extension from the latest commit on main each night, in parallel, +# and then updates the nightly extension registry ONCE for the whole bundle. This avoids the +# concurrent-writer problem you get when each extension publishes to the shared nightly-registry +# branch from its own pipeline. +# +# How the reuse works: +# * The per-extension build -> sign -> publish-release chain is the standard +# stages/release-azd-extension.yml template, instantiated once per extension via an ${{ each }} +# loop. StageSuffix (= _) makes each instance's stage ids and shared artifact names +# unique so they can coexist in one pipeline. +# * Skip.RegistryUpdate = true turns off the per-extension registry update job; the single +# Update_Nightly_Registry stage below owns the registry write instead. +# * AZD_NIGHTLY = true makes eng/scripts/Set-ExtensionVersionVariable.ps1 emit nightly semvers. +# +# The microsoft.foundry meta-package has no binaries, so it is NOT part of the build/sign fan-out. It +# is published last, in the join stage, after its dependencies are already in the nightly registry. + +trigger: none +pr: none + +schedules: + - cron: "0 7 * * *" + displayName: Daily Foundry nightly build (07:00 UTC) + branches: + include: + - main + # Build even when there were no commits since the last successful scheduled run. + always: true + +parameters: + # The Foundry dependency extensions that are built, signed, and released as nightly artifacts. + - name: Extensions + type: object + default: + - { id: azure.ai.agents, sanitized: azure-ai-agents, dir: cli/azd/extensions/azure.ai.agents, skipTests: false } + - { id: azure.ai.connections, sanitized: azure-ai-connections, dir: cli/azd/extensions/azure.ai.connections, skipTests: false } + - { id: azure.ai.inspector, sanitized: azure-ai-inspector, dir: cli/azd/extensions/azure.ai.inspector, skipTests: false } + - { id: azure.ai.projects, sanitized: azure-ai-projects, dir: cli/azd/extensions/azure.ai.projects, skipTests: false } + - { id: azure.ai.routines, sanitized: azure-ai-routines, dir: cli/azd/extensions/azure.ai.routines, skipTests: false } + - { id: azure.ai.skills, sanitized: azure-ai-skills, dir: cli/azd/extensions/azure.ai.skills, skipTests: false } + - { id: azure.ai.toolboxes, sanitized: azure-ai-toolboxes, dir: cli/azd/extensions/azure.ai.toolboxes, skipTests: false } + # The Foundry meta-package (extension pack). Published last, registry-only (no build/sign/release). + - name: PackId + type: string + default: microsoft.foundry + - name: PackDirectory + type: string + default: cli/azd/extensions/microsoft.foundry + +variables: + # Makes the shared version script emit -nightly.. for every build job. + - name: AZD_NIGHTLY + value: "true" + # The per-extension registry update job is skipped; the single join stage owns the registry write. + - name: Skip.RegistryUpdate + value: "true" + - name: NightlyRegistryBranch + value: nightly-registry + - name: ExtensionRegistryFile + value: registry.nightly.json + +extends: + template: /eng/pipelines/templates/stages/1es-redirect.yml + parameters: + stages: + # Fan out: build + sign + publish-release for every dependency extension, all in parallel. + - ${{ each ext in parameters.Extensions }}: + - template: /eng/pipelines/templates/stages/release-azd-extension.yml + parameters: + StageSuffix: _${{ ext.sanitized }} + IsNightly: true + SkipTests: ${{ ext.skipTests }} + AzdExtensionId: ${{ ext.id }} + SanitizedExtensionId: ${{ ext.sanitized }} + AzdExtensionDirectory: ${{ ext.dir }} + + # Join: update the nightly registry once for the whole bundle. Gated identically to the + # per-extension publish stages so its dependsOn never references stages that were not emitted. + - ${{ if and(eq(variables['System.TeamProject'], 'internal'), or(eq(variables['Build.Reason'], 'Manual'), eq(variables['Build.Reason'], 'Schedule'))) }}: + - stage: Update_Nightly_Registry + dependsOn: + - ${{ each ext in parameters.Extensions }}: + - PublishExtension_${{ ext.sanitized }} + + variables: + - template: /eng/pipelines/templates/variables/image.yml + - template: /eng/pipelines/templates/variables/globals.yml + + jobs: + - job: Update + condition: >- + and( + succeeded(), + ne('true', variables['Skip.RegistryUpdate.Bundle']) + ) + + pool: + name: $(LINUXPOOL) + image: $(LINUXVMIMAGE) + os: linux + + steps: + - checkout: self + + - bash: | + set -euo pipefail + curl -fsSL https://aka.ms/install-azd.sh | bash -s -- --verbose + azd version + displayName: Install azd + + - bash: | + set -euo pipefail + azd ext install microsoft.azd.extensions --source azd + displayName: Install microsoft.azd.extensions + + # Check out (or create) the nightly registry branch and seed an empty registry the + # first time. All per-extension publishes below mutate this single file in place. + - bash: | + set -euo pipefail + + branch="$(NightlyRegistryBranch)" + registry_path="cli/azd/extensions/$(ExtensionRegistryFile)" + + git config user.name "azure-sdk" + git config user.email "azuresdk@microsoft.com" + # checkout: self does not persist push credentials. + git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/$(Build.Repository.Name).git" + + git fetch origin "$branch" || true + if git show-ref --verify --quiet "refs/remotes/origin/${branch}"; then + git checkout -B "$branch" "origin/${branch}" + else + git checkout -B "$branch" + fi + + if [ ! -f "$registry_path" ]; then + echo '{ "extensions": [] }' > "$registry_path" + fi + displayName: Prepare nightly registry branch + env: + GH_TOKEN: $(azuresdk-github-pat) + + # Publish each dependency extension into the single registry file (no commit yet). + # The exact nightly version comes from each extension's release metadata artifact. + - ${{ each ext in parameters.Extensions }}: + - task: DownloadPipelineArtifact@2 + displayName: Download ${{ ext.sanitized }} metadata + inputs: + artifactName: release-metadata_${{ ext.sanitized }} + targetPath: $(Pipeline.Workspace)/meta-${{ ext.sanitized }} + + - bash: | + set -euo pipefail + version=$(jq -r '.extVersion' "$(Pipeline.Workspace)/meta-${{ ext.sanitized }}/metadata.json") + echo "Publishing ${{ ext.id }} ${version} to nightly registry" + ( cd "${{ ext.dir }}" && azd x publish \ + --registry "../$(ExtensionRegistryFile)" \ + --repo "$(Build.Repository.Name)" \ + --version "$version" ) + displayName: Publish ${{ ext.sanitized }} to nightly registry + env: + GH_TOKEN: $(azuresdk-github-pat) + + # Publish the Foundry pack last, after its dependencies exist in the registry. The pack + # is registry-only (no GitHub release) so --repo is omitted. Its nightly version is + # computed from its own version.txt by the shared version script (AZD_NIGHTLY=true). + - task: PowerShell@2 + displayName: Compute Foundry pack nightly version + inputs: + pwsh: true + targetType: filePath + filePath: eng/scripts/Set-ExtensionVersionVariable.ps1 + arguments: >- + -ExtensionDirectory ${{ parameters.PackDirectory }} + + - bash: | + set -euo pipefail + echo "Publishing ${{ parameters.PackId }} ${EXT_VERSION} to nightly registry" + ( cd "${{ parameters.PackDirectory }}" && azd x publish \ + --registry "../$(ExtensionRegistryFile)" \ + --version "$EXT_VERSION" ) + displayName: Publish Foundry pack to nightly registry + env: + GH_TOKEN: $(azuresdk-github-pat) + + # Commit the whole bundle's registry changes once and push (single fetch+rebase retry). + - bash: | + set -euo pipefail + + branch="$(NightlyRegistryBranch)" + registry_path="cli/azd/extensions/$(ExtensionRegistryFile)" + + git add "$registry_path" + if git diff --cached --quiet; then + echo "No registry changes to commit." + exit 0 + fi + + for attempt in 1 2 3; do + git commit -m "[foundry-nightly] Nightly registry update ($(Build.BuildNumber))" || true + if git push origin "$branch"; then + echo "Nightly registry updated on ${branch}." + exit 0 + fi + if [ "$attempt" -eq 3 ]; then + echo "Failed to update nightly registry after 3 attempts." >&2 + exit 1 + fi + echo "Push rejected; rebasing onto latest and retrying (${attempt})..." + git fetch origin "$branch" + git reset --soft "origin/${branch}" + done + displayName: Commit and push nightly registry + env: + GH_TOKEN: $(azuresdk-github-pat) diff --git a/eng/pipelines/templates/stages/build-and-test-azd-extension.yml b/eng/pipelines/templates/stages/build-and-test-azd-extension.yml index 703f6e822e0..0420cef5b1b 100644 --- a/eng/pipelines/templates/stages/build-and-test-azd-extension.yml +++ b/eng/pipelines/templates/stages/build-and-test-azd-extension.yml @@ -8,6 +8,15 @@ parameters: - name: SkipTests type: boolean default: false + # StageSuffix is appended to this template's stage id AND to the shared pipeline artifact names it + # produces (release-metadata, changelog) so the template can be instantiated multiple times in a + # single pipeline via an ${{ each }} loop (e.g. the nightly bundle pipeline that builds many + # extensions in parallel). Azure DevOps requires stage ids and artifact names to be unique within a + # pipeline; without a suffix, looping would collide on "BuildAndTest"/"release-metadata"/"changelog" + # and fail. Defaults to '' so existing single-extension pipelines are byte-identical and unaffected. + - name: StageSuffix + type: string + default: '' - name: CrossBuildMatrix type: object default: @@ -66,7 +75,7 @@ parameters: GOOS: windows GOARCH: arm64 stages: - - stage: BuildAndTest + - stage: BuildAndTest${{ parameters.StageSuffix }} variables: - template: /eng/pipelines/templates/variables/globals.yml - template: /eng/pipelines/templates/variables/image.yml @@ -162,10 +171,10 @@ stages: - output: pipelineArtifact path: release-metadata condition: succeeded() - artifact: release-metadata + artifact: release-metadata${{ parameters.StageSuffix }} displayName: Upload release metadata - output: pipelineArtifact path: changelog - artifact: changelog + artifact: changelog${{ parameters.StageSuffix }} displayName: Upload changelog diff --git a/eng/pipelines/templates/stages/publish-extension.yml b/eng/pipelines/templates/stages/publish-extension.yml index 67d73873e5a..fcb46489a31 100644 --- a/eng/pipelines/templates/stages/publish-extension.yml +++ b/eng/pipelines/templates/stages/publish-extension.yml @@ -8,27 +8,64 @@ parameters: - name: PublishToDevRegistry type: boolean default: false + # When true the build publishes to the nightly registry (registry.nightly.json) which lives on a + # dedicated, bot-owned branch and is updated by auto-commit instead of a pull request. + - name: IsNightly + type: boolean + default: false + # StageSuffix is appended to this stage's id (and its dependsOn) AND to the shared pipeline artifact + # names it consumes (release, changelog, release-metadata) so the template can be instantiated + # multiple times in a single pipeline via an ${{ each }} loop (e.g. the nightly bundle pipeline that + # builds many extensions in parallel). Azure DevOps requires stage ids and artifact names to be + # unique within a pipeline; without a suffix, looping would collide and fail to compile. Defaults to + # '' so existing single-extension pipelines are byte-identical and unaffected. + - name: StageSuffix + type: string + default: '' stages: - - stage: PublishExtension - dependsOn: Sign - condition: >- - and( - succeeded(), - ne(variables['Skip.Release'], 'true'), - or( - eq('Manual', variables['BuildReasonOverride']), - and( - eq('', variables['BuildReasonOverride']), - eq('Manual', variables['Build.Reason']) + - stage: PublishExtension${{ parameters.StageSuffix }} + dependsOn: Sign${{ parameters.StageSuffix }} + ${{ if eq(parameters.IsNightly, true) }}: + # Nightly publishes run on a schedule (or when queued manually). + condition: >- + and( + succeeded(), + ne(variables['Skip.Release'], 'true'), + or( + eq('Manual', variables['BuildReasonOverride']), + and( + eq('', variables['BuildReasonOverride']), + or( + eq('Manual', variables['Build.Reason']), + eq('Schedule', variables['Build.Reason']) + ) + ) + ) + ) + ${{ else }}: + condition: >- + and( + succeeded(), + ne(variables['Skip.Release'], 'true'), + or( + eq('Manual', variables['BuildReasonOverride']), + and( + eq('', variables['BuildReasonOverride']), + eq('Manual', variables['Build.Reason']) + ) ) ) - ) variables: - template: /eng/pipelines/templates/variables/image.yml - template: /eng/pipelines/templates/variables/globals.yml - - ${{ if eq(parameters.PublishToDevRegistry, true) }}: + - ${{ if eq(parameters.IsNightly, true) }}: + - name: ExtensionRegistryFile + value: registry.nightly.json + - name: NightlyRegistryBranch + value: nightly-registry + - ${{ elseif eq(parameters.PublishToDevRegistry, true) }}: - name: ExtensionRegistryFile value: registry.dev.json - name: ExtensionRegistryPullRequestPrefix @@ -62,11 +99,11 @@ stages: isProduction: true inputs: - input: pipelineArtifact - artifactName: release + artifactName: release${{ parameters.StageSuffix }} targetPath: release - input: pipelineArtifact - artifactName: changelog + artifactName: changelog${{ parameters.StageSuffix }} targetPath: changelog strategy: @@ -76,6 +113,7 @@ stages: - template: /eng/pipelines/templates/steps/extension-set-metadata-variables.yml parameters: Use1ESArtifactTask: true + ArtifactSuffix: ${{ parameters.StageSuffix }} - pwsh: | # Initial upload locations @@ -111,6 +149,8 @@ stages: - checkout: self - template: /eng/pipelines/templates/steps/extension-set-metadata-variables.yml + parameters: + ArtifactSuffix: ${{ parameters.StageSuffix }} - template: /eng/pipelines/templates/steps/setup-go.yml @@ -125,33 +165,100 @@ stages: azd ext install microsoft.azd.extensions --source azd displayName: Install microsoft.azd.extensions - - bash: | - set -euo pipefail - cd "${{ parameters.AzdExtensionDirectory }}" - azd x publish \ - --registry "../$(ExtensionRegistryFile)" \ - --repo "$(Build.Repository.Name)" \ - --version "$(EXT_VERSION)" - displayName: Update $(ExtensionRegistryFile) (azd x publish) - env: - GH_TOKEN: $(azuresdk-github-pat) - - - ${{ if ne(parameters.PublishToDevRegistry, true) }}: + - ${{ if eq(parameters.IsNightly, true) }}: + # Nightly registry lives on a dedicated, bot-owned branch and is updated by auto-commit + # (no pull request). A publish -> commit -> push retry loop keeps concurrent extension + # nightly runs from clobbering each other's entries on the shared branch. - bash: | set -euo pipefail - cd cli/azd - go build . - go test ./cmd -run 'TestFigSpec|TestUsage' - displayName: Refresh Fig/Usage snapshots + + branch="$(NightlyRegistryBranch)" + registry_rel="$(ExtensionRegistryFile)" + registry_path="cli/azd/extensions/${registry_rel}" + ext_dir="${{ parameters.AzdExtensionDirectory }}" + + git config user.name "azure-sdk" + git config user.email "azuresdk@microsoft.com" + + # checkout: self does not persist push credentials, so authenticate the remote with + # the bot PAT for the fetch/push of the nightly registry branch. + git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/$(Build.Repository.Name).git" + + publish_registry() { + # azd x publish loads an existing registry; seed an empty one on first nightly. + if [ ! -f "$registry_path" ]; then + echo '{ "extensions": [] }' > "$registry_path" + fi + ( cd "$ext_dir" && azd x publish \ + --registry "../${registry_rel}" \ + --repo "$(Build.Repository.Name)" \ + --version "$(EXT_VERSION)" ) + } + + # Check out (or create) the nightly registry branch. + git fetch origin "$branch" || true + if git show-ref --verify --quiet "refs/remotes/origin/${branch}"; then + git checkout -B "$branch" "origin/${branch}" + else + git checkout -B "$branch" + fi + + for attempt in 1 2 3 4 5; do + publish_registry + git add "$registry_path" + if git diff --cached --quiet; then + echo "No registry changes to commit." + break + fi + + git commit -m "[${{ parameters.AzdExtensionId }}] Nightly registry update for $(EXT_VERSION)" + if git push origin "$branch"; then + echo "Nightly registry updated on ${branch}." + break + fi + + if [ "$attempt" -eq 5 ]; then + echo "Failed to update nightly registry after 5 attempts." >&2 + exit 1 + fi + + echo "Push rejected (branch advanced). Rebasing onto latest and retrying (${attempt})..." + git reset --hard HEAD~1 + git fetch origin "$branch" + git reset --hard "origin/${branch}" + done + displayName: Update $(ExtensionRegistryFile) (azd x publish + auto-commit) env: - UPDATE_SNAPSHOTS: "true" + GH_TOKEN: $(azuresdk-github-pat) - - template: /eng/common/pipelines/templates/steps/create-pull-request.yml - parameters: - PRBranchName: $(ExtensionRegistryPullRequestPrefix)/${{ parameters.SanitizedExtensionId }}/$(EXT_VERSION)-$(Build.BuildId) - CommitMsg: "[${{ parameters.AzdExtensionId }}] $(ExtensionRegistryPullRequestTitle) for $(EXT_VERSION)" - PRTitle: "[${{ parameters.AzdExtensionId }}] $(ExtensionRegistryPullRequestTitle) for $(EXT_VERSION)" - PRBody: >- - $(ExtensionRegistryPullRequestTitle) for the - [azd-ext-${{ parameters.SanitizedExtensionId }}_$(EXT_VERSION)](https://github.com/$(Build.Repository.Name)/releases/tag/azd-ext-${{ parameters.SanitizedExtensionId }}_$(EXT_VERSION)) - release. + - ${{ else }}: + - bash: | + set -euo pipefail + cd "${{ parameters.AzdExtensionDirectory }}" + azd x publish \ + --registry "../$(ExtensionRegistryFile)" \ + --repo "$(Build.Repository.Name)" \ + --version "$(EXT_VERSION)" + displayName: Update $(ExtensionRegistryFile) (azd x publish) + env: + GH_TOKEN: $(azuresdk-github-pat) + + - ${{ if ne(parameters.PublishToDevRegistry, true) }}: + - bash: | + set -euo pipefail + cd cli/azd + go build . + go test ./cmd -run 'TestFigSpec|TestUsage' + displayName: Refresh Fig/Usage snapshots + env: + UPDATE_SNAPSHOTS: "true" + + - template: /eng/common/pipelines/templates/steps/create-pull-request.yml + parameters: + PRBranchName: $(ExtensionRegistryPullRequestPrefix)/${{ parameters.SanitizedExtensionId }}/$(EXT_VERSION)-$(Build.BuildId) + CommitMsg: "[${{ parameters.AzdExtensionId }}] $(ExtensionRegistryPullRequestTitle) for $(EXT_VERSION)" + PRTitle: "[${{ parameters.AzdExtensionId }}] $(ExtensionRegistryPullRequestTitle) for $(EXT_VERSION)" + PRBody: >- + $(ExtensionRegistryPullRequestTitle) for the + [azd-ext-${{ parameters.SanitizedExtensionId }}_$(EXT_VERSION)](https://github.com/$(Build.Repository.Name)/releases/tag/azd-ext-${{ parameters.SanitizedExtensionId }}_$(EXT_VERSION)) + release. diff --git a/eng/pipelines/templates/stages/release-azd-extension.yml b/eng/pipelines/templates/stages/release-azd-extension.yml index a2f656bee85..bc06f406bac 100644 --- a/eng/pipelines/templates/stages/release-azd-extension.yml +++ b/eng/pipelines/templates/stages/release-azd-extension.yml @@ -11,10 +11,21 @@ parameters: - name: PublishToDevRegistry type: boolean default: false + # When true the release publishes to the nightly registry and is allowed to run on a schedule. + - name: IsNightly + type: boolean + default: false + # StageSuffix is forwarded to every child stage template so this whole build/sign/publish chain can + # be instantiated once per extension via an ${{ each }} loop in a single pipeline (e.g. the nightly + # bundle). Defaults to '' so existing single-extension pipelines are unaffected. + - name: StageSuffix + type: string + default: '' stages: - template: /eng/pipelines/templates/stages/build-and-test-azd-extension.yml parameters: + StageSuffix: ${{ parameters.StageSuffix }} AzdExtensionId: ${{ parameters.AzdExtensionId }} AzdExtensionDirectory: ${{ parameters.AzdExtensionDirectory }} SkipTests: ${{ parameters.SkipTests }} @@ -78,11 +89,12 @@ stages: SetExecutableBit: true AZURE_DEV_CI_OS: mac-arm64 - # Only sign and release on manual builds from internal - - ${{ if and(eq(variables['System.TeamProject'], 'internal'), eq(variables['Build.Reason'], 'Manual')) }}: + # Only sign and release on manual builds from internal, or scheduled nightly builds from internal. + - ${{ if and(eq(variables['System.TeamProject'], 'internal'), or(eq(variables['Build.Reason'], 'Manual'), and(eq(parameters.IsNightly, true), eq(variables['Build.Reason'], 'Schedule')))) }}: - template: /eng/pipelines/templates/stages/sign-extension.yml parameters: SanitizedExtensionId: ${{ parameters.SanitizedExtensionId }} + StageSuffix: ${{ parameters.StageSuffix }} - template: /eng/pipelines/templates/stages/publish-extension.yml parameters: @@ -90,3 +102,5 @@ stages: AzdExtensionDirectory: ${{ parameters.AzdExtensionDirectory }} SanitizedExtensionId: ${{ parameters.SanitizedExtensionId }} PublishToDevRegistry: ${{ parameters.PublishToDevRegistry }} + IsNightly: ${{ parameters.IsNightly }} + StageSuffix: ${{ parameters.StageSuffix }} diff --git a/eng/pipelines/templates/stages/sign-extension.yml b/eng/pipelines/templates/stages/sign-extension.yml index c5715ba1ff0..25b0ae94664 100644 --- a/eng/pipelines/templates/stages/sign-extension.yml +++ b/eng/pipelines/templates/stages/sign-extension.yml @@ -1,10 +1,19 @@ parameters: - name: SanitizedExtensionId type: string + # StageSuffix is appended to this stage's id (and its dependsOn) AND to the shared pipeline artifact + # names it produces/consumes (signed-mac, signed-win, release, release-metadata) so the template can + # be instantiated multiple times in a single pipeline via an ${{ each }} loop (e.g. the nightly + # bundle pipeline that builds many extensions in parallel). Azure DevOps requires stage ids and + # artifact names to be unique within a pipeline; without a suffix, looping would collide and fail to + # compile. Defaults to '' so existing single-extension pipelines are byte-identical and unaffected. + - name: StageSuffix + type: string + default: '' stages: - - stage: Sign - dependsOn: BuildAndTest + - stage: Sign${{ parameters.StageSuffix }} + dependsOn: BuildAndTest${{ parameters.StageSuffix }} variables: - template: /eng/pipelines/templates/variables/globals.yml @@ -73,14 +82,14 @@ stages: condition: succeeded() displayName: Publish Signed Artifacts inputs: - artifactName: signed-mac + artifactName: signed-mac${{ parameters.StageSuffix }} path: signed-mac/ - task: 1ES.PublishPipelineArtifact@1 condition: failed() displayName: Publish failed Signed Artifacts inputs: - artifactName: signed-mac-FailedAttempt$(System.JobAttempt) + artifactName: signed-mac${{ parameters.StageSuffix }}-FailedAttempt$(System.JobAttempt) path: signed-mac/ - job: SignWindows @@ -140,14 +149,14 @@ stages: condition: succeeded() displayName: Publish Signed Artifacts inputs: - artifactName: signed-win + artifactName: signed-win${{ parameters.StageSuffix }} path: signed-win/ - task: 1ES.PublishPipelineArtifact@1 condition: failed() displayName: Publish failed Signed Artifacts inputs: - artifactName: signed-win-FailedAttempt$(System.JobAttempt) + artifactName: signed-win${{ parameters.StageSuffix }}-FailedAttempt$(System.JobAttempt) path: signed-win/ - job: CreateRelease @@ -164,24 +173,24 @@ stages: outputs: - output: pipelineArtifact path: release - artifact: release + artifact: release${{ parameters.StageSuffix }} condition: succeeded() displayName: Upload azd release artifact steps: - task: DownloadPipelineArtifact@2 inputs: - artifactName: release-metadata + artifactName: release-metadata${{ parameters.StageSuffix }} targetPath: release-metadata - task: DownloadPipelineArtifact@2 inputs: - artifactName: signed-win + artifactName: signed-win${{ parameters.StageSuffix }} targetPath: signed/win - task: DownloadPipelineArtifact@2 inputs: - artifactName: signed-mac + artifactName: signed-mac${{ parameters.StageSuffix }} targetPath: signed/mac # Linux binaries are not signed today so download from build outputs diff --git a/eng/pipelines/templates/steps/extension-set-metadata-variables.yml b/eng/pipelines/templates/steps/extension-set-metadata-variables.yml index 21f942f6ab2..e9fa7eef14e 100644 --- a/eng/pipelines/templates/steps/extension-set-metadata-variables.yml +++ b/eng/pipelines/templates/steps/extension-set-metadata-variables.yml @@ -2,6 +2,11 @@ parameters: - name: Use1ESArtifactTask type: boolean default: false + # Suffix appended to the release-metadata artifact name so this step can consume the per-extension + # artifact produced by a looped build stage in a single multi-extension pipeline. Defaults to ''. + - name: ArtifactSuffix + type: string + default: '' steps: - pwsh: | @@ -19,14 +24,14 @@ steps: condition: eq(variables['DownloadMetadataArtifact'], 'true') displayName: Download release metadata artifact inputs: - artifactName: release-metadata + artifactName: release-metadata${{ parameters.ArtifactSuffix }} targetPath: release-metadata - ${{ else }}: - task: DownloadPipelineArtifact@2 condition: eq(variables['DownloadMetadataArtifact'], 'true') displayName: Download release metadata artifact inputs: - artifactName: release-metadata + artifactName: release-metadata${{ parameters.ArtifactSuffix }} targetPath: release-metadata - pwsh: | diff --git a/eng/scripts/Set-ExtensionVersionVariable.ps1 b/eng/scripts/Set-ExtensionVersionVariable.ps1 index 34277f57182..547170b3b76 100644 --- a/eng/scripts/Set-ExtensionVersionVariable.ps1 +++ b/eng/scripts/Set-ExtensionVersionVariable.ps1 @@ -1,7 +1,85 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +<# +.SYNOPSIS +Sets the EXT_VERSION pipeline variable for an azd extension build. + +.DESCRIPTION +By default EXT_VERSION is the verbatim contents of the extension's version.txt. + +When nightly mode is enabled the base version is transformed into a semver-valid +nightly prerelease of the form: + + -nightly.. (base has no prerelease) + .nightly.. (base already has a prerelease) + +Worked examples (date 2026-06-18, build id 1234): + + version.txt nightly EXT_VERSION + ----------- ----------------------------------- + 0.1.0 0.1.0-nightly.20260618.1234 + 1.2.3 1.2.3-nightly.20260618.1234 + 0.1.0-beta 0.1.0-beta.nightly.20260618.1234 + +Why this ordering matters (semver precedence): + + * A stable release outranks any nightly built from the same base, so the eventual + stable supersedes the nightly: + 0.1.0-nightly.20260618.1234 < 0.1.0 + * Later nightlies outrank earlier ones because the date and then the build id are + compared numerically: + 0.1.0-nightly.20260618.1234 < 0.1.0-nightly.20260618.5678 (same day, higher build) + 0.1.0-nightly.20260618.9999 < 0.1.0-nightly.20260619.1000 (newer day wins) + +Nightly mode and its build id can be supplied explicitly via parameters or implicitly +via the standard Azure DevOps predefined variables, so the build templates that +already invoke this script need no changes — the nightly pipeline only has to set +the AZD_NIGHTLY variable. + +.PARAMETER ExtensionDirectory +Path to the extension directory containing version.txt. + +.PARAMETER Nightly +Forces nightly mode. When omitted the AZD_NIGHTLY environment variable +(true/1) is honored instead. + +.PARAMETER BuildId +Monotonic build identifier used as the nightly build number. Falls back to +BUILD_BUILDID. +#> param( - [string] $ExtensionDirectory + [string] $ExtensionDirectory, + [switch] $Nightly, + [string] $BuildId = $env:BUILD_BUILDID ) -$extVersion = Get-Content "$ExtensionDirectory/version.txt" -Write-Host "Extension Version: $extVersion" -Write-Host "##vso[task.setvariable variable=EXT_VERSION;]$extVersion" +$ErrorActionPreference = 'Stop' + +$baseVersion = (Get-Content "$ExtensionDirectory/version.txt" -Raw).Trim() + +$nightlyEnabled = $Nightly -or ($env:AZD_NIGHTLY -in @('true', '1', 'True')) + +if (-not $nightlyEnabled) { + Write-Host "Extension Version: $baseVersion" + Write-Host "##vso[task.setvariable variable=EXT_VERSION;]$baseVersion" + return +} + +if ([string]::IsNullOrWhiteSpace($BuildId)) { + throw "A build id is required for nightly builds. Pass -BuildId or set BUILD_BUILDID." +} + +$date = [DateTime]::UtcNow.ToString('yyyyMMdd') + +# Append the nightly identifiers to an existing prerelease (e.g. 0.1.0-preview -> +# 0.1.0-preview.nightly...) or introduce a new prerelease segment otherwise. +if ($baseVersion.Contains('-')) { + $nightlyVersion = "$baseVersion.nightly.$date.$BuildId" +} +else { + $nightlyVersion = "$baseVersion-nightly.$date.$BuildId" +} + +Write-Host "Extension Version (nightly): $nightlyVersion" +Write-Host "##vso[task.setvariable variable=EXT_VERSION;]$nightlyVersion"