diff --git a/cli/azd/docs/extensions/extension-framework.md b/cli/azd/docs/extensions/extension-framework.md index 70ffe5b178c..db4605ef49e 100644 --- a/cli/azd/docs/extensions/extension-framework.md +++ b/cli/azd/docs/extensions/extension-framework.md @@ -84,6 +84,26 @@ azd extension source add -n dev -t url -l "https://aka.ms/azd/extensions/registr Extensions installed from the dev registry are automatically promoted to the main registry when a newer version becomes available there. See the [Dev/Experimental Extension Registry](./extension-resolution-and-versioning.md#devexperimental-extension-registry) section for full details on stability expectations, submission guidelines, promotion behavior, and troubleshooting. +#### Nightly Registry + +> [!CAUTION] +> The nightly registry contains automated pre-release builds. They come with **no stability guarantees** and are **not covered by Azure support**. Expect breaking changes between nights. + +The nightly registry is an opt-in source containing daily pre-release builds of first-party extensions, published every night from the latest `main`. It lives on the `nightly-registry` branch of the [`azd` GitHub repo](https://github.com/Azure/azure-dev/blob/nightly-registry/cli/azd/extensions/registry.nightly.json) and is **not** configured by default. + +To opt-in for the nightly registry run the following command: + +```bash +# Add a new extension source named 'nightly' to your `azd` configuration. +azd extension source add -n nightly -t url \ + -l "https://raw.githubusercontent.com/Azure/azure-dev/nightly-registry/cli/azd/extensions/registry.nightly.json" + +# Install the nightly build of an extension. +azd extension install --source nightly +``` + +Nightly builds use a `-nightly..` version format, so the matching stable release always supersedes them — running `azd extension upgrade` moves you back to the `azd` source automatically once that stable version ships. See [Choosing Stable vs. Nightly](./extension-resolution-and-versioning.md#choosing-stable-vs-nightly) for the full workflow, including how to switch channels and opt back out. + #### `azd extension source list` Displays a list of installed extension sources. diff --git a/cli/azd/docs/extensions/extension-resolution-and-versioning.md b/cli/azd/docs/extensions/extension-resolution-and-versioning.md index 79120cec210..851d82c91c8 100644 --- a/cli/azd/docs/extensions/extension-resolution-and-versioning.md +++ b/cli/azd/docs/extensions/extension-resolution-and-versioning.md @@ -547,6 +547,91 @@ If the dev registry URL is unreachable (network issue, DNS failure), operations azd extension source remove dev ``` +## Nightly Extension Registry + +The nightly registry is a separate, opt-in extension source that contains automated daily +pre-release builds of first-party extensions. A scheduled pipeline builds each extension from +the latest `main` and publishes a new prerelease version to the registry every night, so the +nightly source always tracks the most recent in-development code. + +Like `dev`, the nightly source is **not** configured by default and is **not** a replacement for +the stable `azd` registry. It is a parallel pre-release channel for early adopters who want to +validate upcoming changes before they ship. + +| Property | Main Registry | Nightly Registry | +|----------|---------------|------------------| +| Source file | `cli/azd/extensions/registry.json` | `cli/azd/extensions/registry.nightly.json` | +| Location | `https://aka.ms/azd/extensions/registry` | `https://raw.githubusercontent.com/Azure/azure-dev/nightly-registry/cli/azd/extensions/registry.nightly.json` | +| Source name | `azd` (built-in default) | `nightly` (opt-in) | +| Cadence | On stable release | Every night from `main` | +| Stability | Stable releases | **Pre-release; may change or break nightly** | +| Support | Covered by Azure support | **Not covered** | + +### Nightly Version Format + +Nightly builds are stamped by [`eng/scripts/Set-ExtensionVersionVariable.ps1`](../../../../eng/scripts/Set-ExtensionVersionVariable.ps1) +into a semver-valid prerelease derived from the extension's `version.txt`: + +```text +-nightly.. e.g. 0.1.0-nightly.20260618.1234 +.nightly.. (when already has a prerelease label) +``` + +Because these are valid prereleases, semver precedence orders them the way you would expect: + +- A stable release always outranks any nightly built from the same base + (`0.1.0-nightly.20260618.1234` < `0.1.0`), so the stable version supersedes the nightly once it ships. +- A later nightly outranks an earlier one, comparing the date and then the build id numerically + (`...20260618.1234` < `...20260618.5678` and `...20260618.9999` < `...20260619.1000`). + +### Choosing Stable vs. Nightly + +The two channels coexist; you decide per extension which one to install from. + +**Stay on stable (default):** Do nothing. Without the `nightly` source configured, every +`azd extension install` and `azd extension upgrade` resolves against the official `azd` registry only. + +**Opt in to nightly:** Add the source, then install with `--source nightly`: + +```bash +# Add the nightly source once. +azd extension source add -n nightly -t url \ + -l "https://raw.githubusercontent.com/Azure/azure-dev/nightly-registry/cli/azd/extensions/registry.nightly.json" + +# Install the nightly build of an extension. +azd extension install --source nightly + +# Confirm both sources are configured. +azd extension source list +``` + +When an extension exists in both `azd` and `nightly` and you omit `--source`, `azd` prompts you to +choose (interactive) or errors with `"found in multiple sources"` (non-interactive). See +[Troubleshooting Multi-Registry Scenarios](#troubleshooting-multi-registry-scenarios). + +**Switch back to stable:** The nightly source uses the same one-way promotion logic as the dev +registry — see [Upgrade and Dev→Main Promotion](#upgrade-and-devmain-promotion). Because a stable +release always outranks the nightly built from the same base, running `azd extension upgrade` moves +you from `nightly` back to the `azd` source automatically once the corresponding stable version is +published. The extension's stored source flips from `nightly` to `azd`, and it stays on stable +from then on. To force the switch immediately (for example, to leave a nightly that is ahead of the +current stable), reinstall explicitly: + +```bash +# --force is required only when the stable version is lower than the installed nightly. +azd extension install --source azd --force +``` + +**Leave nightly entirely:** Remove the source to stop pulling nightly builds: + +```bash +azd extension source remove nightly +``` + +> [!CAUTION] +> Nightly builds come with **no stability guarantees** and are **not covered by Azure support**. +> Expect breaking changes between nights. Use a stable source for production workflows. + ## Related Documentation | Document | Description | diff --git a/cli/azd/pkg/extensions/update_checker_test.go b/cli/azd/pkg/extensions/update_checker_test.go index 3b66f1d8603..f85799bb529 100644 --- a/cli/azd/pkg/extensions/update_checker_test.go +++ b/cli/azd/pkg/extensions/update_checker_test.go @@ -249,3 +249,139 @@ func Test_UpdateChecker_InvalidVersions(t *testing.T) { require.NoError(t, err) require.False(t, result.HasUpdate) // Should gracefully handle invalid version } + +// Test_UpdateChecker_NightlyVersions verifies that the update checker correctly +// orders the nightly prerelease versions produced by Set-ExtensionVersionVariable.ps1. +// +// Nightly versions built from a base without an existing prerelease take the shape: +// +// -nightly.. e.g. 0.1.0-nightly.20260618.1234 +// +// Several transitions must hold for the upgrade experience to work: +// - nightly -> stable: once the stable ships it must supersede any nightly +// built from that base (semver: a release outranks its prereleases). +// - nightly -> next nightly: a later nightly (newer date, or same date with a +// higher build id) must supersede an earlier one, and an earlier nightly must +// never be offered as an update over a newer one. +// - nightly vs. milestone prereleases: for a clean base, semver orders +// alpha < beta < nightly < rc < stable, so a nightly supersedes alpha/beta of the +// same base but is itself superseded by rc and the final release. When the base +// already carries a prerelease label (e.g. 0.1.0-beta), the appended ".nightly" +// segment outranks every numbered beta of that base, so only the final release +// (or a higher base) supersedes it. +func Test_UpdateChecker_NightlyVersions(t *testing.T) { + tests := []struct { + name string + installed string + available []string + wantUpdate bool + }{ + { + name: "nightly to stable", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234", "0.1.0"}, + wantUpdate: true, + }, + { + name: "nightly to next nightly newer date", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234", "0.1.0-nightly.20260619.1234"}, + wantUpdate: true, + }, + { + name: "nightly to next nightly same date higher build", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234", "0.1.0-nightly.20260618.5678"}, + wantUpdate: true, + }, + { + name: "same nightly is not an update", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234"}, + wantUpdate: false, + }, + { + name: "older nightly is not offered over newer installed", + installed: "0.1.0-nightly.20260619.1234", + available: []string{"0.1.0-nightly.20260618.1234"}, + wantUpdate: false, + }, + { + // Clean base: nightly outranks beta of the same base (b < n). + name: "nightly supersedes beta of same base", + installed: "0.1.0-beta.1", + available: []string{"0.1.0-beta.1", "0.1.0-nightly.20260618.1234"}, + wantUpdate: true, + }, + { + // Clean base: an older beta must never be offered over an installed nightly. + name: "beta is not offered over installed nightly", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-beta.5", "0.1.0-nightly.20260618.1234"}, + wantUpdate: false, + }, + { + // Clean base: rc outranks nightly of the same base (n < r). + name: "rc supersedes nightly of same base", + installed: "0.1.0-nightly.20260618.1234", + available: []string{"0.1.0-nightly.20260618.1234", "0.1.0-rc.1"}, + wantUpdate: true, + }, + { + // Prerelease base gotcha: "0.1.0-beta.nightly..." outranks every numbered + // beta of that base because numeric identifiers lose to alphanumeric ones, + // so a later beta.N is NOT offered as an update. + name: "numbered beta does not supersede beta-base nightly", + installed: "0.1.0-beta.nightly.20260618.1234", + available: []string{"0.1.0-beta.2", "0.1.0-beta.nightly.20260618.1234"}, + wantUpdate: false, + }, + { + // Prerelease base: only the final stable release supersedes the beta-base nightly. + name: "stable supersedes beta-base nightly", + installed: "0.1.0-beta.nightly.20260618.1234", + available: []string{"0.1.0-beta.nightly.20260618.1234", "0.1.0"}, + wantUpdate: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + tempDir := t.TempDir() + t.Setenv("AZD_CONFIG_DIR", tempDir) + + cacheManager, err := NewRegistryCacheManager() + require.NoError(t, err) + + ctx := t.Context() + sourceName := "test-source" + + versions := make([]ExtensionVersion, 0, len(tt.available)) + for _, v := range tt.available { + versions = append(versions, ExtensionVersion{Version: v}) + } + + err = cacheManager.Set(ctx, sourceName, []*ExtensionMetadata{ + { + Id: "test.extension", + DisplayName: "Test Extension", + Versions: versions, + }, + }) + require.NoError(t, err) + + updateChecker := NewUpdateChecker(cacheManager) + + extension := &Extension{ + Id: "test.extension", + DisplayName: "Test Extension", + Version: tt.installed, + Source: sourceName, + } + + result, err := updateChecker.CheckForUpdate(ctx, extension) + require.NoError(t, err) + require.Equal(t, tt.wantUpdate, result.HasUpdate) + }) + } +} diff --git a/docs/architecture/extension-framework.md b/docs/architecture/extension-framework.md index 6f961f5430a..7c8d4943383 100644 --- a/docs/architecture/extension-framework.md +++ b/docs/architecture/extension-framework.md @@ -25,6 +25,7 @@ Extensions are discovered from registries — JSON manifests that list available - **Official registry:** `https://aka.ms/azd/extensions/registry` — Stable, signed, production-ready extensions vetted by the azd team. - **Dev registry:** `https://aka.ms/azd/extensions/registry/dev` — Experimental and pre-release extensions (unsigned builds, backed by `cli/azd/extensions/registry.dev.json`). Not configured by default; users opt in with `azd extension source add`. +- **Nightly registry:** `https://raw.githubusercontent.com/Azure/azure-dev/nightly-registry/cli/azd/extensions/registry.nightly.json` — Automated daily pre-release builds of first-party extensions, published from the latest `main` and backed by `registry.nightly.json` on the `nightly-registry` branch. Not configured by default; users opt in with `azd extension source add -n nightly`. See [Choosing Stable vs. Nightly](../../cli/azd/docs/extensions/extension-resolution-and-versioning.md#choosing-stable-vs-nightly). - **Local sources:** File-based manifests for development The dev registry serves as a staging area for extensions before they graduate to the main registry. Extensions installed from the dev registry are automatically promoted to the main registry when a newer stable version becomes available there. See the [Extension Resolution and Versioning](../../cli/azd/docs/extensions/extension-resolution-and-versioning.md#devexperimental-extension-registry) guide for detailed criteria, stability expectations, and submission guidelines. diff --git a/eng/pipelines/release-ext-foundry-nightly.yml b/eng/pipelines/release-ext-foundry-nightly.yml new file mode 100644 index 00000000000..29dc945231c --- /dev/null +++ b/eng/pipelines/release-ext-foundry-nightly.yml @@ -0,0 +1,217 @@ +# Single nightly publishing pipeline for the Microsoft Foundry bundle. +# +# One pipeline builds every Foundry extension from the latest commit on main each night, in parallel, +# and then updates the nightly extension registry ONCE for the whole bundle. This avoids the +# concurrent-writer problem you get when each extension publishes to the shared nightly-registry +# branch from its own pipeline. +# +# How the reuse works: +# * The per-extension build -> sign -> publish-release chain is the standard +# stages/release-azd-extension.yml template, instantiated once per extension via an ${{ each }} +# loop. StageSuffix (= _) makes each instance's stage ids and shared artifact names +# unique so they can coexist in one pipeline. +# * Skip.RegistryUpdate = true turns off the per-extension registry update job; the single +# Update_Nightly_Registry stage below owns the registry write instead. +# * AZD_NIGHTLY = true makes eng/scripts/Set-ExtensionVersionVariable.ps1 emit nightly semvers. +# +# The microsoft.foundry meta-package has no binaries, so it is NOT part of the build/sign fan-out. It +# is published last, in the join stage, after its dependencies are already in the nightly registry. + +trigger: none +pr: none + +schedules: + - cron: "0 7 * * *" + displayName: Daily Foundry nightly build (07:00 UTC) + branches: + include: + - main + # Build even when there were no commits since the last successful scheduled run. + always: true + +parameters: + # The Foundry dependency extensions that are built, signed, and released as nightly artifacts. + - name: Extensions + type: object + default: + - { id: azure.ai.agents, sanitized: azure-ai-agents, dir: cli/azd/extensions/azure.ai.agents, skipTests: false } + - { id: azure.ai.connections, sanitized: azure-ai-connections, dir: cli/azd/extensions/azure.ai.connections, skipTests: false } + - { id: azure.ai.inspector, sanitized: azure-ai-inspector, dir: cli/azd/extensions/azure.ai.inspector, skipTests: false } + - { id: azure.ai.projects, sanitized: azure-ai-projects, dir: cli/azd/extensions/azure.ai.projects, skipTests: false } + - { id: azure.ai.routines, sanitized: azure-ai-routines, dir: cli/azd/extensions/azure.ai.routines, skipTests: false } + - { id: azure.ai.skills, sanitized: azure-ai-skills, dir: cli/azd/extensions/azure.ai.skills, skipTests: false } + - { id: azure.ai.toolboxes, sanitized: azure-ai-toolboxes, dir: cli/azd/extensions/azure.ai.toolboxes, skipTests: false } + # The Foundry meta-package (extension pack). Published last, registry-only (no build/sign/release). + - name: PackId + type: string + default: microsoft.foundry + - name: PackDirectory + type: string + default: cli/azd/extensions/microsoft.foundry + +variables: + # Makes the shared version script emit -nightly.. for every build job. + - name: AZD_NIGHTLY + value: "true" + # The per-extension registry update job is skipped; the single join stage owns the registry write. + - name: Skip.RegistryUpdate + value: "true" + - name: NightlyRegistryBranch + value: nightly-registry + - name: ExtensionRegistryFile + value: registry.nightly.json + +extends: + template: /eng/pipelines/templates/stages/1es-redirect.yml + parameters: + stages: + # Fan out: build + sign + publish-release for every dependency extension, all in parallel. + - ${{ each ext in parameters.Extensions }}: + - template: /eng/pipelines/templates/stages/release-azd-extension.yml + parameters: + StageSuffix: _${{ ext.sanitized }} + IsNightly: true + SkipTests: ${{ ext.skipTests }} + AzdExtensionId: ${{ ext.id }} + SanitizedExtensionId: ${{ ext.sanitized }} + AzdExtensionDirectory: ${{ ext.dir }} + + # Join: update the nightly registry once for the whole bundle. Gated identically to the + # per-extension publish stages so its dependsOn never references stages that were not emitted. + - ${{ if and(eq(variables['System.TeamProject'], 'internal'), or(eq(variables['Build.Reason'], 'Manual'), eq(variables['Build.Reason'], 'Schedule'))) }}: + - stage: Update_Nightly_Registry + dependsOn: + - ${{ each ext in parameters.Extensions }}: + - PublishExtension_${{ ext.sanitized }} + + variables: + - template: /eng/pipelines/templates/variables/image.yml + - template: /eng/pipelines/templates/variables/globals.yml + + jobs: + - job: Update + condition: >- + and( + succeeded(), + ne('true', variables['Skip.RegistryUpdate.Bundle']) + ) + + pool: + name: $(LINUXPOOL) + image: $(LINUXVMIMAGE) + os: linux + + steps: + - checkout: self + + - bash: | + set -euo pipefail + curl -fsSL https://aka.ms/install-azd.sh | bash -s -- --verbose + azd version + displayName: Install azd + + - bash: | + set -euo pipefail + azd ext install microsoft.azd.extensions --source azd + displayName: Install microsoft.azd.extensions + + # Check out (or create) the nightly registry branch and seed an empty registry the + # first time. All per-extension publishes below mutate this single file in place. + - bash: | + set -euo pipefail + + branch="$(NightlyRegistryBranch)" + registry_path="cli/azd/extensions/$(ExtensionRegistryFile)" + + git config user.name "azure-sdk" + git config user.email "azuresdk@microsoft.com" + # checkout: self does not persist push credentials. + git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/$(Build.Repository.Name).git" + + git fetch origin "$branch" || true + if git show-ref --verify --quiet "refs/remotes/origin/${branch}"; then + git checkout -B "$branch" "origin/${branch}" + else + git checkout -B "$branch" + fi + + if [ ! -f "$registry_path" ]; then + echo '{ "extensions": [] }' > "$registry_path" + fi + displayName: Prepare nightly registry branch + env: + GH_TOKEN: $(azuresdk-github-pat) + + # Publish each dependency extension into the single registry file (no commit yet). + # The exact nightly version comes from each extension's release metadata artifact. + - ${{ each ext in parameters.Extensions }}: + - task: DownloadPipelineArtifact@2 + displayName: Download ${{ ext.sanitized }} metadata + inputs: + artifactName: release-metadata_${{ ext.sanitized }} + targetPath: $(Pipeline.Workspace)/meta-${{ ext.sanitized }} + + - bash: | + set -euo pipefail + version=$(jq -r '.extVersion' "$(Pipeline.Workspace)/meta-${{ ext.sanitized }}/metadata.json") + echo "Publishing ${{ ext.id }} ${version} to nightly registry" + ( cd "${{ ext.dir }}" && azd x publish \ + --registry "../$(ExtensionRegistryFile)" \ + --repo "$(Build.Repository.Name)" \ + --version "$version" ) + displayName: Publish ${{ ext.sanitized }} to nightly registry + env: + GH_TOKEN: $(azuresdk-github-pat) + + # Publish the Foundry pack last, after its dependencies exist in the registry. The pack + # is registry-only (no GitHub release) so --repo is omitted. Its nightly version is + # computed from its own version.txt by the shared version script (AZD_NIGHTLY=true). + - task: PowerShell@2 + displayName: Compute Foundry pack nightly version + inputs: + pwsh: true + targetType: filePath + filePath: eng/scripts/Set-ExtensionVersionVariable.ps1 + arguments: >- + -ExtensionDirectory ${{ parameters.PackDirectory }} + + - bash: | + set -euo pipefail + echo "Publishing ${{ parameters.PackId }} ${EXT_VERSION} to nightly registry" + ( cd "${{ parameters.PackDirectory }}" && azd x publish \ + --registry "../$(ExtensionRegistryFile)" \ + --version "$EXT_VERSION" ) + displayName: Publish Foundry pack to nightly registry + env: + GH_TOKEN: $(azuresdk-github-pat) + + # Commit the whole bundle's registry changes once and push (single fetch+rebase retry). + - bash: | + set -euo pipefail + + branch="$(NightlyRegistryBranch)" + registry_path="cli/azd/extensions/$(ExtensionRegistryFile)" + + git add "$registry_path" + if git diff --cached --quiet; then + echo "No registry changes to commit." + exit 0 + fi + + for attempt in 1 2 3; do + git commit -m "[foundry-nightly] Nightly registry update ($(Build.BuildNumber))" || true + if git push origin "$branch"; then + echo "Nightly registry updated on ${branch}." + exit 0 + fi + if [ "$attempt" -eq 3 ]; then + echo "Failed to update nightly registry after 3 attempts." >&2 + exit 1 + fi + echo "Push rejected; rebasing onto latest and retrying (${attempt})..." + git fetch origin "$branch" + git reset --soft "origin/${branch}" + done + displayName: Commit and push nightly registry + env: + GH_TOKEN: $(azuresdk-github-pat) diff --git a/eng/pipelines/templates/stages/build-and-test-azd-extension.yml b/eng/pipelines/templates/stages/build-and-test-azd-extension.yml index 703f6e822e0..0420cef5b1b 100644 --- a/eng/pipelines/templates/stages/build-and-test-azd-extension.yml +++ b/eng/pipelines/templates/stages/build-and-test-azd-extension.yml @@ -8,6 +8,15 @@ parameters: - name: SkipTests type: boolean default: false + # StageSuffix is appended to this template's stage id AND to the shared pipeline artifact names it + # produces (release-metadata, changelog) so the template can be instantiated multiple times in a + # single pipeline via an ${{ each }} loop (e.g. the nightly bundle pipeline that builds many + # extensions in parallel). Azure DevOps requires stage ids and artifact names to be unique within a + # pipeline; without a suffix, looping would collide on "BuildAndTest"/"release-metadata"/"changelog" + # and fail. Defaults to '' so existing single-extension pipelines are byte-identical and unaffected. + - name: StageSuffix + type: string + default: '' - name: CrossBuildMatrix type: object default: @@ -66,7 +75,7 @@ parameters: GOOS: windows GOARCH: arm64 stages: - - stage: BuildAndTest + - stage: BuildAndTest${{ parameters.StageSuffix }} variables: - template: /eng/pipelines/templates/variables/globals.yml - template: /eng/pipelines/templates/variables/image.yml @@ -162,10 +171,10 @@ stages: - output: pipelineArtifact path: release-metadata condition: succeeded() - artifact: release-metadata + artifact: release-metadata${{ parameters.StageSuffix }} displayName: Upload release metadata - output: pipelineArtifact path: changelog - artifact: changelog + artifact: changelog${{ parameters.StageSuffix }} displayName: Upload changelog diff --git a/eng/pipelines/templates/stages/publish-extension.yml b/eng/pipelines/templates/stages/publish-extension.yml index 67d73873e5a..fcb46489a31 100644 --- a/eng/pipelines/templates/stages/publish-extension.yml +++ b/eng/pipelines/templates/stages/publish-extension.yml @@ -8,27 +8,64 @@ parameters: - name: PublishToDevRegistry type: boolean default: false + # When true the build publishes to the nightly registry (registry.nightly.json) which lives on a + # dedicated, bot-owned branch and is updated by auto-commit instead of a pull request. + - name: IsNightly + type: boolean + default: false + # StageSuffix is appended to this stage's id (and its dependsOn) AND to the shared pipeline artifact + # names it consumes (release, changelog, release-metadata) so the template can be instantiated + # multiple times in a single pipeline via an ${{ each }} loop (e.g. the nightly bundle pipeline that + # builds many extensions in parallel). Azure DevOps requires stage ids and artifact names to be + # unique within a pipeline; without a suffix, looping would collide and fail to compile. Defaults to + # '' so existing single-extension pipelines are byte-identical and unaffected. + - name: StageSuffix + type: string + default: '' stages: - - stage: PublishExtension - dependsOn: Sign - condition: >- - and( - succeeded(), - ne(variables['Skip.Release'], 'true'), - or( - eq('Manual', variables['BuildReasonOverride']), - and( - eq('', variables['BuildReasonOverride']), - eq('Manual', variables['Build.Reason']) + - stage: PublishExtension${{ parameters.StageSuffix }} + dependsOn: Sign${{ parameters.StageSuffix }} + ${{ if eq(parameters.IsNightly, true) }}: + # Nightly publishes run on a schedule (or when queued manually). + condition: >- + and( + succeeded(), + ne(variables['Skip.Release'], 'true'), + or( + eq('Manual', variables['BuildReasonOverride']), + and( + eq('', variables['BuildReasonOverride']), + or( + eq('Manual', variables['Build.Reason']), + eq('Schedule', variables['Build.Reason']) + ) + ) + ) + ) + ${{ else }}: + condition: >- + and( + succeeded(), + ne(variables['Skip.Release'], 'true'), + or( + eq('Manual', variables['BuildReasonOverride']), + and( + eq('', variables['BuildReasonOverride']), + eq('Manual', variables['Build.Reason']) + ) ) ) - ) variables: - template: /eng/pipelines/templates/variables/image.yml - template: /eng/pipelines/templates/variables/globals.yml - - ${{ if eq(parameters.PublishToDevRegistry, true) }}: + - ${{ if eq(parameters.IsNightly, true) }}: + - name: ExtensionRegistryFile + value: registry.nightly.json + - name: NightlyRegistryBranch + value: nightly-registry + - ${{ elseif eq(parameters.PublishToDevRegistry, true) }}: - name: ExtensionRegistryFile value: registry.dev.json - name: ExtensionRegistryPullRequestPrefix @@ -62,11 +99,11 @@ stages: isProduction: true inputs: - input: pipelineArtifact - artifactName: release + artifactName: release${{ parameters.StageSuffix }} targetPath: release - input: pipelineArtifact - artifactName: changelog + artifactName: changelog${{ parameters.StageSuffix }} targetPath: changelog strategy: @@ -76,6 +113,7 @@ stages: - template: /eng/pipelines/templates/steps/extension-set-metadata-variables.yml parameters: Use1ESArtifactTask: true + ArtifactSuffix: ${{ parameters.StageSuffix }} - pwsh: | # Initial upload locations @@ -111,6 +149,8 @@ stages: - checkout: self - template: /eng/pipelines/templates/steps/extension-set-metadata-variables.yml + parameters: + ArtifactSuffix: ${{ parameters.StageSuffix }} - template: /eng/pipelines/templates/steps/setup-go.yml @@ -125,33 +165,100 @@ stages: azd ext install microsoft.azd.extensions --source azd displayName: Install microsoft.azd.extensions - - bash: | - set -euo pipefail - cd "${{ parameters.AzdExtensionDirectory }}" - azd x publish \ - --registry "../$(ExtensionRegistryFile)" \ - --repo "$(Build.Repository.Name)" \ - --version "$(EXT_VERSION)" - displayName: Update $(ExtensionRegistryFile) (azd x publish) - env: - GH_TOKEN: $(azuresdk-github-pat) - - - ${{ if ne(parameters.PublishToDevRegistry, true) }}: + - ${{ if eq(parameters.IsNightly, true) }}: + # Nightly registry lives on a dedicated, bot-owned branch and is updated by auto-commit + # (no pull request). A publish -> commit -> push retry loop keeps concurrent extension + # nightly runs from clobbering each other's entries on the shared branch. - bash: | set -euo pipefail - cd cli/azd - go build . - go test ./cmd -run 'TestFigSpec|TestUsage' - displayName: Refresh Fig/Usage snapshots + + branch="$(NightlyRegistryBranch)" + registry_rel="$(ExtensionRegistryFile)" + registry_path="cli/azd/extensions/${registry_rel}" + ext_dir="${{ parameters.AzdExtensionDirectory }}" + + git config user.name "azure-sdk" + git config user.email "azuresdk@microsoft.com" + + # checkout: self does not persist push credentials, so authenticate the remote with + # the bot PAT for the fetch/push of the nightly registry branch. + git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/$(Build.Repository.Name).git" + + publish_registry() { + # azd x publish loads an existing registry; seed an empty one on first nightly. + if [ ! -f "$registry_path" ]; then + echo '{ "extensions": [] }' > "$registry_path" + fi + ( cd "$ext_dir" && azd x publish \ + --registry "../${registry_rel}" \ + --repo "$(Build.Repository.Name)" \ + --version "$(EXT_VERSION)" ) + } + + # Check out (or create) the nightly registry branch. + git fetch origin "$branch" || true + if git show-ref --verify --quiet "refs/remotes/origin/${branch}"; then + git checkout -B "$branch" "origin/${branch}" + else + git checkout -B "$branch" + fi + + for attempt in 1 2 3 4 5; do + publish_registry + git add "$registry_path" + if git diff --cached --quiet; then + echo "No registry changes to commit." + break + fi + + git commit -m "[${{ parameters.AzdExtensionId }}] Nightly registry update for $(EXT_VERSION)" + if git push origin "$branch"; then + echo "Nightly registry updated on ${branch}." + break + fi + + if [ "$attempt" -eq 5 ]; then + echo "Failed to update nightly registry after 5 attempts." >&2 + exit 1 + fi + + echo "Push rejected (branch advanced). Rebasing onto latest and retrying (${attempt})..." + git reset --hard HEAD~1 + git fetch origin "$branch" + git reset --hard "origin/${branch}" + done + displayName: Update $(ExtensionRegistryFile) (azd x publish + auto-commit) env: - UPDATE_SNAPSHOTS: "true" + GH_TOKEN: $(azuresdk-github-pat) - - template: /eng/common/pipelines/templates/steps/create-pull-request.yml - parameters: - PRBranchName: $(ExtensionRegistryPullRequestPrefix)/${{ parameters.SanitizedExtensionId }}/$(EXT_VERSION)-$(Build.BuildId) - CommitMsg: "[${{ parameters.AzdExtensionId }}] $(ExtensionRegistryPullRequestTitle) for $(EXT_VERSION)" - PRTitle: "[${{ parameters.AzdExtensionId }}] $(ExtensionRegistryPullRequestTitle) for $(EXT_VERSION)" - PRBody: >- - $(ExtensionRegistryPullRequestTitle) for the - [azd-ext-${{ parameters.SanitizedExtensionId }}_$(EXT_VERSION)](https://github.com/$(Build.Repository.Name)/releases/tag/azd-ext-${{ parameters.SanitizedExtensionId }}_$(EXT_VERSION)) - release. + - ${{ else }}: + - bash: | + set -euo pipefail + cd "${{ parameters.AzdExtensionDirectory }}" + azd x publish \ + --registry "../$(ExtensionRegistryFile)" \ + --repo "$(Build.Repository.Name)" \ + --version "$(EXT_VERSION)" + displayName: Update $(ExtensionRegistryFile) (azd x publish) + env: + GH_TOKEN: $(azuresdk-github-pat) + + - ${{ if ne(parameters.PublishToDevRegistry, true) }}: + - bash: | + set -euo pipefail + cd cli/azd + go build . + go test ./cmd -run 'TestFigSpec|TestUsage' + displayName: Refresh Fig/Usage snapshots + env: + UPDATE_SNAPSHOTS: "true" + + - template: /eng/common/pipelines/templates/steps/create-pull-request.yml + parameters: + PRBranchName: $(ExtensionRegistryPullRequestPrefix)/${{ parameters.SanitizedExtensionId }}/$(EXT_VERSION)-$(Build.BuildId) + CommitMsg: "[${{ parameters.AzdExtensionId }}] $(ExtensionRegistryPullRequestTitle) for $(EXT_VERSION)" + PRTitle: "[${{ parameters.AzdExtensionId }}] $(ExtensionRegistryPullRequestTitle) for $(EXT_VERSION)" + PRBody: >- + $(ExtensionRegistryPullRequestTitle) for the + [azd-ext-${{ parameters.SanitizedExtensionId }}_$(EXT_VERSION)](https://github.com/$(Build.Repository.Name)/releases/tag/azd-ext-${{ parameters.SanitizedExtensionId }}_$(EXT_VERSION)) + release. diff --git a/eng/pipelines/templates/stages/release-azd-extension.yml b/eng/pipelines/templates/stages/release-azd-extension.yml index a2f656bee85..bc06f406bac 100644 --- a/eng/pipelines/templates/stages/release-azd-extension.yml +++ b/eng/pipelines/templates/stages/release-azd-extension.yml @@ -11,10 +11,21 @@ parameters: - name: PublishToDevRegistry type: boolean default: false + # When true the release publishes to the nightly registry and is allowed to run on a schedule. + - name: IsNightly + type: boolean + default: false + # StageSuffix is forwarded to every child stage template so this whole build/sign/publish chain can + # be instantiated once per extension via an ${{ each }} loop in a single pipeline (e.g. the nightly + # bundle). Defaults to '' so existing single-extension pipelines are unaffected. + - name: StageSuffix + type: string + default: '' stages: - template: /eng/pipelines/templates/stages/build-and-test-azd-extension.yml parameters: + StageSuffix: ${{ parameters.StageSuffix }} AzdExtensionId: ${{ parameters.AzdExtensionId }} AzdExtensionDirectory: ${{ parameters.AzdExtensionDirectory }} SkipTests: ${{ parameters.SkipTests }} @@ -78,11 +89,12 @@ stages: SetExecutableBit: true AZURE_DEV_CI_OS: mac-arm64 - # Only sign and release on manual builds from internal - - ${{ if and(eq(variables['System.TeamProject'], 'internal'), eq(variables['Build.Reason'], 'Manual')) }}: + # Only sign and release on manual builds from internal, or scheduled nightly builds from internal. + - ${{ if and(eq(variables['System.TeamProject'], 'internal'), or(eq(variables['Build.Reason'], 'Manual'), and(eq(parameters.IsNightly, true), eq(variables['Build.Reason'], 'Schedule')))) }}: - template: /eng/pipelines/templates/stages/sign-extension.yml parameters: SanitizedExtensionId: ${{ parameters.SanitizedExtensionId }} + StageSuffix: ${{ parameters.StageSuffix }} - template: /eng/pipelines/templates/stages/publish-extension.yml parameters: @@ -90,3 +102,5 @@ stages: AzdExtensionDirectory: ${{ parameters.AzdExtensionDirectory }} SanitizedExtensionId: ${{ parameters.SanitizedExtensionId }} PublishToDevRegistry: ${{ parameters.PublishToDevRegistry }} + IsNightly: ${{ parameters.IsNightly }} + StageSuffix: ${{ parameters.StageSuffix }} diff --git a/eng/pipelines/templates/stages/sign-extension.yml b/eng/pipelines/templates/stages/sign-extension.yml index c5715ba1ff0..25b0ae94664 100644 --- a/eng/pipelines/templates/stages/sign-extension.yml +++ b/eng/pipelines/templates/stages/sign-extension.yml @@ -1,10 +1,19 @@ parameters: - name: SanitizedExtensionId type: string + # StageSuffix is appended to this stage's id (and its dependsOn) AND to the shared pipeline artifact + # names it produces/consumes (signed-mac, signed-win, release, release-metadata) so the template can + # be instantiated multiple times in a single pipeline via an ${{ each }} loop (e.g. the nightly + # bundle pipeline that builds many extensions in parallel). Azure DevOps requires stage ids and + # artifact names to be unique within a pipeline; without a suffix, looping would collide and fail to + # compile. Defaults to '' so existing single-extension pipelines are byte-identical and unaffected. + - name: StageSuffix + type: string + default: '' stages: - - stage: Sign - dependsOn: BuildAndTest + - stage: Sign${{ parameters.StageSuffix }} + dependsOn: BuildAndTest${{ parameters.StageSuffix }} variables: - template: /eng/pipelines/templates/variables/globals.yml @@ -73,14 +82,14 @@ stages: condition: succeeded() displayName: Publish Signed Artifacts inputs: - artifactName: signed-mac + artifactName: signed-mac${{ parameters.StageSuffix }} path: signed-mac/ - task: 1ES.PublishPipelineArtifact@1 condition: failed() displayName: Publish failed Signed Artifacts inputs: - artifactName: signed-mac-FailedAttempt$(System.JobAttempt) + artifactName: signed-mac${{ parameters.StageSuffix }}-FailedAttempt$(System.JobAttempt) path: signed-mac/ - job: SignWindows @@ -140,14 +149,14 @@ stages: condition: succeeded() displayName: Publish Signed Artifacts inputs: - artifactName: signed-win + artifactName: signed-win${{ parameters.StageSuffix }} path: signed-win/ - task: 1ES.PublishPipelineArtifact@1 condition: failed() displayName: Publish failed Signed Artifacts inputs: - artifactName: signed-win-FailedAttempt$(System.JobAttempt) + artifactName: signed-win${{ parameters.StageSuffix }}-FailedAttempt$(System.JobAttempt) path: signed-win/ - job: CreateRelease @@ -164,24 +173,24 @@ stages: outputs: - output: pipelineArtifact path: release - artifact: release + artifact: release${{ parameters.StageSuffix }} condition: succeeded() displayName: Upload azd release artifact steps: - task: DownloadPipelineArtifact@2 inputs: - artifactName: release-metadata + artifactName: release-metadata${{ parameters.StageSuffix }} targetPath: release-metadata - task: DownloadPipelineArtifact@2 inputs: - artifactName: signed-win + artifactName: signed-win${{ parameters.StageSuffix }} targetPath: signed/win - task: DownloadPipelineArtifact@2 inputs: - artifactName: signed-mac + artifactName: signed-mac${{ parameters.StageSuffix }} targetPath: signed/mac # Linux binaries are not signed today so download from build outputs diff --git a/eng/pipelines/templates/steps/extension-set-metadata-variables.yml b/eng/pipelines/templates/steps/extension-set-metadata-variables.yml index 21f942f6ab2..e9fa7eef14e 100644 --- a/eng/pipelines/templates/steps/extension-set-metadata-variables.yml +++ b/eng/pipelines/templates/steps/extension-set-metadata-variables.yml @@ -2,6 +2,11 @@ parameters: - name: Use1ESArtifactTask type: boolean default: false + # Suffix appended to the release-metadata artifact name so this step can consume the per-extension + # artifact produced by a looped build stage in a single multi-extension pipeline. Defaults to ''. + - name: ArtifactSuffix + type: string + default: '' steps: - pwsh: | @@ -19,14 +24,14 @@ steps: condition: eq(variables['DownloadMetadataArtifact'], 'true') displayName: Download release metadata artifact inputs: - artifactName: release-metadata + artifactName: release-metadata${{ parameters.ArtifactSuffix }} targetPath: release-metadata - ${{ else }}: - task: DownloadPipelineArtifact@2 condition: eq(variables['DownloadMetadataArtifact'], 'true') displayName: Download release metadata artifact inputs: - artifactName: release-metadata + artifactName: release-metadata${{ parameters.ArtifactSuffix }} targetPath: release-metadata - pwsh: | diff --git a/eng/scripts/Set-ExtensionVersionVariable.ps1 b/eng/scripts/Set-ExtensionVersionVariable.ps1 index 34277f57182..547170b3b76 100644 --- a/eng/scripts/Set-ExtensionVersionVariable.ps1 +++ b/eng/scripts/Set-ExtensionVersionVariable.ps1 @@ -1,7 +1,85 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +<# +.SYNOPSIS +Sets the EXT_VERSION pipeline variable for an azd extension build. + +.DESCRIPTION +By default EXT_VERSION is the verbatim contents of the extension's version.txt. + +When nightly mode is enabled the base version is transformed into a semver-valid +nightly prerelease of the form: + + -nightly.. (base has no prerelease) + .nightly.. (base already has a prerelease) + +Worked examples (date 2026-06-18, build id 1234): + + version.txt nightly EXT_VERSION + ----------- ----------------------------------- + 0.1.0 0.1.0-nightly.20260618.1234 + 1.2.3 1.2.3-nightly.20260618.1234 + 0.1.0-beta 0.1.0-beta.nightly.20260618.1234 + +Why this ordering matters (semver precedence): + + * A stable release outranks any nightly built from the same base, so the eventual + stable supersedes the nightly: + 0.1.0-nightly.20260618.1234 < 0.1.0 + * Later nightlies outrank earlier ones because the date and then the build id are + compared numerically: + 0.1.0-nightly.20260618.1234 < 0.1.0-nightly.20260618.5678 (same day, higher build) + 0.1.0-nightly.20260618.9999 < 0.1.0-nightly.20260619.1000 (newer day wins) + +Nightly mode and its build id can be supplied explicitly via parameters or implicitly +via the standard Azure DevOps predefined variables, so the build templates that +already invoke this script need no changes — the nightly pipeline only has to set +the AZD_NIGHTLY variable. + +.PARAMETER ExtensionDirectory +Path to the extension directory containing version.txt. + +.PARAMETER Nightly +Forces nightly mode. When omitted the AZD_NIGHTLY environment variable +(true/1) is honored instead. + +.PARAMETER BuildId +Monotonic build identifier used as the nightly build number. Falls back to +BUILD_BUILDID. +#> param( - [string] $ExtensionDirectory + [string] $ExtensionDirectory, + [switch] $Nightly, + [string] $BuildId = $env:BUILD_BUILDID ) -$extVersion = Get-Content "$ExtensionDirectory/version.txt" -Write-Host "Extension Version: $extVersion" -Write-Host "##vso[task.setvariable variable=EXT_VERSION;]$extVersion" +$ErrorActionPreference = 'Stop' + +$baseVersion = (Get-Content "$ExtensionDirectory/version.txt" -Raw).Trim() + +$nightlyEnabled = $Nightly -or ($env:AZD_NIGHTLY -in @('true', '1', 'True')) + +if (-not $nightlyEnabled) { + Write-Host "Extension Version: $baseVersion" + Write-Host "##vso[task.setvariable variable=EXT_VERSION;]$baseVersion" + return +} + +if ([string]::IsNullOrWhiteSpace($BuildId)) { + throw "A build id is required for nightly builds. Pass -BuildId or set BUILD_BUILDID." +} + +$date = [DateTime]::UtcNow.ToString('yyyyMMdd') + +# Append the nightly identifiers to an existing prerelease (e.g. 0.1.0-preview -> +# 0.1.0-preview.nightly...) or introduce a new prerelease segment otherwise. +if ($baseVersion.Contains('-')) { + $nightlyVersion = "$baseVersion.nightly.$date.$BuildId" +} +else { + $nightlyVersion = "$baseVersion-nightly.$date.$BuildId" +} + +Write-Host "Extension Version (nightly): $nightlyVersion" +Write-Host "##vso[task.setvariable variable=EXT_VERSION;]$nightlyVersion"