ci: rerun failed integration tests once to handle flaky nightly failures#9118
Merged
Merged
Conversation
The nightly `azure-dev - cli` pipeline has been red on nearly every run because integration tests provision live Azure resources and fail intermittently for environmental reasons that rotate night to night (credential token expiry during long provisioning, transient Azure service capacity such as App Service "No available instances to satisfy this request", and teardown races). With no retry, a single such blip fails the whole nightly. Enable gotestsum's rerun of failed integration tests (once), so flaky failures are retried while genuine regressions still fail: - --rerun-fails=1: re-run each failed test one time. - --rerun-fails-max-failures=5: if the initial run has more than 5 failures, don't rerun anything (a mass failure is a real regression, not flake). - --rerun-fails-report: record which tests were rerun so flaky tests stay visible for triage. - --packages "./...": required by --rerun-fails (the package list can no longer be passed positionally after --). Only the integration test invocation is changed; unit tests are fast and deterministic and keep running without retry. Fixes #8386 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
vhvb1989
requested review from
JeffreyCA,
RickWinter,
hemarina and
tg-msft
as code owners
July 13, 2026 21:36
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the azd CI test runner to make nightly integration test execution more resilient to transient live-Azure flakes by rerunning failed integration tests once via gotestsum.
Changes:
- Adds a new
IntegrationTestRerunReportparameter for capturing rerun details. - Switches the integration test gotestsum invocation to use
--rerun-fails=1with a max-failures cap and a rerun report output file.
jongio
approved these changes
Jul 13, 2026
Contributor
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
JeffreyCA
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #8386
The nightly scheduled
azure-dev - clipipeline (definition 4643) has been red on nearly every run. As documented in #8386, no single test is the persistent blocker — the failures rotate night to night, which is the signature of flaky test infrastructure, not a code regression.This PR implements the retry proposed in #8386: re-run failed integration tests once, so an environmental blip no longer reds the entire nightly, while genuine regressions still fail.
What's failing and why retry helps
Investigating the recent nightlies, the dominant real failures are live-Azure, environmental issues that a single rerun typically clears:
Test_CLI_Deploy_SlotDeploymentandTest_CLI_Deploy_StoppedWebAppfail creating the App Service plan withConflict: No available instances to satisfy this request. App Service is attempting to increase capacity.This is Azure-side scale-unit capacity in the test region (East US 2), not a subscription quota — live App Service core quota is ~4–5 of 36 used, nowhere near the limit. It appears in episodic clusters (e.g. ~5 consecutive nights) that a rerun rides out.Test_DeploymentStacks,context deadline exceededduring long provisioning.Test_CLI_Aspire_DetectGen,Test_CLI_Up_Down_FuncApp,Test_CLI_*_ContainerApp_RemoteBuild,Test_StorageBlobClient/Crud, etc.Change
In
cli/azd/ci-test.ps1, the integration test invocation now uses gotestsum's rerun:--rerun-fails=1— re-run each failed test once (only the failed tests, not the full suite), so a flaky test that passes on retry is counted as passed.--rerun-fails-max-failures=5— if the initial run has more than 5 failures, don't rerun anything. A mass failure is a genuine regression and should not be masked.--rerun-fails-report— write the list of rerun tests totest-rerun-int.txtso flaky tests stay visible for triage / trend tracking.--packages "./..."— required by--rerun-fails; gotestsum errors if the package list is passed positionally after--.Unit tests are intentionally left without retry — they are fast and deterministic; if they fail it is almost certainly real.
Time impact
Rerunning only the failed tests adds a couple of minutes at most (e.g. a single ~120s functional test), not the ~90-minute suite. Acceptable for a nightly.
Testing
pwshparses the script without errors.gotestsumthat--rerun-failsrequires--packages(positional package list after--errors out), so the flag placement in this change is correct.Not addressed here
The App Service capacity failures are environmental (Azure East US 2 scale-unit capacity). Retry mitigates their impact on the nightly signal, but spreading the webapp deploy tests across regions and/or an Azure support follow-up for East US 2 dedicated-Linux capacity are worth tracking separately.