The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1128402	dotnet/aspire	Build	Log
2776816	dotnet-dotnet	Build (azurelinuxArm64MuslCrossContainer)	Log
2776970	dotnet-dotnet	🛡 Guardian: ESLint (TypeScript)	Log
2776765	dotnet-dotnet	Build (centOSStreamContainer)	Log
2776918	dotnet-roslyn	Download secrets: EngKeyVault	Log
1128233	dotnet/roslyn	Run Unit Tests	Log
2776791	dotnet-dotnet	Build	Log
1128154	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118803
1128112	dotnet/runtime	Send tests to Helix (Windows)	Log
1128115	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118946
1128125	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#63351
1128121	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#63351
1128126	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50264
1128119	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#63351
1128108	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50010
1128109	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50052
1127920	dotnet/runtime	Send to Helix	Log	dotnet/runtime#106880
1128113	dotnet/roslyn	Test	Log	dotnet/roslyn#79997
1128008	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118101
1128067	dotnet/aspnetcore	Run build.cmd helix target	Log
2776756	dotnet-performance	Send job to Helix (Unix)	Log
1128044	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#118371
1128029	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#118944
1127997	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#118101
1128003	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#118101
1127942	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#118941
2776632	dotnet-dotnet	Validate Hosted Pool Information (1ES PT)	Log
2776626	dotnet-dotnet	Secure Supply Chain Analysis (auto-injected by policy)	Log
2776591	dotnet-roslyn	Run Unit Tests	Log
2776568	dotnet-dotnet	Build (centOSStreamContainer)	Log
2776535	dotnet-runtime	Send job to Helix (Unix)	Log
2776533	dotnet-runtime	Send job to Helix (Unix)	Log
2776539	dotnet-aspnetcore	Run build.cmd	Log
1127836	dotnet/roslyn	Run Unit Tests	Log
2776486	dotnet-dotnet	Build	Log
1127817	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50389
2774117	dotnet-runtime	Send job to Helix (Unix)	Log
2774698	dotnet-runtime	Send job to Helix (Unix)	Log
2775531	dotnet-runtime	Send job to Helix (Unix)	Log
1127744	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#118935
1127721	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#118101
2776473	dotnet-docker-tools	Download secrets: DotNetContainersKeyVault	Log
2776437	dotnet-runtime	Send to Helix	Log
2776457	dotnet-roslyn	Run Unit Tests	Log
1127657	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#50389
2776416	dotnet-dotnet	Build (azurelinuxX64MuslCrossContainer)	Log
1127483	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118919
1123213	dotnet/runtime	Send to Helix	Log	dotnet/runtime#118757
2776360	dotnet-dotnet	🛡 Guardian: PSScriptAnalyzer	Log
2776343	dotnet-dotnet	Build (centOSStreamContainer)	Log
2776352	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
2776349	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
2776340	dotnet-sdk	Run AoT Tests	Log	#52510
2776312	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
2776307	dotnet-dotnet	🛡 Guardian: CredScan (Source)	Log
2776261	dotnet-sdk	Run AoT Tests in Helix	Log	#52509
2776259	dotnet-sdk	Run AoT Tests in Helix	Log	#52508
2776260	dotnet-sdk	Run Tests in Helix	Log	#52507
2776156	dotnet-sdk	Run TestBuild Tests	Log	#52510
2776157	dotnet-sdk	Run TestBuild Tests	Log	#52511
1127374	dotnet/sdk	Checkout dotnet/sdk@refs/pull/50243/merge to s	Log	dotnet/sdk#50243
2776155	dotnet-sdk	Run Tests in Helix	Log	#52508
2776147	dotnet-aspnetcore	Initialize containers	Log	#52506
2776107	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
2776085	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
2776077	dotnet-dotnet	🛡 Guardian: CredScan (Source)	Log
2775983	dotnet-runtime	Run performance job script	Log
2775992	dotnet-dotnet-monitor	Build	Log
2775984	dotnet-runtime	Send job to Helix (Unix)	Log
2775965	dotnet-sdk	Run AoT Tests	Log	#52495
2775959	dotnet-aspnetcore	Run build.cmd helix target	Log	#52491
2775963	dotnet-sdk	Run AoT Tests	Log	#52494
2775957	dotnet-sdk	Run AoT Tests in Helix	Log	#52490
2775958	dotnet-sdk	Run Tests in Helix	Log	#52488
2775956	dotnet-sdk	Run Tests in Helix	Log	#52489
2775898	dotnet-runtime	Send to Helix	Log
2775870	dotnet-roslyn	Run Unit Tests	Log
2775861	dotnet-runtime	Send job to Helix (Unix)	Log
2775848	dotnet-runtime	Build product	Log
2775801	dotnet-vscode-dotnet-runtime	🛡 Guardian: Publish SDL Logs to Artifacts	Log
2775778	dotnet-roslyn	Run Unit Tests	Log
2775681	dotnet-roslyn	Run Unit Tests	Log
2775639	dotnet-dotnet	Build (centOSStreamContainer)	Log
2775646	dotnet-docker-tools	Validate Hosted Pool Information (1ES PT)	Log
2775640	dotnet-dotnet	Download secrets: EngKeyVault	Log
2775643	dotnet-dotnet	Checkout dotnet-dotnet@dev/mthalman/issue1113.2xx to s	Log
2775636	dotnet-dotnet	Initialize containers	Log
2775635	dotnet-dotnet	Checkout dotnet-dotnet@dev/mthalman/issue1113.2xx to s	Log
2775602	dotnet-runtime	Send to Helix	Log	#52242
2775629	dotnet-dotnet	Checkout dotnet-dotnet@dev/mthalman/issue1113 to s	Log
2775628	dotnet-dotnet	Checkout dotnet-dotnet@dev/mthalman/issue1113 to s	Log
2775626	dotnet-dotnet	🔒 Start Network Isolation	Log
2775627	dotnet-dotnet	Checkout dotnet-dotnet@dev/mthalman/issue1113.2xx to s	Log
2775623	dotnet-dotnet	Initialize containers	Log
2775622	dotnet-dotnet	Agent Pool Validation	Log
1126702	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79943
2775608	dotnet-dotnet	SDL Binary Analysis Checks (1ES PT)	Log
2775580	dotnet-dotnet	Pre-job: Download secrets: EngKeyVault	Log
2775557	dotnet-runtime	Run performance job script	Log
2775495	dotnet-dotnet	Build (centOSStreamContainer)	Log
Displaying 100 of 965 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
55	254	965

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez