restarted. Azure DevOps can't recover from restarts.

[lewing]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784852

Build leg reported

Build / linux-x64 debug Libraries_AllConfigurations

Pull Request

dotnet/runtime#106599

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "restarted. Azure DevOps can't recover from restarts.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=784852
Error message validated: [restarted. Azure DevOps can't recover from restarts.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/26/2024 7:12:18 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1001161	dotnet/roslyn	Test_Windows_CoreClr_UsedAssemblies_Debug
1000207	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113306
1001047	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114080
1001023	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113976
1000981	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114100
1000976	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114101
1000941	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114098
1000681	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114009
1000730	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113976
1000633	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#103351
1000623	dotnet/runtime	browser-wasm linux Release LibraryTests
1000363	dotnet/sdk	Browser_Shortstack_wasm		dotnet/sdk#48021
1000348	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114086
1000236	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114027
1000113	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#48016
1000098	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114085
1000072	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113983
1000059	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113286
1000025	dotnet/sdk	Browser_Shortstack_wasm		dotnet/sdk#48013
999915	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114080
999920	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#48016
999881	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110818
999744	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114051
999726	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113876
999717	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#48029
999676	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114070
999668	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114071
999514	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#48015
999511	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113466
999212	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111791
998987	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114042
999137	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110818
999091	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114008
999020	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47997
999002	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113976
998983	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114060
998938	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113954
998926	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113111
998865	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#48018
998378	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113976
998341	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113409
998247	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113409
998230	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#111557
998128	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47974
997799	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110554
997857	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113907
997842	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47995
997618	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114017
997595	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114023
997380	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113982
997548	dotnet/runtime	browser-wasm linux Release LibraryTests
997516	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114020
997437	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113409
997409	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113743
997340	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113995
997277	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47987
997238	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114010
997230	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114009
995786	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113963
997220	dotnet/runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter
997199	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113987
997201	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47961
997176	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114008
996996	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#114002
996917	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113944
996594	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113905
996727	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113995
996678	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113993
996453	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113743
996373	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113460
995797	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113483
993750	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113905
995928	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47949
995921	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113968
995726	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47941
995711	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47898
995590	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47939
995419	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47933
995385	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#110554
995346	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113956
995283	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113908
995194	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113953
995110	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113500
994887	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47867
994774	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113907
994693	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113929
994664	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113927
994088	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47903
994018	dotnet/sdk	Browser_Shortstack_wasm		dotnet/sdk#47895
993950	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113911
993839	dotnet/sdk	Android_Shortstack_arm64		dotnet/sdk#47611
993303	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113385
2671590	dotnet-dotnet	AzureLinux_x64_Cross_arm
992499	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113766
992362	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113112
991955	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113846
991144	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
990850	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113822
985121	dotnet/runtime	browser-wasm linux Release LibraryTests		dotnet/runtime#113507
989925	dotnet/runtime	windows-x64 Debug CoreCLR_Libraries		dotnet/runtime#113666
Displaying 100 of 375 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
22	92	375

[am11]

It seems like all reports are pointing to linux-x64 dev-innerloop leg from this definition: https://dev.azure.com/dnceng-public/public/_build?definitionId=133. GitHub doesn't sync the status and keeps showing as if the job is running for days.. Opened dotnet/runtime#108581 to disable the leg.

[am11]

Just before the timeout we see low memory warnings like these:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=829130&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857

then it hangs for ~20 minutes or so before giving up. The build command has -allConfigurations so it builds all product+test assemblies for all platforms ({linux,win,osx,freebsd,illumos}-{x86,x64,arm,arm64,riscv64.. etc.}) all in one invocation of build (which isn't exactly efficient as we should probably group them..), which means, as it stands, this leg needs decent amount of RAM.

[steveisok]

@ilyas1974 @markwilkie I don't think our RAM consumption has increased that much to not be able to handle this configuration. Thoughts?

[ericstj]

I took a look at a few passing builds, many have the same logs where they approach 95% memory usage but eventually succeed.

Here are some samples:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=836601&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857
https://dev.azure.com/dnceng-public/public/_build/results?buildId=838110&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=3923
https://dev.azure.com/dnceng-public/public/_build/results?buildId=837745&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=4228

Near this point I see logs like this:

  initializing ChangeMakerService with capabilities: Baseline, AddMethodToExistingType, AddStaticFieldToExistingType, AddInstanceFieldToExistingType, NewTypeDefinition, ChangeCustomAttributes, UpdateParameters, GenericAddMethodToExistingType, GenericUpdateMethod, GenericAddFieldToExistingType
  baseline ready
  got a change
  parsing patch #1 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v1.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.1.dmeta
  got a change
  parsing patch #2 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v2.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.2.dmeta
  done

It looks to me like this is coming from https://github.com/dotnet/hotreload-utils/blob/254ec75de6127c368827d15c3af2477095b8b1b4/src/Microsoft.DotNet.HotReload.Utils.Generator/EnC/ChangeMakerService.cs#L28

Does anyone have an idea why hotreload would be running during a build?? I could imagine that if some hot reload service was runnign during a build or if tests were running while the product was building that could explain high memory usage.

[snakex64]

This seems to be happening consistently in PR #109320 . Should I just try re-running a third time or is it never gonna work?