Skip to content

CI flakiness: mono interpreter build getting killed #114123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
lewing opened this issue Apr 1, 2025 · 4 comments
Open

CI flakiness: mono interpreter build getting killed #114123

lewing opened this issue Apr 1, 2025 · 4 comments
Labels
area-Infrastructure-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' disabled-test The test is disabled in source code against the issue Known Build Error Use this to report build issues in the .NET Helix tab untriaged New issue has not been triaged by the area owner

Comments

@lewing
Copy link
Member

lewing commented Apr 1, 2025 •
edited by build-analysis bot
Loading

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=998900&view=logs&j=3e93a48f-6568-53f2-76b3-1285c183c39d
Build error leg or test failing: Build / browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter / Build Tests
Pull request: #114058

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "RuntimeVariant=monointerpreter.*exited with code 137",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=998900
Error message validated: [RuntimeVariant=monointerpreter*exited with code 137.]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 4/1/2025 9:14:37 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=998900
Error message validated: [RuntimeVariant=monointerpreter.*exited with code 137]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/1/2025 9:15:42 PM UTC

Report

Build Definition Step Name Console log Pull Request
1055500 dotnet/runtime Build Tests Log #116166
1054323 dotnet/runtime Build Tests Log #116107
1054044 dotnet/runtime Build Tests Log #115609
1053821 dotnet/runtime Build Tests Log #113956
1051675 dotnet/runtime Build Tests Log
1050749 dotnet/runtime Build Tests Log
1050568 dotnet/runtime Build Tests Log #115704
1049991 dotnet/runtime Build Tests Log #115858
1049202 dotnet/runtime Build Tests Log
1047881 dotnet/runtime Build Tests Log
1047361 dotnet/runtime Build Tests Log #115818
1045781 dotnet/runtime Build Tests Log #115750
1045198 dotnet/runtime Build Tests Log #115724
1045019 dotnet/runtime Build Tests Log #114148
1044192 dotnet/runtime Build Tests Log #115544
1043795 dotnet/runtime Build Tests Log #115576
1043787 dotnet/runtime Build Tests Log
1043474 dotnet/runtime Build Tests Log #115039
1043408 dotnet/runtime Build Tests Log #115531
1043376 dotnet/runtime Build Tests Log #115576
1043385 dotnet/runtime Build Tests Log
1043365 dotnet/runtime Build Tests Log
1043357 dotnet/runtime Build Tests Log #115545
1043242 dotnet/runtime Build Tests Log #115408
1043079 dotnet/runtime Build Tests Log #113956
1042975 dotnet/runtime Build Tests Log #115574
1042922 dotnet/runtime Build Tests Log #115591
1042899 dotnet/runtime Build Tests Log
1042786 dotnet/runtime Build Tests Log #115574
1042728 dotnet/runtime Build Tests Log #115549
1042673 dotnet/runtime Build Tests Log #114438
1042459 dotnet/runtime Build Tests Log #115573
1042605 dotnet/runtime Build Tests Log #115058
1042420 dotnet/runtime Build Tests Log #115563
1042578 dotnet/runtime Build Tests Log #115574
1042552 dotnet/runtime Build Tests Log #115610
1042330 dotnet/runtime Build Tests Log #115408
1042465 dotnet/runtime Build Tests Log #115605
1042414 dotnet/runtime Build Tests Log #115427
1042349 dotnet/runtime Build Tests Log #115573
1041448 dotnet/runtime Build Tests Log #115553
1042324 dotnet/runtime Build Tests Log #113956
1041806 dotnet/runtime Build Tests Log #115494
1042282 dotnet/runtime Build Tests Log #115251
1042106 dotnet/runtime Build Tests Log #115584
1042019 dotnet/runtime Build Tests Log #115375
1041906 dotnet/runtime Build Tests Log #115544
1041632 dotnet/runtime Build Tests Log #115563
1041747 dotnet/runtime Build Tests Log #115584
1041706 dotnet/runtime Build Tests Log #115580
1041705 dotnet/runtime Build Tests Log
1041628 dotnet/runtime Build Tests Log #115573
1041490 dotnet/runtime Build Tests Log #115408
1041509 dotnet/runtime Build Tests Log #115568
1041497 dotnet/runtime Build Tests Log #115566
1041461 dotnet/runtime Build Tests Log #115335
1041453 dotnet/runtime Build Tests Log #115563
1041435 dotnet/runtime Build Tests Log #114148
1041427 dotnet/runtime Build Tests Log #115113
1041398 dotnet/runtime Build Tests Log #115375
1041288 dotnet/runtime Build Tests Log #115556
1041350 dotnet/runtime Build Tests Log #114148
1041343 dotnet/runtime Build Tests Log #115408
1041306 dotnet/runtime Build Tests Log #115553
1041309 dotnet/runtime Build Tests Log #115513
1041259 dotnet/runtime Build Tests Log #114438
1041224 dotnet/runtime Build Tests Log #115408
1041207 dotnet/runtime Build Tests Log
1041158 dotnet/runtime Build Tests Log #115549
1040895 dotnet/runtime Build Tests Log #115538
1040841 dotnet/runtime Build Tests Log #114438
1040823 dotnet/runtime Build Tests Log #115375
1040764 dotnet/runtime Build Tests Log #114881
1040707 dotnet/runtime Build Tests Log
1040533 dotnet/runtime Build Tests Log #115375
1040530 dotnet/runtime Build Tests Log #114881
1040520 dotnet/runtime Build Tests Log #115516
1040497 dotnet/runtime Build Tests Log
1040339 dotnet/runtime Build Tests Log
1040229 dotnet/runtime Build Tests Log #115408
1040208 dotnet/runtime Build Tests Log #115465
1040024 dotnet/runtime Build Tests Log #114881
1040013 dotnet/runtime Build Tests Log #115497
1039991 dotnet/runtime Build Tests Log #115494
1039973 dotnet/runtime Build Tests Log
1039922 dotnet/runtime Build Tests Log #115436
1039916 dotnet/runtime Build Tests Log #115437
1039841 dotnet/runtime Build Tests Log
1039786 dotnet/runtime Build Tests Log #115443
1039768 dotnet/runtime Build Tests Log #114901
1039606 dotnet/runtime Build Tests Log
1039581 dotnet/runtime Build Tests Log #115335
1039493 dotnet/runtime Build Tests Log #115408
1039394 dotnet/runtime Build Tests Log #115428
1039386 dotnet/runtime Build Tests Log #114901
1039324 dotnet/runtime Build Tests Log #115335
1039302 dotnet/runtime Build Tests Log
1038979 dotnet/runtime Build Tests Log
1038946 dotnet/runtime Build Tests Log #115408
1038942 dotnet/runtime Build Tests Log #115443
Displaying 100 of 218 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 6 218
@lewing lewing added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels Apr 1, 2025
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Apr 1, 2025
@dotnet-policy-service Dotnet Policy Service
Copy link
Contributor

Tagging subscribers to this area: @akoeplinger, @matouskozak
See info in area-owners.md if you want to be subscribed.

@akoeplinger akoeplinger changed the title Killing mono interpreter build CI flakiness: mono interpreter build getting killed Apr 1, 2025
@akoeplinger akoeplinger mentioned this issue Apr 1, 2025
Merged
@matouskozak
Copy link
Member

fyi: @BrzVlad

@AndyAyersMS AndyAyersMS mentioned this issue Apr 11, 2025
Merged
4 tasks
@pavelsavara
Copy link
Member

pavelsavara commented May 16, 2025 •
edited
Loading

We use mono interpreter to run msbuild, right ?

This is building 3 groups of many small tests

Image

That combined with WASM toolchain memory demands makes msbuild run out of memory (docker kills it).
This is OOM of the msbuild, not OOM of the individual test projects.
I think this is OOM kill, not timeout because the duration before the kill fluctuates.

As far as I can tell, it never passes for WASM and it's killed in the group 2 already.

Log

2025-05-15T23:09:37.6650337Z   Number of test projects in group 2: 2366
2025-05-15T23:09:37.7341670Z   Building managed test group 2: "/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"
2025-05-15T23:09:37.9271786Z   MSBuild version 17.14.0-preview-25177-05+903614e16 for .NET
2025-05-15T23:09:37.9852691Z   
2025-05-15T23:09:40.5433817Z     Number of test projects in group 2: 2366
2025-05-15T23:09:37.7341670Z   Building managed test group 2: "/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"
2025-05-15T23:09:37.9271786Z   MSBuild version 17.14.0-preview-25177-05+903614e16 for .NET
2025-05-15T23:09:37.9852691Z   
2025-05-15T23:09:40.5433817Z     Number of test projects in group 2: 2366
2025-05-15T23:09:41.8292504Z     XUnitWrapperGenerator -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/Common/XUnitWrapperGenerator/XUnitWrapperGenerator/XUnitWrapperGenerator.dll
2025-05-15T23:09:42.6596953Z     KeepAliveBoxOpt -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.dll
2025-05-15T23:09:42.9728086Z     sinlib_cs -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/Boxing/xlang/sinlib_cs/sinlib_cs.dll
2025-05-15T23:09:43.2633097Z     testlib_misc -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/misc/testlib_misc/testlib_misc.dll
2025-05-15T23:09:43.6004947Z     threads1_cs_do -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/misc/threads1_cs_do/threads1_cs_do.dll
2025-05-15T23:09:43.8861163Z     testlib_xassem -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/xassem/testlib_xassem/testlib_xassem.dll
...
2025-05-15T23:23:01.9327210Z     b53977 -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Regression/CLR-x86-JIT/V1-M12-Beta2/b53977/b53977/b53977.dll
2025-05-15T23:23:08.5186861Z   /__w/1/s/dotnet.sh: line 27: 50331 Killed                  ${dotnetPath} "$@"
2025-05-15T23:23:08.5814254Z ##[error]src/tests/build.proj(501,5): error MSB3073: The command ""/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"" exited with code 137.

@pavelsavara pavelsavara mentioned this issue May 16, 2025
Merged
@pavelsavara pavelsavara added the disabled-test The test is disabled in source code against the issue label May 16, 2025
@pavelsavara
Copy link
Member

I disabled part of the test on browser, because 100% fail rate was not helping anybody. This is ActiveIssue now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-Infrastructure-mono blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' disabled-test The test is disabled in source code against the issue Known Build Error Use this to report build issues in the .NET Helix tab untriaged New issue has not been triaged by the area owner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lewing @pavelsavara @matouskozak