Per-project opt-out of minimum number of tests

[RussKie]

Background and Motivation

I'm helping a sister team with SFI work. The team maintains a handful of repositories with more or less the same structure and the same approach which involves productions projects, unit and integration test project and manual verification test projects (which exercise resources directly in Azure).

More often than not, all tests in the manual verification test projects are decorated with the IgnoreAttribute.
This causes the test SDK to fail the test run (i.e., dotnet test) because it violates the minimum tests policy:

The manual verification test projects are normal test projects as far as the infra is concerned. That is, there look similar to this:

 <Project Sdk="MSTest.Sdk/3.6.4">
   <PropertyGroup>
     <TargetFramework>net8.0</TargetFramework>
     <OutputType>Library</OutputType>
   </PropertyGroup>
 </Project>

Proposed Feature

The test SDK allows to specify the minimum number of tests that are expected to run with `--minimum-expected-tests', however it doesn't accept 0.

It's possible to mitigate the failure by employing -ignore-exit-code with the exit code 8.

The issue with both of these approaches (even if the first point was possible) is that it affects all test projects. And ignoring possible test issues with traditional unit and integration test project is extremely undesirable.

It would be great to have a per-project opt-out or suppression mechanism that would allow to use dotnet test for the whole solution.

[nohwnd]

Thanks for reporting, let's discuss how to do this :)

Team:

As described above, some options would be nice to apply just to a project or subset of projects. I don't think we have something like that in place, unlike VSTest where you can provide runsettings per project.

The thing to discuss is how (and if) to expose the desired functionality to the users, and how to make it easy to understand which setting is per project and which one is per solution.

What looks promising is the testconfig.json file, which can be placed next to project, but it does not expose all the options that are available on commandline, and also does not easily allow to change the values based on environment. e.g. if locally I want to filter out all integration tests, but not on CI.

(also 3.8.* refactoring fixed copying the testconfig.json to output, which does not work in 3.6.4, so use that version or newer when trying this).

[Evangelink]

@nohwnd it's possible to have a testconfig.json file attached to a given project and picked up only by that project. The file should either be named Assembly.testconfig.json and copied to the output or if the user has a deps to MTP MSBuild, then the file will be renamed and copied to the output automatically: https://github.com/microsoft/testfx/blob/4beefc56a7b04c38d52ad4ae3c9c3bb7d6581640/src/Platform/Microsoft.Testing.Platform.MSBuild/Tasks/ConfigurationFileTask.cs

[nohwnd]

Yup, (that is what I am describing above) but can I configure also the ignore-exit-code via that file? I tried and failed to do that.

[Youssef1313]

If you need to ignore exit code for a specific project, then you need to add the following property in csproj:

<TestingPlatformCommandLineArguments>$(TestingPlatformCommandLineArguments) --ignore-exit-code 8</TestingPlatformCommandLineArguments>

[Youssef1313]

@RussKie btw, the <OutputType>Library</OutputType> part shouldn't exist. It's going to be corrected by the SDK anyways though.

[RussKie]

@RussKie btw, the <OutputType>Library</OutputType> part shouldn't exist. It's going to be corrected by the SDK anyways though.

TIL
It may worth catching that with an analyser of some sort.

[RussKie]

If you need to ignore exit code for a specific project, then you need to add the following property in csproj:

$(TestingPlatformCommandLineArguments) --ignore-exit-code 8

I've tested this suggestion, and it worked! Thank you very much.

I'm wondering how this rather obscure knowledge can be made more discoverable. The error message is very opaque - there's not even an error code one could go search for. The documentation is technically there (e.g., TestingPlatformCommandLineArguments, SDK options (-ignore-exit-code), ignore exit codes), but it's spread across various pages, and unless you know what exactly you need to search for the change of you finding it is close to 0.

Consider making the error more developer-friendly - e.g., if it failed because the required number of tests hasn't been run (or successful), then make it clear. Also, state the return code was 8 (or 9, or 27). If you define error codes, then there could be doc pages where the error codes are described, and the docs can provide some actionable suggestions on how the errors could be (self-)resolved.

[Youssef1313]

Thanks for the feedback @RussKie. I agree the documentation needs to improve, and I'm planning to make some improvements. Also tagging @mariam-abdulla to see if there is an opportunity for dotnet test to show the the exit code with a link to https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-exit-codes

[Youssef1313]

Some doc updates are being done in dotnet/docs#45095 (and more will come in follow-up PR(s)). Keeping this issue open for the question on if we can improve the experience with dotnet test.