-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unit Testing - Tests with coding errors disappear from test run results #18
Comments
@spascoe, thank you for taking time to write a comprehensive report! I've tested this and can replicate the behavior you've described. Both fact based unit tests and native LibPQ unit tests are affected. Here is why it happens:
Even though everything works exactly as documented, I agree it is not the best possible behavior. I think we can propagate errors for modules that look like test suites with a fake "test runner". The question is how do we decide that a module looks like a test suite? What's your opinion? I'll start implementing the workaround with some placeholder logic for now. |
Previously if a module containing the test suite could not be imported without errors such test suite would be invisible to discovery tools. Now import errors are propagated to discovery results if a module looks like a test suite. See: #18
Here is what I've come up with: 8482ec7 If a module errors out on import and its name starts with "test" (case insensitive), all import errors are propagated to |
I tested your solution, and got exactly the result that I was expecting to see. That is an awesome solution.
|
Thanks :) Your preferred report layout is already supported by LibPQ/Modules/UnitTest.Discover.pq Lines 1 to 10 in 8482ec7
I'll leave this in a feature branch overnight to see if some bright idea pops up in my head. I don't like relying on the module name much, because it's not really required to start with "test" by any other part of UnitTest framework. Inspecting raw source code for the module with import error seems to be excessively difficult. The loader does not expose it in any reusable way, and replicating a lot of the loader logic or modifying the loader itself are both worse in my opinion than current filename solution. PS: I am glad that you find this project interesting! Would you mind leaving some feedback on how/where you use it? Please reply to this issue |
I've found a clean way to propagate custom metadata along with module evaluation errors, so now test suites can be correctly discovered in modules with arbitrary names. You'll need to update Demo of improved behavior:
I've updated the changelog and merged changes into |
When a unit test file has an error in it, the entire file's tests disappear from the UnitTest Results. To Reproduce do the following:
The Test.MicrosoftUnitTestDemo tests are no longer listed. It would be more desirable to list the file as a single row with an error. I've looked through the code, and tried a number of things, without success, so thought it was worthy of an "Issue".
I'm glad to help with resolving, if someone could give me some pointers.
The text was updated successfully, but these errors were encountered: