Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XCTest results aren't correct when you have identical XCTestSuites different test targets #1036

Open
plemarquand opened this issue Aug 28, 2024 · 0 comments
Labels
bug Something isn't working XCTest Issues related to testing integration for XCTest

Comments

@plemarquand
Copy link
Contributor

On non Darwin platforms, XCTest results aren't correct when you have identical XCTestSuites different test targets.

To Reproduce

Create a test project with two test targets defined.

// Package.swift
...
targets: [
  .target(name: "TestApp"),
  .testTarget(name: "TestTarget1"),
  .testTarget(name: "TestTarget2"),
]

And create two test suites, one in each target.

// TestTarget1/TestFoo.swift
final class MyTestSuite: XCTestCase {
  func testFoo() {}
}
// TestTarget2/TestFoo.swift
final class MyTestSuite: XCTestCase {
  func testFoo() {}
}

Now select both test suites in the Test Explorer and run both. Notice only one test is marked as passing. This stems from the fact that unlike on macOS, on non Darwin platforms the test target is not included in the test run output.

Expected behavior
Both tests show as passing.

Environment

  • OS: Ubuntu 20.04
  • Swift version: Swift version 6.0-dev (LLVM 31f0cdc6d5ff8a6, Swift 497b266c2188eae)
  • Visual Studio Code version: 1.92.2
  • vscode-swift version: 1.10.7

Additional context

Here is the console output for a single suite run on Ubuntu

Test Suite 'Selected tests' started at 2024-08-27 20:23:17.187
Test Suite 'MyTestSuite' started at 2024-08-27 20:23:17.188
Test Case 'MyTestSuite.testFoo' started at 2024-08-27 20:23:17.188
Test Case 'MyTestSuite.testFoo' passed (0.0 seconds)
Test Suite 'MyTestSuite' passed at 2024-08-27 20:23:17.188

And the same run on macOS:

Test Suite 'Selected tests' started at 2024-08-28 09:13:42.501.
Test Suite 'TestTarget1.xctest' started at 2024-08-28 09:13:42.502.
Test Suite 'MyTestSuite' started at 2024-08-28 09:13:42.502.
Test Case '-[TestTarget1.MyTestSuite testFoo]' started.
Test Case '-[TestTarget1.MyTestSuite testFoo]' passed (0.001 seconds).
@plemarquand plemarquand added the bug Something isn't working label Aug 28, 2024
@award999 award999 added the XCTest Issues related to testing integration for XCTest label Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working XCTest Issues related to testing integration for XCTest
Development

No branches or pull requests

2 participants