🚫 Test quarantine now also works for Swift Testing

Changelog Website
, , , ,
1

:light_bulb: What’s new**

Test quarantine on Bitrise now works for Swift Testing tests, including display-named tests and tests in nested suites. Additionally, $BITRISE_QUARANTINED_TESTS_JSON includes a testCaseIdentifier key with each test’s Swift Testing identifier, for use in custom test runners.

:puzzle_piece: Problems we solved

  1. Quarantine capability identified a test by its class name and test case name. For Swift Testing tests with a display name, such as @Test("Score is computed when the guess matches the target"), the test report stored the display name instead of the function name. xcodebuild did not recognize it and ran the test anyway, with no error or warning. The test showed as quarantined but still would be executed in builds.
  2. Tests in nested Swift Testing suites had the same problem.

:white_check_mark: Solution

Bitrise now stores each test’s Swift Testing identifier, such as MySuite/scoreIsComputedWhenGuessMatchesTarget(), and uses it to exclude the test from the run.

:guide_dog:How to use it

Use Xcode 16.3 or later, Xcode Test for simulator 6.4.1 or later (or Xcode Test without building 1.0.0 or later), and Deploy to Bitriseio 2.25.1 or later. Tests quarantined before this release have no stored identifier. If one still runs, remove it from quarantine and add it again from a new test report. iOS Device Testing does not return Swift Testing results, so this does not apply there.

Detecting and quarantining flaky tests