New step: Xcode Test Shard Calculation

[jeremyp]

Calling all iOS developers!

We’ve released a new feature which helps automate test sharding for customers with Xcode projects, reducing wall clock time and time and effort spent managing CI configuration.

What problem does this solve?

Imagine you have a 30 minute long un-sharded test suite (first diagram). Say it contains 300 tests and you choose to split them randomly into 3 shards of 100 tests each. As each test has a different execution time you might end up with what you see in the second diagram - 3 shards of differing length. The wall clock time is less than the original 30 minutes but it’s somewhat more than the optimal split, shown in the final diagram.

To minimise build times, the trick is to arrange your tests in a way which means each shard has the same duration, which might mean having different numbers of tests in each shard. Calculating the optimal split takes effort to setup and maintain. Some Bitrise customers have the time and resources to do this. This new Xcode Test Sharding step is for those who don’t and want Bitrise to do it for them.

How does it work?

All you have to do is use the new Xcode Test Shard Calculation step
combined with the Parallelism feature.

Follow the steps below and within a few clicks you’ll have a pipeline which automatically shards your Xcode tests based on an optimization algorithm provided by Bitrise, without you needing to do anything at all to your Xcode project.

[Note: this example assumes that you already have a setup where you build your app once and use the shared test bundle for test execution. If you want to know how to achieve this, check out this workflow recipe.]

Here’s how to set up your Xcode project pipeline to include test sharding based on Bitrise’s algorithm:

1. Go to the Env Vars menu option in the Workflow Editor and create a new environment variable called something like $NUMBER_OF_SHARDS. Assign a value to this env var
2. Switch to the Pipeline menu option in the Workflow Editor left nav and add the Xcode Test Shard Calculation step to the workflow that builds your Xcode app
3. Click on the step to open up its configuration options
4. Fill in the Product Path parameter with the path to your xctestrun file, then add your new environment variable to the Shard Count field, like so:

5. Once the shards are calculated the rest of the pipeline workflows also need to know about it. This is achieved by sharing the shard calculation step via the Deploy To Bitrise.io step. Click on the Deploy workflow and enter the system env vars BITRISE_TEST_SHARDS_PATH and BITRISE_TEST_BUNDLE_PATH in the Files to Share Between Pipeline Stages field as shown:

6. On the pipeline canvas, click the gear icon the workflow that does the testing (in this example it’s called test-without-building), open the Pipeline Conditions section and add your new environment variable to the Parallel Copies field (see below)

That’s it ! You just set up test sharding for your project without building and maintaining your own shard optimization calculation.

Happy testing!