iOS automated test durations & multiple targets

We currently use KIF to run our automated UI tests, which take over 3.5 hours to run. Due to this, we split our single test target into 4 smaller targets, each now taking ~ 45 mins to run.

  • Is it possible for us to run 4 Xcode targets in parallel, as a part of a single build?

  • If not, is there a way to run multiple workflows in parallel per trigger?

Automated UI tests can be very time consuming, and I’d be interested if you all had any feedback on how others are setting these up to mitigate the build duration constraints.

1 Like

Great question @MattNewberry, thanks for asking this here, I’d be interested to hear how others solve this as well!

Just a couple of notes:

If anyone have any insight or tips/tricks for this, please let us know! Also, if we can improve something, feel free to create #feature-requests or vote and comment on existing ones (e.g. Parallel Builds: Seperate stacks for seperate workflows + trigger multiple workflows (builds) at the same time)

I’ve got an idea to run by you… Aside from bluepill, which I’m testing with great results, here’s a thought:

  1. Setup workflows for each Xcode scheme consisting of a subsect of our UI tests (4 in total)
  2. Create a simple workflow that runs a script calling the Build Trigger API for each of the 4 workflows
  3. Run the API workflow as the main trigger for each push, pr, etc…

Essentially, since the UI doesn’t let me execute multiple workflows per trigger in parallel, this would allow me to run our 4 targets in parallel, correct?

The main gain here is by running in parallel, which each test runner using bluepill (or pxctest) to further parallelize test batches, we’d end up with a very fast build system.

1 Like

Ohh, definitely! Unfortunately this is not a “just a few clicks” operation, but it definitely can be done - I shared more info about something like this here, where the question was how you could trigger a build for one repository when a different one changes, which requires the same “API workflow” solution you mentioned: Can a bitrise “build” be triggered from more than one repository?

If you have any sample config and any stats how much that speeds up things, please let us know, we’d definitely be interested in this! (bluepill is on our list to explore and see if we could integrate into an existing step, or create a new step, if it worth it)

Sounds absolutely fantastic! We have plans to make this easier too, by allowing you to define which things to run in parallel right in your build config. We have the basic sketches about the “design”, but we’re still discussing the details, how it could be compatible with the current Bitrise yml format and in general, etc. Hopefully we can work on the new “v2” config this year, but first we have to finish a couple of other things.

One more thing, we’re experimenting with new, more powerful build machine configs, which can speed up your builds if the build is CPU and/or RAM intensive. These new configs should be available soon, but will cost more, unfortunately.

If you’re interested in joining the private beta, we can give you a free trial to see if it helps to speed up your builds (might not, depends on the project and whether it can utilize the faster CPU and additional available RAM).

I’d like to join the private beta.

1 Like