We currently use KIF to run our automated UI tests, which take over 3.5 hours to run. Due to this, we split our single test target into 4 smaller targets, each now taking ~ 45 mins to run.
Is it possible for us to run 4 Xcode targets in parallel, as a part of a single build?
If not, is there a way to run multiple workflows in parallel per trigger?
Automated UI tests can be very time consuming, and I’d be interested if you all had any feedback on how others are setting these up to mitigate the build duration constraints.
Great question @MattNewberry, thanks for asking this here, I’d be interested to hear how others solve this as well!
Just a couple of notes:
If you need more than 45 mins for a single build and you have a subscription with at least 2 concurrencies, feel free to ping us and we can bump the build limit to 90 mins, but of course faster test results are way better.
There are a couple of new tools which aim to help with parallelized iOS testing: Allow Xcode testing step for multiple devices/operating systems - #4 by MattNewberry - but as these are not “official” tools we did not integrate any of these into the standard Xcode Test step; but you should be able to use these from e.g. a Script step
I’ve got an idea to run by you… Aside from bluepill, which I’m testing with great results, here’s a thought:
Setup workflows for each Xcode scheme consisting of a subsect of our UI tests (4 in total)
Create a simple workflow that runs a script calling the Build Trigger API for each of the 4 workflows
Run the API workflow as the main trigger for each push, pr, etc…
Essentially, since the UI doesn’t let me execute multiple workflows per trigger in parallel, this would allow me to run our 4 targets in parallel, correct?
The main gain here is by running in parallel, which each test runner using bluepill (or pxctest) to further parallelize test batches, we’d end up with a very fast build system.
Ohh, definitely! Unfortunately this is not a “just a few clicks” operation, but it definitely can be done - I shared more info about something like this here, where the question was how you could trigger a build for one repository when a different one changes, which requires the same “API workflow” solution you mentioned: Can a bitrise "build" be triggered from more than one repository? - #4 by viktorbenei
If you have any sample config and any stats how much that speeds up things, please let us know, we’d definitely be interested in this! (bluepill is on our list to explore and see if we could integrate into an existing step, or create a new step, if it worth it)
Sounds absolutely fantastic! We have plans to make this easier too, by allowing you to define which things to run in parallel right in your build config. We have the basic sketches about the “design”, but we’re still discussing the details, how it could be compatible with the current Bitrise yml format and bitrise.io in general, etc. Hopefully we can work on the new “v2” config this year, but first we have to finish a couple of other things.
One more thing, we’re experimenting with new, more powerful build machine configs, which can speed up your builds if the build is CPU and/or RAM intensive. These new configs should be available soon, but will cost more, unfortunately.
If you’re interested in joining the private beta, we can give you a free trial to see if it helps to speed up your builds (might not, depends on the project and whether it can utilize the faster CPU and additional available RAM).