iOS Unit Tests Fail randomly due to timeouts waiting for expectations

Question & Answer Builds
1

Bitrise Build Issue Report template

Description of the issue

We have a set of hundreds of unit tests. The tests always pass locally, running on clean versions of the same simulator we use in Bitrise. However, when the tests run in Bitrise there’s a 50-75% chance that at least one of the tests will fail (it’s never the same test twice) due to a timeout waiting for expectations. ( for example, " Asynchronous wait failed: Exceeded timeout of 2 seconds, with unfulfilled expectations: “delay”.)

Environment:

Where did the issue happen?

During the Xcode-test workflow

If on Bitrise.io: which stack? If not on Bitrise.io: on what operating system? (Plus any other information you can share)

Xcode 12.4.x, on macOS 10.15.7 (Catalina)

Which build Step causes the issue and which version of the step?

Xcode-test

Reproducibility

  • Does a “Rebuild” help? (You can trigger a rebuild from the Build’s page, by clicking the “Rebuild” button in the top right corner of a finished build) : NO
  • Does a rebuild without caches help? (You can remove the Cache:Pull and Cache:Push steps temporarily to not to use the cache, or you can delete all the caches on the Settings tab of the app. : NO
  • Does the issue happen sporadically, or every time? : Sporadically, and the test that fails changes randomly as well.
  • Does upgrading the build Step to the latest version help? : NO
  • When did the issue start? : We have had this issue ever since we started using Bitrise > 2 years ago.

Local reproduction

Can it be reproduced on your own Mac/PC by following our local debug guide? Please follow at least the first section (“Testing with a full clean git clone”) to make sure to test the state of the code what bitrise.io will get when it does a git clone in the clean environment! If possible please note which sections you tried.

No. Tests do not fail locally.

Local reproduction: Linux / Android (docker based) stack builds

Can it be reproduced by running the build locally, after doing a new git clone of the repository into the /tmp directory and running the build from there with the Bitrise CLI ( https://www.bitrise.io/cli )? If no, can it be reproduced with Docker (using the same docker images / environment we use on bitrise.io)? Related guide: http://devcenter.bitrise.io/docker/run-your-build-locally-in-docker/ .

Build log

Please copy paste the build’s bitrise.io URL here (or if the issue happens somewhere else then the full logs), or if you can’t share the url / log here then send the url or full log through a private channel (e.g. email - https://www.bitrise.io/contact ), with a link to the related Discuss issue.

https://app.bitrise.io/api/build/162f4065df9bd6b3/logs.json?&download=log

2

Any thoughts or suggestions from Bitrise on this? Should we update our expectation timeouts to 30 seconds across the board? Is 30 seconds long enough? I saw a similar post from someone else on these forums a few months back, and there was no response to them either.

A CI system that constantly reports false test errors is incredibly frustrating and a big time sink as we constantly need to check if the test failure was a “real” failure.

1 Like
3

Hello @lj-dickey :wave:

The reason for the error could be that we’ve had performance issues on our Public Cloud recently, but we made improvements, and it should be better now.

Could you let me know if you’ve experienced any changes, or if the issue is still persistent?
If so, could you send us a related build URL, and enable Support Access (Enabling the Bitrise Support user for your app | Bitrise DevCenter) so we can take a closer look? :slight_smile:

4

I have already turned on support access, a build link is included above, and this is a problem we’ve had ever since we’ve started using Bitrise. We fix some tests, and then new ones, or ones that hadn’t failed before, start to fail. We rarely go a day without a failure that corrects itself once we re-run the test job.

5

Hello there :wave:

In case anyone encounters a similar issue and come across this thread, I wanted to let you know that increasing the timeouts to 30 seconds solved the issue :slight_smile:

6

I’ve also been experiencing this. I tried bumping my tests up to 40 second timeouts and it still happens. Actually at this point we just hit our 30 minute build cap and it gets aborted.

7

Hey @josh150 :wave:

Could you please submit a ticket to Bitrise at Bitrise with the details? Make sure you send us a build URL and enable Support Access as well! :slight_smile:

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.