iOS Unit Tests Fail randomly due to timeouts waiting for expectations

#1

Bitrise Build Issue Report template

Description of the issue

We have a set of hundreds of unit tests. The tests always pass locally, running on clean versions of the same simulator we use in Bitrise. However, when the tests run in Bitrise there’s a 50-75% chance that at least one of the tests will fail (it’s never the same test twice) due to a timeout waiting for expectations. ( for example, " Asynchronous wait failed: Exceeded timeout of 2 seconds, with unfulfilled expectations: “delay”.)

Environment:

Where did the issue happen?

During the Xcode-test workflow

If on Bitrise.io: which stack? If not on Bitrise.io: on what operating system? (Plus any other information you can share)

Xcode 12.4.x, on macOS 10.15.7 (Catalina)

Which build Step causes the issue and which version of the step?

Xcode-test

Reproducibility

  • Does a “Rebuild” help? (You can trigger a rebuild from the Build’s page, by clicking the “Rebuild” button in the top right corner of a finished build) : NO
  • Does a rebuild without caches help? (You can remove the Cache:Pull and Cache:Push steps temporarily to not to use the cache, or you can delete all the caches on the Settings tab of the app. : NO
  • Does the issue happen sporadically, or every time? : Sporadically, and the test that fails changes randomly as well.
  • Does upgrading the build Step to the latest version help? : NO
  • When did the issue start? : We have had this issue ever since we started using Bitrise > 2 years ago.

Local reproduction

Can it be reproduced on your own Mac/PC by following our local debug guide? Please follow at least the first section (“Testing with a full clean git clone”) to make sure to test the state of the code what bitrise.io will get when it does a git clone in the clean environment! If possible please note which sections you tried.

No. Tests do not fail locally.

Local reproduction: Linux / Android (docker based) stack builds

Can it be reproduced by running the build locally, after doing a new git clone of the repository into the /tmp directory and running the build from there with the Bitrise CLI ( https://www.bitrise.io/cli )? If no, can it be reproduced with Docker (using the same docker images / environment we use on bitrise.io)? Related guide: http://devcenter.bitrise.io/docker/run-your-build-locally-in-docker/ .

Build log

Please copy paste the build’s bitrise.io URL here (or if the issue happens somewhere else then the full logs), or if you can’t share the url / log here then send the url or full log through a private channel (e.g. email - https://www.bitrise.io/contact ), with a link to the related Discuss issue.

https://app.bitrise.io/api/build/162f4065df9bd6b3/logs.json?&download=log

0 Likes