iOS Running tests in host application fails

testing
ios
#1

Bitrise Build Issue Report template

Description of the issue

I have some tests in a framework I’m developing related to the keychain.

In order for Keychain to be accessible in tests, the build that is running needs to have a certain entitlement (I think something like application-identifier, though I can’t find the reference for this currently).

The way to fix this is to set a host application for the test target, so it is running inside an application which has this entitlement.

I created a new blank application for the purpose of this, which contains nothing; it just exists to provide the entitlement to the test target.

However, this same solution isn’t working on bitrise.

I’ve made sure the scheme for the host application is shared.

But the tests are failing before the tests can be run, as the host application doesn’t seem to build:

...TestApp.app (3297) encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. Underlying error: Test runner exited before starting test execution.)

Environment:

Where did the issue happen?

The build is on the latest Xcode 10.0.x, on macOS 10.13 (High Sierra) stack

Which build Step causes the issue and which version of the step?

Xcode Test for iOS 1.18.4

Reproducibility

  • Does a “Rebuild” help? (You can trigger a rebuild from the Build’s page, by clicking the “Rebuild” button in the top right corner of a finished build) : NO
  • Does a rebuild without caches help? (You can remove the Cache:Pull and Cache:Push steps temporarily to not to use the cache, or you can delete all the caches on the Settings tab of the app. : NO - tried removing steps, but still failed
  • Does the issue happen sporadically, or every time? : Every time
  • Does upgrading the build Step to the latest version help? : NO
  • When did the issue start? : 20th September 2018

Local reproduction

Can it be reproduced on your own Mac/PC by following our local debug guide? Please follow at least the first section (“Testing with a full clean git clone”) to make sure to test the state of the code what bitrise.io will get when it does a git clone in the clean environment! If possible please note which sections you tried.

Attempted to reproduce locally with a full clean git clone, but could not reproduce.

Build log

Please copy paste the build’s bitrise.io URL here (or if the issue happens somewhere else then the full logs), or if you can’t share the url / log here then send the url or full log through a private channel (e.g. email - https://www.bitrise.io/contact ), with a link to the related Discuss issue.

#2

Hi @MattHallatt,
Thanks for reporting this issue! We are looking into it and will get back to you as soon as we find a solution to it!

#3

Hi @MattHallatt,
could you please set the Xcode Test step’s OutputTool input to: xcodebuild and run a new build? Does it solve the issue?

#4

Hi @godreikrisztian,

I’ve updated this field and tests are still failing at the same point.

Here’s a link to the build:

Matt

#5

Hi @MattHallatt,

thanks for the URL!

It seems you are using a Bitrise stack with Xcode 10, but in Xcode Test step you specified SimulatorOsVersion: 11.4.

Based on your log you should the ios version 12:

Testing started on 'iPhone 6s Plus'
...
        SimDevice: iPhone 6s Plus (3E469B12-0BE7-48DC-B07D-23D8356D4BB3, iOS 11.4, Booted)
} (11.4 (15F79))
dyld: program was built for a platform that is not supported by this runtime
....
        SimDevice: iPhone 6s Plus (3E469B12-0BE7-48DC-B07D-23D8356D4BB3, iOS 11.4, Booted)
#6

Hi @godreikrisztian,

I’m specifying 11.4 as when I last ran the tests on an iOS 12 simulator, there were issues there that stopped the tests from passing. I believe this was related to a timing problem with starting the simulator up, as I could only recreate it locally if I was running a completely fresh iOS 12 simulator. All was fine with a completely fresh iOS 11.4 simulator locally. Switching Bitrise builds to iOS 11.4 previously fixed this problem.

Xcode 10 is completely capable of building and running iOS 11.4 applications, so I don’t see why this would be a problem?

I’ve tried another build specifying latest as the OS version in the build step, but that still fails.

Matt

#7

It is worth noting that now the Host Application tests seem to pass (the Keychain related tests)!

The reason for the failure is back to what made me switch back to iOS 11.4, and that’s some tests failing on Bitrise that pass locally, that I think are related to timing problems with launching new iOS 12 simulators, though I could be wrong.

#8

Hi @MattHallatt
For now i can not reproduce the issue.
May i ask you to enable bitrise support for your project? I would create a workflow called: support, where i would run the xcode-test step with increased boot wait time.

#9

Hi @godreikrisztian,

Sorry, I completely forgot about your request!

I’ve set up a support workflow, and have pushed a branch called feature/bitrise-failing-tests which contains the tests that pass locally but fail here on Bitrise.

I’ve enabled support too.

Please let me know if you need anything further from me.

#10

I’m currently experiencing the similar issue.

Here’s the link to the build:

#11

Hi @MattHallatt,

do you use the new build system locally?

I have:

  • set Output tool to xcodebuild
  • disable the new build system (setting Additional options forxcodebuild build testcall to: -UseModernBuildSystem=NO )

and started a new build against feature/bitrise-failing-tests and it succed: https://app.bitrise.io/build/71a55c11592109de

#12

@LeeroyDing could you please also try to disable the new build system and use xcodebuild as Output tool for the Xcode Test step?

#13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.