iOS tests failed randomly/exit status 65

Question & Answer Builds
1

On Bitrise platform our tests:

  • failed randomly
  • or exit with status 65

On our local machine, all the tests succeed:

  • in xcode
  • when launching the fastlane lane
  • with the bitrise cli

After reading some threads on github, we try to:

  • increase FASTLANE_XCODE_LIST_TIMEOUT
  • replace scan with xcodebuild and xctest

But it’s the same.

Any ideas why it’s not working ? :slight_smile:

2

Hi @jp.alary,

Without the logs hard to say :wink:

In general I’d suggest you to check this “local debugging” guide How to debug your build locally / “It works on my Mac/PC but not on bitrise.io” - specifically the Additional notes for iOS projects section (e.g. “Make sure that you run the tests in the same simulator as the one bitrise.io runs.” and “please make sure to reset the iOS Simulator”).

If that wouldn’t help probably the easiest way is if you could Screen Share / Remote Desktop into the build VM and run the tests there, and see/debug what the issue might be. Note, please do Reset the Simulator there too if you run more than one test, as that’s how standard bitrise.io builds work; every build runs in a completely clean VM, and the VM is destroyed at the end of the build, so e.g. the iOS Simulator is always clean.

Guide for how you can connect to the MacOS Build VMs: https://gist.github.com/viktorbenei/c6d4fe1e68de739dbb5f4f15de76b9db

Please let us know how it goes or if you’d have any questions! :slight_smile:

3

Hello @viktorbenei

Thanks for your reply. I’ve tested the two solutions you gave me and no better results. I’ve tried to debug on Bitrise’s VM but I have the same output as the one in web console: it’s not helping me.

Also, I don’t have logs to give you, it just says:
** TEST FAILED **
[08:57:01]: Exit status: 65

I think there is a crash when tests are executed but hard to find it without more informations. This is weird because we have this, only in Bitrise VM…

We spent too much time on iOS (and Android for almost the same problem) to debug things on Bitrise and we are close to stop using it. As we don’t have more time to investigate on this, we’ll see later how to resolve this problem.

Thanks for your help.

4

Can you please copy paste a related build’s bitrise.io URL? Without logs / context it’s incredibly hard to say anything more helpful :slight_smile:

Not weird at all, as our VMs are clean MacOS installs, pretty much the same state you get when you install MacOS on a brand new Mac / when you do a clean MacOS install, while your own MacOS is most likely heavily modified with additional tools / configs. Again, without logs I can’t really tell anything more helpful so please copy paste a build’s URL and we’ll check it ASAP! :slight_smile:

5

That’s exactly the point, to reproduce the issue in an interactive environment! :wink:

Simply remote desktop into the VM and open Xcode.app or anything you’re familiar with and debug the issue there :slight_smile:

6

Okay thanks for your answers.
This is the build: https://www.bitrise.io/build/0b0fb55f198d014c

Sorry I told you 0 failure:
Executed 937 tests, with 1 failure (0 unexpected) in 3.638 (4.439) seconds

But in the recap, there is 0 failure:
** TEST FAILED **
[03:15:33]: Exit status: 65
±-------------------±----+
| Test Results |
±-------------------±----+
| Number of tests | 935 |
| Number of failures | 0 |
±-------------------±----+

When I have time, I’ll try to debug in vnc and see with xcode, as you told me (I just tested in ssh previously)

1 Like
7

Thanks for the URL @jp.alary! :slight_smile:

Checked the log. Why fastlane reports different infos at Executed 937 tests, with 1 failure (0 unexpected) in 3.638 (4.439) seconds and | Number of failures | 0 | I can’t answer, most likely a fastlane bug - you might want to create a bug report on their issue tracker, or use our Xcode Test step instead.

But that’s probably less important. The error can be found in the “full” xcode build output log, which is saved by fastlane into a separate file (it’s not in the standard log), and our Deploy to Bitrise.io step attaches that file to your Build.

Please see the “Apps & Artifacts” section of the build’s page, you’ll find this file there: scan_ShaprTests-ShaprUnitTests.log
That’s the full xcode build output log.

In that you can see this exception:

2017-12-19 03:14:38.788864-0800 Shapr[4139:17776] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'API violation - multiple calls made to -[XCTestExpectation fulfill] for Output expectation.'

Without the code hard to say what causes it. Are you sure that you use the same Xcode version on Bitrise.io and on your own Mac?

If Xcode version would be the same, then I’d suggest you to:

  1. Remote desktop into the build VM (as described above iOS tests failed randomly/exit status 65 )
  2. Then open your project in Xcode.app (the code is at /Users/vagrant/git ; the git directory right in the home folder)
  3. Then run the tests from there.

You should be able to see the error in Xcode.app there, and debug what causes the issue.

If you’d have any questions just let us know! :slight_smile:

8

Thanks @viktorbenei

Indeed, with vnc access, I saw in xcode that some tests crashed due to:

API violation - multiple calls made to -[XCTestExpectation fulfill] for **** expectation.

But it’s still weird… On our local machine, with a clean, rm of DerivedData, we don’t have this crashes…

We’ll see what’s wrong when we’ll have more time, but, anyway, thanks for you help! :slight_smile:

1 Like
9

Any time - if you’d have any questions just let us know! :wink: