Tests involving keychain access in iOS

[david.everlof.knowit]

I have tests that interact with the keychain in iOS, which fails on Bitrise and succeeds locally.

I’m using the simulator when testing locally.

It’s only these tests that fail, that was recently added (note, it’s the unit-tests for library so I’ve added a simple hosting application for it to run locally, that thats a part of the projects, so…)

Any ideas?

[viktorbenei]

fails on Bitrise and succeeds locally.

Can you please follow this guide and let us know if that helps:

How to debug your build locally / "It works on my Mac/PC but not on bitrise.io"

If the build works on your local Mac/PC but not on bitrise.io, that’s usually because of uncommitted / gitignored files or a tool version difference. There can be other reasons, for example in case of iOS code signing errors it’s usually because a code signing file which is available on your Mac, but you did not upload it to bitrise.io and so it’s not available there. In any case, the first steps you should try, preferably before you’d report the issue (as this will be the first thing we’ll ask…

Also, how do you run your tests? With our Xcode Test step?
What’s the error?

[david.everlof.knowit]

️ Can build from clean clone

️ Can build locally with bitrise cli

️ Testing using Xcode: Unit Test step: 1.19.0

The error that Im seeing is simply that my unit-tests fails (the ones interacting with the keychaing)

[viktorbenei]

@david.everlof.knowit

The error that Im seeing is simply that my unit-tests fails (the ones interacting with the keychaing)

Anything else in the logs, why it fails?

Please make sure that you have the Deploy to Bitrise.io step after the Xcode Test step in the workflow, that should attach the full/raw, unfiltered Xcode log to the build (top of the build page, Apps & Artifacts).

[viktorbenei]

Another thing: did you try to Reset the iOS Simulator on your Mac and then running the tests again? It’s quite a common source of these “works locally but not on bitrise.io” issues that the app behaves a bit differently at the first run, e.g. it presents a popup which only have to be accepted once (at the first run, then it’s stored in user defaults).

[david.everlof.knowit]

• I don’t deploy to bitrise.io since it’s a library
• The tests success locally even after resetting the simulator

More exact error:

/Users/vagrant/git/../..Tests.swift:11: error: -[..Tests.KeychainTests testErase] : XCTAssertTrue failed - 
Nothing was retrieved from the keychain. Status: The operation couldn’t be completed. (swift.keychain.error error -25300.): -25300

[viktorbenei]

@david.everlof.knowit

I don’t deploy to bitrise.io since it’s a library

The Deploy to Bitrise.io step can be used for attaching any artifact to the build, most of our steps share additional / debug logs which can help with debugging issues, and are stored in the $BITRISE_DEPLOY_DIR which is the default dir deployed by Deploy to Bitrise.io.

@david.everlof.knowit

The tests success locally even after resetting the simulator

Hmm… That’s strange. Did you run the tests in the same iOS Simulator / iOS version?

@david.everlof.knowit

swift.keychain.error error -25300

It seems the error code 25300 is simply that the item does not exist in the keychain. A possible debug idea:

https://forums.developer.apple.com/thread/11129

-25300 is errSecItemNotFound, which is a pretty standard keychain error. To debug this, my recommendation is that, when it occurs, you dump out the contents of your keychain to verify that things are in the state that you expect them to be in.

You can find more details/tips at the link.