I am trying to get my tests to run in Bitrise. My app is an Xcode-based Swift CLI app, which means if I create a new unit testing bundle target, I cannot select the existing CLI executable as the “target to be tested”. Instead, I have to create the testing bundle with no target, and manually add existing files containing code to be tested as members of the testing bundle target. (This is described here.)
When attempting to use the xcode-test-mac step, it tries to run xcodebuild as follows:
xcodebuild: error: Failed to build project tesspost with scheme TessPostTests.
Reason: The scheme 'TessPostTests' is not configured for Running.
Recovery suggestion: The scheme 'TessPostTests' has nothing configured to build for Running and has no executable specified to Run. Edit the scheme to configure the Run action.
I think this is expected, and the issue is that xcodebuild is being told to do a "build" action instead of just a "test". When I remove "build" and run xcodebuild locally, it works. Inside of Xcode, there is no issue with running tests, presumably because the target is being Built for Testing and not Built for Running. What can I do to get bitrise to just do tests for this scheme?
Environment:
Bitrise.io: Xcode 11.3.x, on macOS 10.14.6 (Mojave)
Step xcode-test-mac@1.4.0
Reproducibility
Does a “Rebuild” help? : NO
Does a rebuild without caches help? : NO
Does the issue happen sporadically, or every time? : Every time
Does upgrading the build Step to the latest version help? : NO
I’m trying to dig a bit deeper into my options, and am looking at the xcodebuild step instead of Xcode Test for Mac. It doesn’t look very helpful: for one, the xcodebuild step always succeeds, even if there are test failures or other failures. It also seems to be only a partial representation of xcodebuild’s available options, in that it requires a workspace instead of allowing the use of projects, and only provides options for the iphonesimulator.
Continuing to dig further, I’m now using a customized version of xcode-test-mac that runs the “build-for-testing” action instead of “build”. It now runs locally in Bitrise CLI, but still fails in the cloud environment.
The same step now fails with:
error: No signing certificate "Mac Development" found: No "Mac Development" signing certificate matching team ID "66UTZ62RWH" with a private key was found. (in target 'TessPostTests' from project 'tesspost')
The default certificate-and-profile-installer step is running successfully prior to this, but it seems to be very iOS-centric, so maybe that’s the problem. I don’t know.
Blindly copying from the xcode-analyze step, I added "CODE_SIGNING_ALLOWED=NO" as an additional option to xcodebuild for the test step, and this seems to have avoided this error, but now there’s a brand new one:
xctest (1951) encountered an error (Failed to load the test bundle. (Underlying error: The bundle “TessPostTests” couldn’t be loaded because it is damaged or missing necessary resources. The bundle is damaged or missing necessary resources. dlopen_preflight(/Users/vagrant/Library/Developer/Xcode/DerivedData/tesspost-asuohjcfyahndgfzajztczibqoxv/Build/Products/Debug/TessPostTests.xctest/Contents/MacOS/TessPostTests): Library not loaded: /System/Library/Frameworks/PDFKit.framework/Versions/A/PDFKit
Referenced from: /Users/vagrant/Library/Developer/Xcode/DerivedData/tesspost-asuohjcfyahndgfzajztczibqoxv/Build/Products/Debug/TessPostTests.xctest/Contents/MacOS/TessPostTests
Reason: image not found))
My hunch is that this has something to do with the import PDFKit (from the macOS SDK) that I’m doing in my test target. Really not sure where to go from here…
I finally figured out how to scroll all the way to the right end of that error line and did some more digging. I was surprised to see that on the runner VM, /System/Library/Frameworks/PDFKit.framework does not exist! On my machine, which admittedly is running Catalina, it does. I’m now trying to compare with a typical 10.14 system to figure out if that’s an abnormality with the bitrise runner, or if it’s something else. In the meantime, I can get my build to work by first symlinking the missing framework from inside Quartz.framework.
I figured out that the Bitrise runner has the framework in the usual place on 10.14, so I think I now have this build running as well as can be expected until the environment gets upgraded to Catalina.
I created a PR for my tweak to xcode-test-mac. I think there are also a lot of improvements to be made to the xcodebuild step, but this one isn’t bitrise-provided (perhaps it should be, though).