In our project we have implemented snapshot tests for some of our screens.
Although the tests are green locally they actually fails when running on Bitrise.
I have ran the tests locally by reset-ing the simulator data and Xcode derived data but unfortunately it did not make a difference.
As it would be very helpful to check what are the failure diffs images when running on Bitrise, is there a way to access the data generated from a failed build instantiation?
Hi @iamfabiomilano,
If you use our Xcode Test step to run the tests it does have an option to export all the ui test screenshots generated by Xcode.
Otherwise, regardless of which tool to use, you can use the Deploy to Bitrise.io step to attach artifacts to the build. Related guide: Build artifacts online - Bitrise Docs
If you have any questions just let us know!
If you use our Xcode Test step to run the tests it does have an option to export all the ui test screenshots generated by Xcode.
We are using FBSnapShotTestCase for this. The command that you mention seems to be related to the new Xcode 9 feature of taking snapshots from UI tests.
Otherwise, regardless of which tool to use, you can use the Deploy to Bitrise.io step to attach artifacts to the build. Related guide: http://devcenter.bitrise.io/tips-and-tricks/attach-any-file-to-build/1
Thank you I’ll try that one and see what the artifacts can tell me.
I think even Xcode 7 did take screenshots during UI tests, but Xcode 8s definitely do. But indeed that’s a different thing, as far as I know FBSnapShotTestCase does not use the Xcode built in UI test screenshots.
That said, it doesn’t really matter which tool you use if that runs as part of the Xcode Test. Our Xcode Test step runs the same Xcode command which is performed when you click “Test” in Xcode.app, just runs it through the Xcode command line tools (xcodebuild ... test ...).
So, as far as your test tool runs as part of the Xcode test session the Xcode Test step should run it too. The only question is where that tool saves the screenshots, the Xcode Test step has built in support for the Xcode built in UI test screenshots only, for other tools you’ll have to move the screenshots into the $BITRISE_DEPLOY_DIR or add a new Deploy to Bitrise.io step to the workflow and specify the path where those screenshots are stored (the default path is $BITRISE_DEPLOY_DIR but of course you can change that to any path you want to).
Thank you for your help.
I found out the issue was that locally I was testing by running my Unit tests (therefore FBSnapshotTest cases) with an iPhone SE while on CI the iPhone 6s plus was used. This made the generation of the reference images different, with a Plus also @3x images are created, even if the option of the test cases where device agnostic.
I have used the Deploy to Bitrise.io step but if I replace current:
- deploy-to-bitrise-io@1.3.6: {}
with the version that takes the FailureDiffs path
- deploy-to-bitrise-io@1.3.6: inputs: - deploy_path: "${BITRISE_SOURCE_DIR}/Tests/FailureDiffs" - is_compress: 'true'
I miss the artifacts about the nice html output for my tests and the xcodebuild logs.
As every CI run can fail for many reason, if I keep the deploy-to-bitrise-io step only for FailureDiffs it will fail to deploy in case the failure was not related to snapshots failures (because the folder to deploy is not there) and I’d like to have xcodebuild logs instead.
Is it possible to deploy different artifacts at the same time, one for failure diffs and one for xcodebuild logs?
If yes, how can I use run-if option to not run the bitrise-deploy step if there is no FailureDiifs folder?
Never mind I just figured that I can use env variable to set my failure tests to $BITRISE_DEPLOY_DIR. Thank you!
Just one note, re:
An alternative solution would be to move/copy the files you want to attach as artifact into a single directory, e.g. $BITRISE_DEPLOY_DIR, then you can deploy all files with a single Deploy to Bitrise.io step.
