How to do Calabash UITesting on Bitrise

How To
,
1

This guide introduces automated Calabash UITesting for Xamarin projects on Bitrise.

Project setup

The sample project, i will use in this guide is our xamarin-cross-platform sample.
It contains iOS and Android projects, both with Calabash test.

The test are placed into the project directories:

  • for ios: REPO_ROOT/iOS/features
  • for android: REPO_ROOT/Droid/fetaures

and they are default generated/sample tests, read: Calabash Quickstart for Xamarin.iOS and Creating A Calabash Test for an APK guides.

The testing dependencies are managed via bundler:

PROJECT_ROOT_DIR/iOS/Gemfile:


source 'https://rubygems.org'
gem 'calabash-cucumber', '0.20.3'
gem 'xamarin-test-cloud', '2.1.0'

NOTE:
calabash-cucumber gem version should match to your Xamarin.TestCloud.Agent version, in my case: 0.20.3.

PROJECT_ROOT_DIR/Droid/Gemfile:

source 'https://rubygems.org'
gem 'calabash-android', '0.9.0'
gem 'xamarin-test-cloud', '2.1.0'

Since i want to run both calabash test in the same workflow and i want to build the solution only once, i created a new solution configuration (calabash) which maps to the desired project configuration:

  • in the iOS project, it maps to Debug configuration (Debug build will start the calabash test server)
  • in the Android project it maps to Release configuration, since Shared Mono Runtime is enabled for my Debug config

Screen Shot 2017-01-31 at 15.27.12.png961Ă—698 46.2 KB

Setup on Bitrise

If you have not yet added your app to bitrise go to the Add new App page and initialize your project on bitrise.

Once the VALIDATING REPOSITORY step finishes, select:

  • Xamarin solution configuration: calabash
  • Xamarin solution platform: Any CPU

Screen Shot 2017-01-31 at 13.57.51.png753Ă—296 17.4 KB

If your project uses Xamarin Component dependencies, do not forget to fill the Xamarin login form.

The generated workflow should look like:

Screen Shot 2017-01-31 at 14.02.37.png318Ă—872 26.4 KB

Setup Calabash Testing for iOS

The only thing you need to do is: add the Calabash iOS UI test step after the Xamarin Archive step.

Lets configure the newly added Calabash iOS UI test step:

  • Directory of your calabash features input needs to be filled: in our sample the features folder is located at: PROJECT_ROOT_DIR/iOS/features/, so the correct value is: ./iOS
  • The gem file is placed in the iOS directory as well, so we can leave Gemfile path input on default value ($work_dir/Gemfile).
  • Path to the iOS .app file to test input default value is: $BITRISE_APP_PATH, which environment will be exported by Xamarin Archive step.
  • Device on iPhone 6 and latest OS version is ok for now.
  • Since we manages Calabash gems with bundler we do not need to set calabash-cucumber gem version input.

WARNING:
The used .app file should support the selected simulator’s architecture. You can ensure Supported architectures in your project’s Project Options/Code Generation and Runtime/Supported architectures:

  • If i386 architecture is selected, simulator device should be a 32-bit device.
  • If x86_64 architecture is selected, simulator device should be a 64-bit device.
  • If i386 + x86_64 architecture is selected, simulator can be both 32-bit and 64-bit device.

If you app’s target architecture is i386 and you try to run the app on 64-bit simultaor (like iPhone 6), once the app started a popupp will apprear: [App Name] may slow down your iPhone”, “The developer of this app needs to update it to improve its compatibility. It may influence your UI tests.

That’s it, the iOS part of our solution is ready for Calabash testing on Bitrise.

Setup Calabash Testing for Android

To run Calabash Android UI test you will need a running Android emulator to run the test against.

Add Create Android emulator step to your workflow, configure it. After Create Android emulator step add Start Android emulator step to run the created emulator.

Now add Calabash Android UI test step after Start Android emulator step.

Lets configure the newly added Calabash Android UI test step:

  • Directory of your calabash features input needs to be filled: in our sample the features folder is located at: PROJECT_ROOT_DIR/Droid/features/, so the correct value is: ./Droid
  • The gem file is placed in the Droid directory as well, so we can leave Gemfile path input on default value ($work_dir/Gemfile).
  • APK path input default value is: $BITRISE_APK_PATH, which environment will be exported by Xamarin Archive step.
  • Android Home Directory is set to $ANDROID_HOME.
  • Since we manages Calabash gems with bundler we do not need to set calabash-cucumber gem version input.

That’s it, the Android part of our solution is ready for Calabash testing on Bitrise.

Example build configuration

You can find a full example build configuration in the sample app’s repository, at: sample-apps-xamarin-cross-platform/bitrise.yml at master · bitrise-io/sample-apps-xamarin-cross-platform · GitHub

2 Likes
3

A status update based on a discussion with one of our users:

Calabash seems to be deprecated at the moment.

At this time our recommendation is to use another testing framework if possible. It seems that the Calabash project was abandoned by their original maintainers (Xamarin) and was transferred to new maintainers, so in the future the situation might change.