How to run Android UI tests on virtual devices

How To Android
, , ,
1

The new feature, Virtual Device Testing for Android is here!
It is currently in private-beta test phase, please ping us if you want to join! One test can be ran in one build at the moment, and robo test type has some limitations of watching the screenshots and the activity-map.

Let’s see an example flow for an espresso instruments test. I’ve used this sample project:

Add your app, let the scanner setup your workflow

It will create the default workflows, what we will need is the primary workflow. Originally this workflow will build a debug apk.

App settings

Turn on Virtual Device Testing on the app’s settings tab.

How to change your primary workflow

  • Add an extra task assembleDebugAndroidTest in your gradle-runner’s gradle task input, so it will be something like: assembleDebug assembleDebugAndroidTest.
  • Add Virtual Device testing for Android step after your gradle-runner step.
  • As a minimum, in Virtual Device testing for Android step set Test APK path inputs to your built debug apk, in my case Test APK path will be: ./app/build/outputs/apk/app-debug-androidTest-unaligned.apk Take care of configuring only one section of the step, which matches your desired test type!

(APK path is exported by gradle-runner step, and it is set as the default for the step’s input, if you want to use different APK then don’t forget to change the step’s input accordingly!)

If you’ve done everything well, this is what you’ll see in your log:

27 PM

The available devices and its versions:

     ┌─────────────┬──────────┬────────────────────┬─────────────┬────────────────┬
     │   MODEL_ID  │   MAKE   │     MODEL_NAME     │  RESOLUTION │ OS_VERSION_IDS │
     ├─────────────┼──────────┼────────────────────┼─────────────┼────────────────┼
     │ Nexus4      │ LG       │ Nexus 4            │ 1280 x 768  │ 19,21,22       │
     │ Nexus10     │ Samsung  │ Nexus 10           │ 2560 x 1600 │ 19,21,22       │
     │ Nexus5      │ LG       │ Nexus 5            │ 1920 x 1080 │ 19,21,22,23    │
     │ Nexus5X     │ LG       │ Nexus 5X           │ 1920 x 1080 │ 23,24,25,26    │
     │ Nexus6      │ Motorola │ Nexus 6            │ 2560 x 1440 │ 21,22,23,24,25 │
     │ Nexus6P     │ Google   │ Nexus 6P           │ 2560 x 1440 │ 23,24,25,26    │
     │ Nexus7      │ ASUS     │ Nexus 7 (2012)     │ 1280 x 800  │ 19,21,22       │
     │ Nexus9      │ HTC      │ Nexus 9            │ 2048 x 1536 │ 21,22,23,24,25 │
     │ NexusLowRes │ Generic  │ Low-res MDPI phone │  640 x 360  │ 23,24,25,26    │
     └─────────────┴──────────┴────────────────────┴─────────────┴────────────────┴
5 Likes
2

I tried it, it’s awesome @tamaspapik!! :tada: :wink:

1 Like
3

Can you also share your full .yml script? I am wondering how you setup the output variables for consecutive gradle runners? The second(last) gradle runner output(debug-androidTest.apk) variable overridden on first gradle runner(debug.apk) output so UI test runner is not being able to see the first gradle runner output(debug.apk) and fails.

How can I solve this issue? @viktorbenei @tamaspapik

1 Like
4

Hey @burak.malkoc!

Thank you for asking! Please see my test workflow below for this case. :slight_smile:

---
format_version: '3'
default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git
project_type: android
trigger_map:
- push_branch: "*"
  workflow: primary
- pull_request_source_branch: "*"
  workflow: primary
app:
  envs:
  - opts:
      is_expand: false
    GRADLE_BUILD_FILE_PATH: build.gradle
  - opts:
      is_expand: false
    GRADLEW_PATH: "./gradlew"
workflows:
  primary:
    steps:
    - activate-ssh-key@3.1.1:
        run_if: '{{getenv "SSH_RSA_PRIVATE_KEY" | ne ""}}'
    - git-clone@3.5.2: {}
    - cache-pull@2.0.0: {}
    - install-missing-android-tools@2.0.2: {}
    - gradle-runner@1.7.7:
        inputs:
        - gradle_file: "$GRADLE_BUILD_FILE_PATH"
        - gradle_task: assembleDebugAndroidTest
        - gradlew_path: "$GRADLEW_PATH"
        - apk_file_exclude_filter: 
    - script@1.1.4:
        title: Store Test APK path with envman
        inputs:
        - content: |-
            #!/bin/bash
            envman add --key "BITRISE_TEST_APK" --value "$BITRISE_APK_PATH"
    - gradle-runner@1.7.7:
        inputs:
        - gradle_file: "$GRADLE_BUILD_FILE_PATH"
        - gradle_task: assembleDebug
        - gradlew_path: "$GRADLEW_PATH"
    - cache-push@2.0.1: {}
    - virtual-device-testing-for-android@0.9.7:
        inputs:
        - test_type: instrumentation
        - test_apk_path: "$BITRISE_TEST_APK"
2 Likes
5

@tamaspapik I’m thinking how to config for appium test for Android (maven). How exactly to config in my case?

1 Like
6

Seems like there is no option to run appium test with an adb command like adb shell am instrument -w <test_package_name>/<runner_class> so I think it is not possible at the moment.

1 Like
7

@bootstraponline do you have a tutorial for running Appium on Firebase? Or is that indeed not possible?

8

Firebase test lab does not currently support appium. It’s not possible.

1 Like
9

Thanks for confirming @bootstraponline! :slight_smile:

10

Thanks for the answer, so I need back to work with Emulator which always hang at the install apk step :disappointed_relieved:

11

You could try Sauce Labs for emulators which support appium.

2 Likes
12

Hi, first off, awesome job! Thank you.

I’m worrying about the time it says it took the whole testing step process, the log says the Virtual step took about 20 minutes, but the UI tests video shows just 12 minutes (including some initial delays doing nothing), so I was wondering what’s the quota time the system is going to take into account, regarding the build time consumption?

1 Like
13

Hi @kuassivi,

There’s no special quota, it can run as long as your subscription’s build timeout limit.

Indeed, especially for popular devices, Firebase Test Lab (the service we use right now under the hood for android device testing feature, as FBTL was the most reliable solution in our tests) might take a really long time to boot the emulator. Depends on their available resources. Initially my regular tests (scheduled to run every 30 mins) which are set to use NexusLowRes,24,en,portrait took about 4 mins to boot and run my minimal test suite, but lately it usually takes more than 8 mins to do the same (basically just boot the emulator, as my tests really are minimal). Unfortunately we don’t have control over FBTL, but we’re still searching to see if we can find a service which is at least as reliable as FTBL but can run the tests faster (boot faster).

One thing you might want to try is our own Emulator steps with x86 emulators (available on our Linux/Android stack right now). Not the same as FTBL but significantly faster than the old ARM emulators, and quite stable too. Related blog post: https://blog.bitrise.io/introducing-avd-manager

If you have any questions just let us know! :slight_smile:

1 Like
14

Hi, I’ve been trying many different configurations with the x86 emulators, but unfortunately all my test are failing now.
Getting messages like:

- RuntimeException: Waited for the root of the view hierarchy to have window focus and not request layout for 10 seconds.

I applied all techniques I found on my research but I can’t solve it.
I don’t even know how or where to pass the test-result and artifact to see them on the build tab.

Any help?

15

Hi @kuassivi ,

Can you please create an issue report in #issues:build-issues (http://discuss.bitrise.io/c/issues/build-issues) ? It’d help a lot (especially the log / build URL), so that we can look into it ASAP! :slight_smile:

16

Sure, I’ll do it in a moment. Thanks.
Anyway, about the env variable to use to pass the test-results, Am I right with the assumption that I need to do it manually (with a script), or Is it automatically collected by Bitrise after the Gradle Step somehow?

17

Depends, if you’re referring to output files of a tool (other than the base / standard ones, like APK in case of Gradle Runner) you might have to specify where the file(s) are located or move them into the $BITRISE_DEPLOY_DIR. Related docs: Build artifacts online - Bitrise Docs

18

I refer to the test-result files/folder generated after the connectedAndroidTest task. Do I need to move them manually to the $BITRISE_DEPLOY_DIR?

19

Yes, you most likely have to.

In general I’d suggest you to try the CLI locally on your Mac/Linux (https://www.bitrise.io/cli / How to experiment with Bitrise configs locally, on your Mac/Linux), that way you can check what’s moved automatically into the $BITRISE_DEPLOY_DIR and what isn’t.

The build log in general should also include the files which are moved into the deploy dir; e.g. the Gradle Runner prints every file at the bottom of it’s “step log box” which are moved into the deploy dir.

If you’d have any questions just let us know! :slight_smile:

1 Like
20

Hi @viktorbenei, all the links you provided were very helpful, thanks.

It’s working properly now, I don’t see the window-focus=false issue anymore. And it’s indeed saving build time, and probably more with a proper cache!

Only to remark that the Deploy Step doesn’t work from the local CLI.

1 Like