How to run Android UI tests on virtual devices

test
build-cache
android
ui
#33

How is that possible @tamaspapik? The virtual device/ emulator only lives within the time the Virtual Device Testing step right?
I’ve already tried having Gradle Runner task right after Virtual Device Testing but I got java.lang.RuntimeException: No device(s) found. e.g. https://www.bitrise.io/build/e48b280ad5781812

0 Likes

#34

I believe that @eric-grab asked about Gradle task used to perform the tests like connectedCheck or spoon NOT about one generating APKs (like assemble).

In case of Firebase Test Lab devices are not connected to local ADB server. APKs (testing and tested) need to be uploaded and then you (or step) can execute a command which starts the tests: https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run

0 Likes

#35

Hi @koral, are you saying that behind the scene, Virtual Device Testing step actually calls Firebase Test Lab?

Regardless, I believe I’m very close to achieving my goals, please help me a just a bit more koral and @tamaspapik :smiley:
Within the life span of Virtual Device Testing step, how can I trigger ./gradlew someTaskIwant?

0 Likes

#36

Exactly:

The tests run on Firebase Test Lab and we display the results we get from them.

Source:

Current version of Virtual Device Testing step waits until tests are finished by polling every 5 seconds:

It should be possible to split this step into separate “Upload APKs and start tests on Virtual devices” and “Retrieve Virtual devices test results”. Between them you can insert whatever step you want.

0 Likes

#37

Thanks @koral

Reading main.go and Firebase Test Lab documentation, it’s unclear to me whether you guys are calling a REST API (not found in their documentation) or Gcloud CLI (doesn’t seem to support custom Gradle task).
Can’t see myself being able to have both Virtual Device Testing + Spoon working together any time soon.
Will put this off for a while.
Thanks a lot for your help thus far @koral and @tamaspapik :slight_smile:

0 Likes

#38

Firebase REST API doesn’t support stuff like that, we have our own server to handle requests, so this is why you won’t see our calls in the official documentation :slight_smile: Thanks @koral!

So basically VDT uses an “offline” solution to test, which means that it prepares/uploads all stuff required for the test, and then runs the given configuration. not like you have a device connected that can be used from adb. I think this is why you won’t be able to use VDT for this case

2 Likes

#39

Checkout https://github.com/TestArmada/flank for a custom FTL runner that works on bitrise. Spoon is not FTL compatible.

1 Like

#40

This is great. Been struggling with the Create Emulator, Start Emulator, etc. until I noticed this step.

Is it supported to generate jacoco report from the instrumental tests?

0 Likes

Report coverage Android vitual device testing
#41

Glad to hear that! :slightly_smiling_face:

Does your jacoco report is generated to the device’s sdcard or so? In that case you can fill the pull directories input of the step, so it will be exported to the test artifacts.

1 Like

#42

Thanks for replying.

Here’s my current settings:

Environment Variables:

coverage=true
coverageFile=/sdcard/coverage.instrumentaltests

Directories to pull:

/sdcard

And I can see the coverage file downloaded, found and uploaded by the codecov script. But in codecov.io the coverage on the relevant classes are still zero. Could you suggest where should I start investigating?

1 Like

#43

If you’re using android test orchestrator, there’s a known bug that breaks coverage generation. Otherwise it should work.

Make sure your build.gradle is configured correctly for jacoco coverage and using the latest versions.

1 Like

#44

After some further investigation I found that codecov can’t process the coverage.ec generated by instrument tests. When I turn it into an xml report, codecov can update the coverage correctly.

But instrument tests only seems to generate the .ec file, not the .em, so I can’t convert with ‘emma report’. Well, anyways, it’s beyond bitrise’s power. I appreciate your great work.

0 Likes

#45

Use jacoco, not emma. Jacoco can output different types of reports.

2 Likes

#46

OK, I got a report with java -jar jacococli.jar report, finally! Thank you so much for your help!

3 Likes

#47

Thanks for the tutorial.

I have about 5 different modules all containing instrumentation tests. How can I configure this with bitrise? Wouldn’t I need to specifiy 5 different apks for each module?

0 Likes

#48

@PaulWoitaschek right now, with our built in Virtual Device Testing feature you can only run instrumentation tests for a single APK per build.

Feel free to create a #feature-request, or register the apps as separate apps on bitrise.io (you can register the same repo multiple times), or use Firebase TestLab directly via scripts (How-to run Android tests on Firebase TestLab).

If you’d have any questions just let us know! :wink:

0 Likes

#49

Hi, I have a project which has multiple modules containing the UI tests. I want to test them all. However, when I install the apk generated by assembleDebugAndroidTest and run virtual device test, I run tests from the app module only. Can you help me with this?

0 Likes

#50

Hi @kevalpatel2106,

Does the build generate multiple APKs? If it does, then please see my reply above :point_up_2: How to run Android UI tests on virtual devices

If not, then please create an issue report with the build’s URL, so that we can look into it (https://discuss.bitrise.io/c/issues/build-issues)

0 Likes

#51

My tests are failing on virtual device as the location is not enabled. All my tests uses espresso api’s and to over come this I am running UI automator calls to enable the location whenever the location dialog is reported. But no luck. Is there anyway to get rid of this. ? Thanks in advance

0 Likes

#52

Hello @ns.mesta!
Could you please write to us on the on-site chat, send a build url and please enable support user on the app?

0 Likes