How to run Android UI tests on virtual devices

Hi @koral, are you saying that behind the scene, Virtual Device Testing step actually calls Firebase Test Lab?

Regardless, I believe I’m very close to achieving my goals, please help me a just a bit more koral and @tamaspapik :smiley:
Within the life span of Virtual Device Testing step, how can I trigger ./gradlew someTaskIwant?

1 Like

Exactly:

The tests run on Firebase Test Lab and we display the results we get from them.

Source:

Current version of Virtual Device Testing step waits until tests are finished by polling every 5 seconds:

It should be possible to split this step into separate “Upload APKs and start tests on Virtual devices” and “Retrieve Virtual devices test results”. Between them you can insert whatever step you want.

Thanks @koral

Reading main.go and Firebase Test Lab documentation, it’s unclear to me whether you guys are calling a REST API (not found in their documentation) or Gcloud CLI (doesn’t seem to support custom Gradle task).
Can’t see myself being able to have both Virtual Device Testing + Spoon working together any time soon.
Will put this off for a while.
Thanks a lot for your help thus far @koral and @tamaspapik :slight_smile:

Firebase REST API doesn’t support stuff like that, we have our own server to handle requests, so this is why you won’t see our calls in the official documentation :slight_smile: Thanks @koral!

So basically VDT uses an “offline” solution to test, which means that it prepares/uploads all stuff required for the test, and then runs the given configuration. not like you have a device connected that can be used from adb. I think this is why you won’t be able to use VDT for this case

2 Likes

Checkout https://github.com/TestArmada/flank for a custom FTL runner that works on bitrise. Spoon is not FTL compatible.

1 Like

This is great. Been struggling with the Create Emulator, Start Emulator, etc. until I noticed this step.

Is it supported to generate jacoco report from the instrumental tests?

Glad to hear that! :slightly_smiling_face:

Does your jacoco report is generated to the device’s sdcard or so? In that case you can fill the pull directories input of the step, so it will be exported to the test artifacts.

1 Like

Thanks for replying.

Here’s my current settings:

Environment Variables:

coverage=true
coverageFile=/sdcard/coverage.instrumentaltests

Directories to pull:

/sdcard

And I can see the coverage file downloaded, found and uploaded by the codecov script. But in codecov.io the coverage on the relevant classes are still zero. Could you suggest where should I start investigating?

1 Like

If you’re using android test orchestrator, there’s a known bug that breaks coverage generation. Otherwise it should work.

Make sure your build.gradle is configured correctly for jacoco coverage and using the latest versions.

1 Like

After some further investigation I found that codecov can’t process the coverage.ec generated by instrument tests. When I turn it into an xml report, codecov can update the coverage correctly.

But instrument tests only seems to generate the .ec file, not the .em, so I can’t convert with ‘emma report’. Well, anyways, it’s beyond bitrise’s power. I appreciate your great work.

Use jacoco, not emma. Jacoco can output different types of reports.

2 Likes

OK, I got a report with java -jar jacococli.jar report, finally! Thank you so much for your help!

3 Likes

Thanks for the tutorial.

I have about 5 different modules all containing instrumentation tests. How can I configure this with bitrise? Wouldn’t I need to specifiy 5 different apks for each module?

@PaulWoitaschek right now, with our built in Virtual Device Testing feature you can only run instrumentation tests for a single APK per build.

Feel free to create a #feature-request, or register the apps as separate apps on bitrise.io (you can register the same repo multiple times), or use Firebase TestLab directly via scripts (How-to run Android tests on Firebase TestLab).

If you’d have any questions just let us know! :wink:

Hi, I have a project which has multiple modules containing the UI tests. I want to test them all. However, when I install the apk generated by assembleDebugAndroidTest and run virtual device test, I run tests from the app module only. Can you help me with this?

Hi @kevalpatel2106,

Does the build generate multiple APKs? If it does, then please see my reply above :point_up_2: How to run Android UI tests on virtual devices

If not, then please create an issue report with the build’s URL, so that we can look into it (http://discuss.bitrise.io/c/issues/build-issues)

My tests are failing on virtual device as the location is not enabled. All my tests uses espresso api’s and to over come this I am running UI automator calls to enable the location whenever the location dialog is reported. But no luck. Is there anyway to get rid of this. ? Thanks in advance

Hello @ns.mesta!
Could you please write to us on the on-site chat, send a build url and please enable support user on the app?

Hi, I am trying to run this step but it is failing and it is not possible to determine the cause, the following is the only output I get:
Uploading app and test files
App path (/bitrise/deploy/GithubBrowserSample-app-debug.apk), is bundle: false
Assets requested: {isBundle:false testApp: Apk:{UploadURL: GcsPath: Filename:GithubBrowserSample-app-debug.apk} Aab:{UploadURL: GcsPath: Filename:} TestApk:{UploadURL: GcsPath: Filename:GithubBrowserSample-app-debug-androidTest.apk} RoboScript:{UploadURL: GcsPath: Filename:} ObbFiles:[]}
Uploading file(/bitrise/deploy/GithubBrowserSample-app-debug.apk) to (gs://bitrise-addon-ftl-prod-bucket/android-tests/1377cbd8db50543a/app.apk)
=> Files uploaded
Starting test
AndroidInstrumentationTest: &{AppApk:0xc0000ccbc0 AppBundle: AppPackageId: OrchestratorOption:DO_NOT_USE_ORCHESTRATOR TestApk:0xc0000ccc00 TestPackageId: TestRunnerClass: TestTargets:[] ForceSendFields:[] NullFields:[]}
=> Test started
Waiting for test results

  • Validating
  • (1/1) running
  • (0/1) running
    => Test finished
    Test results:
    Model API Level Locale Orientation Outcome
    NexusLowRes 24 en portrait failure

Hi,
I am running Android Instrumentation Unit tests on FTL and I could not find any option to disable video recording & performance monitoring.
I do not want to initialise the gCloud sdk using a shell script, rather I would like to go ahead with the available plugin. Is there any option which I can use to pass following options for my case, since it is of no use for me while running Instrumented Unit tests.
--no-record-video --no-performance-metrics
As of now, i found that the plugin records both of these by default (as these are enabled by default). An option to pass such flags would help a lot in reducing time for execution of Instrumented Unit tests.

Thanks!
Gaurav Dalal