How to run Android UI tests on virtual devices

Hi @koral, are you saying that behind the scene, Virtual Device Testing step actually calls Firebase Test Lab?

Regardless, I believe Iā€™m very close to achieving my goals, please help me a just a bit more koral and @tamaspapik :smiley:
Within the life span of Virtual Device Testing step, how can I trigger ./gradlew someTaskIwant?

1 Like

Exactly:

The tests run on Firebase Test Lab and we display the results we get from them.

Source:

Current version of Virtual Device Testing step waits until tests are finished by polling every 5 seconds:

It should be possible to split this step into separate ā€œUpload APKs and start tests on Virtual devicesā€ and ā€œRetrieve Virtual devices test resultsā€. Between them you can insert whatever step you want.

Thanks @koral

Reading main.go and Firebase Test Lab documentation, itā€™s unclear to me whether you guys are calling a REST API (not found in their documentation) or Gcloud CLI (doesnā€™t seem to support custom Gradle task).
Canā€™t see myself being able to have both Virtual Device Testing + Spoon working together any time soon.
Will put this off for a while.
Thanks a lot for your help thus far @koral and @tamaspapik :slight_smile:

Firebase REST API doesnā€™t support stuff like that, we have our own server to handle requests, so this is why you wonā€™t see our calls in the official documentation :slight_smile: Thanks @koral!

So basically VDT uses an ā€œofflineā€ solution to test, which means that it prepares/uploads all stuff required for the test, and then runs the given configuration. not like you have a device connected that can be used from adb. I think this is why you wonā€™t be able to use VDT for this case

2 Likes

Checkout https://github.com/TestArmada/flank for a custom FTL runner that works on bitrise. Spoon is not FTL compatible.

1 Like

This is great. Been struggling with the Create Emulator, Start Emulator, etc. until I noticed this step.

Is it supported to generate jacoco report from the instrumental tests?

Glad to hear that! :slightly_smiling_face:

Does your jacoco report is generated to the deviceā€™s sdcard or so? In that case you can fill the pull directories input of the step, so it will be exported to the test artifacts.

1 Like

Thanks for replying.

Hereā€™s my current settings:

Environment Variables:

coverage=true
coverageFile=/sdcard/coverage.instrumentaltests

Directories to pull:

/sdcard

And I can see the coverage file downloaded, found and uploaded by the codecov script. But in codecov.io the coverage on the relevant classes are still zero. Could you suggest where should I start investigating?

1 Like

If youā€™re using android test orchestrator, thereā€™s a known bug that breaks coverage generation. Otherwise it should work.

Make sure your build.gradle is configured correctly for jacoco coverage and using the latest versions.

1 Like

After some further investigation I found that codecov canā€™t process the coverage.ec generated by instrument tests. When I turn it into an xml report, codecov can update the coverage correctly.

But instrument tests only seems to generate the .ec file, not the .em, so I canā€™t convert with ā€˜emma reportā€™. Well, anyways, itā€™s beyond bitriseā€™s power. I appreciate your great work.

Use jacoco, not emma. Jacoco can output different types of reports.

2 Likes

OK, I got a report with java -jar jacococli.jar report, finally! Thank you so much for your help!

3 Likes

Thanks for the tutorial.

I have about 5 different modules all containing instrumentation tests. How can I configure this with bitrise? Wouldnā€™t I need to specifiy 5 different apks for each module?

@PaulWoitaschek right now, with our built in Virtual Device Testing feature you can only run instrumentation tests for a single APK per build.

Feel free to create a #feature-request, or register the apps as separate apps on bitrise.io (you can register the same repo multiple times), or use Firebase TestLab directly via scripts (How-to run Android tests on Firebase TestLab).

If youā€™d have any questions just let us know! :wink:

Hi, I have a project which has multiple modules containing the UI tests. I want to test them all. However, when I install the apk generated by assembleDebugAndroidTest and run virtual device test, I run tests from the app module only. Can you help me with this?

Hi @kevalpatel2106,

Does the build generate multiple APKs? If it does, then please see my reply above :point_up_2: How to run Android UI tests on virtual devices

If not, then please create an issue report with the buildā€™s URL, so that we can look into it (http://discuss.bitrise.io/c/issues/build-issues)

My tests are failing on virtual device as the location is not enabled. All my tests uses espresso apiā€™s and to over come this I am running UI automator calls to enable the location whenever the location dialog is reported. But no luck. Is there anyway to get rid of this. ? Thanks in advance

Hello @ns.mesta!
Could you please write to us on the on-site chat, send a build url and please enable support user on the app?

Hi, I am trying to run this step but it is failing and it is not possible to determine the cause, the following is the only output I get:
Uploading app and test files
App path (/bitrise/deploy/GithubBrowserSample-app-debug.apk), is bundle: false
Assets requested: {isBundle:false testApp: Apk:{UploadURL: GcsPath: Filename:GithubBrowserSample-app-debug.apk} Aab:{UploadURL: GcsPath: Filename:} TestApk:{UploadURL: GcsPath: Filename:GithubBrowserSample-app-debug-androidTest.apk} RoboScript:{UploadURL: GcsPath: Filename:} ObbFiles:[]}
Uploading file(/bitrise/deploy/GithubBrowserSample-app-debug.apk) to (gs://bitrise-addon-ftl-prod-bucket/android-tests/1377cbd8db50543a/app.apk)
=> Files uploaded
Starting test
AndroidInstrumentationTest: &{AppApk:0xc0000ccbc0 AppBundle: AppPackageId: OrchestratorOption:DO_NOT_USE_ORCHESTRATOR TestApk:0xc0000ccc00 TestPackageId: TestRunnerClass: TestTargets:[] ForceSendFields:[] NullFields:[]}
=> Test started
Waiting for test results

  • Validating
  • (1/1) running
  • (0/1) running
    => Test finished
    Test results:
    Model API Level Locale Orientation Outcome
    NexusLowRes 24 en portrait failure

Hi,
I am running Android Instrumentation Unit tests on FTL and I could not find any option to disable video recording & performance monitoring.
I do not want to initialise the gCloud sdk using a shell script, rather I would like to go ahead with the available plugin. Is there any option which I can use to pass following options for my case, since it is of no use for me while running Instrumented Unit tests.
--no-record-video --no-performance-metrics
As of now, i found that the plugin records both of these by default (as these are enabled by default). An option to pass such flags would help a lot in reducing time for execution of Instrumented Unit tests.

Thanks!
Gaurav Dalal