How to run Android UI tests on virtual devices

Glad to hear, thanks for reporting @kuassivi! :slight_smile:

Indeed; there’s a technical explanation for this, simply that Artifacts can only be uploaded for a specific Build, and when you run the step locally you don’t have any Build URL / “build context” to work with.

If you have any idea for this (where would it upload the artifact from your own local Mac/PC) just let us know!

Right now we’re thinking about allowing the step to run locally (in a “non CI environment”), but in “non CI mode” it would simply print out what files it would upload, it would simply skip the actual upload.

Yup, that was what I would have suggested. There is no real need of moving the files locally.

I also realised you have to create a new step for every different path you want to upload. Why not adding a multiline inputbox like other steps?

Regarding the zip option you have to choose for recursive folders, I can suggest to add a capability like the inline zip extractor functionality of google drive which is based on a js lib that can, perhaps, store temporally the extracted files into the “deployment build directory” and show off the files and paths in a prettify way on the Artifacts tab.

I hope to get some free time to contribute with some steps :slightly_smiling_face:.

1 Like

Definitely great ideas, thank you @kuassivi! :slight_smile:

I’d suggest you to create a Feature Requests at least for the zip extractor, actually we already have a related one, might worth to just add it as a comment there for better visibility / tracking: Artifacts: Make it possible to view HTML files with relative links (CSS etc. refs) - although zip extractor alone might not solve that issue directly…

If you’d have any questions just let us know! :wink:

Hi @tamaspapik, it’s working very well, thank you very much!
However, screenshots are missing e.g. https://www.bitrise.io/build/618fca3ce921e377
Are you able to help?
I’m using version 0.9.9 of the step

Hey @eric-grab!

In the build you’ve sent me shall not be any screenshots. Robo test is a test type that makes screenshots automatically, but you’ve ran instrumentation test which will create screenshots only if you code your test to do it. Instrumentation tests also has a video of it, what you can check under the Virtual Device Tests tab

1 Like

Hi @tamaspapik, do you know Spoon and how it can work with this awesome Bitrise step? :smiley:

I believe the things that need to happen are:

  • Replace the Gradle task that runs the tests with ./gradlew spoon (I’m using a Gradle plugin; this runs the tests + generate reports)
  • Copy the reports in “./app/build/spoon-outputs/” to “Apps & Artifacts”

Thanks a bunch

Hey @eric-grab!

Unfortunately Spoon looks like a client side tool, and to use VDT step, you will need to generate the exact test and app apks.

Just checked Spoon’s documentation, and I can’t see a way of generating them.

Sorry @tamaspapik, let’s forget Spoon for a moment, I’ll rephrase my question as follows:

  • Replace the Gradle task that runs the tests with ./gradlew someTask
  • Copy the files in ./app/build/someFolder/ to “Apps & Artifacts” or “Virtual Device Testing” output directory

Is that possible?

You should be able to replace the gradle task. And also if you have Deploy to Bitrise.io step in your workflow, then files copied to $BITRISE_DEPLOY_DIR should be deployed to the “Apps & Artifacts” tab.

Please contact us on our on-site chat if you encounter any issue doing these, or please send a detailed issue report in the

http://discuss.bitrise.io/c/issues/build-issues

section. :blush:

1 Like

Sorry @tamaspapik, how exactly can I “replace the Gradle task”?
There is absolutely no option to configure Gradle for the Virtual Device Testing step

Understood on the use of Deploy to Bitrise.io step, thanks a lot :smiley:

1 Like

So in your VDT step have the two APK path and Test APK path configured. To change gradle task to generate another flavor APKs, you will need to set “tasks” input in Gradle Runner step.

1 Like

How is that possible @tamaspapik? The virtual device/ emulator only lives within the time the Virtual Device Testing step right?
I’ve already tried having Gradle Runner task right after Virtual Device Testing but I got java.lang.RuntimeException: No device(s) found. e.g. https://www.bitrise.io/build/e48b280ad5781812

I believe that @eric-grab asked about Gradle task used to perform the tests like connectedCheck or spoon NOT about one generating APKs (like assemble).

In case of Firebase Test Lab devices are not connected to local ADB server. APKs (testing and tested) need to be uploaded and then you (or step) can execute a command which starts the tests: gcloud firebase test android run  |  Google Cloud CLI Documentation

Hi @koral, are you saying that behind the scene, Virtual Device Testing step actually calls Firebase Test Lab?

Regardless, I believe I’m very close to achieving my goals, please help me a just a bit more koral and @tamaspapik :smiley:
Within the life span of Virtual Device Testing step, how can I trigger ./gradlew someTaskIwant?

1 Like

Exactly:

The tests run on Firebase Test Lab and we display the results we get from them.

Source:

Current version of Virtual Device Testing step waits until tests are finished by polling every 5 seconds:

It should be possible to split this step into separate “Upload APKs and start tests on Virtual devices” and “Retrieve Virtual devices test results”. Between them you can insert whatever step you want.

Thanks @koral

Reading main.go and Firebase Test Lab documentation, it’s unclear to me whether you guys are calling a REST API (not found in their documentation) or Gcloud CLI (doesn’t seem to support custom Gradle task).
Can’t see myself being able to have both Virtual Device Testing + Spoon working together any time soon.
Will put this off for a while.
Thanks a lot for your help thus far @koral and @tamaspapik :slight_smile:

Firebase REST API doesn’t support stuff like that, we have our own server to handle requests, so this is why you won’t see our calls in the official documentation :slight_smile: Thanks @koral!

So basically VDT uses an “offline” solution to test, which means that it prepares/uploads all stuff required for the test, and then runs the given configuration. not like you have a device connected that can be used from adb. I think this is why you won’t be able to use VDT for this case

2 Likes

Checkout https://github.com/TestArmada/flank for a custom FTL runner that works on bitrise. Spoon is not FTL compatible.

1 Like

This is great. Been struggling with the Create Emulator, Start Emulator, etc. until I noticed this step.

Is it supported to generate jacoco report from the instrumental tests?

Glad to hear that! :slightly_smiling_face:

Does your jacoco report is generated to the device’s sdcard or so? In that case you can fill the pull directories input of the step, so it will be exported to the test artifacts.

1 Like