Using AVD Manager to run test suite for reporting to SonarQube fails

emulator
#1

Help required: Proper way to push combined code coverage of unit, robolectric- and instrumented-tests to SonarQube

Description of the issue

I’ve created a jacocoTestReport and a sonarqube gradle task. I can execute them locally on my machine and push a combined coverage report for the complete test suite, i.e. unit-, robolectric- and instrumented tests. The instrumented tests run on a locally spawned Android emulator. The psuh to SonarQube is successful and includes the complete coverage report.

Before implementing the jacocoTestReport and sonarqube gradle tasks, I used the Virtual Device Testing step (beta) to execute my instrumented tests, which worked properly. Since I need to get a combined xml report for the test coverage I have to execute all test categories on a single machine so that the tests can update the existing reports with more data. With the Virtual Device Testing step this does not seem to be possible easily. I can think of two options:

a) Since I have to execute the instrumented tests first and then the robolectric- and unit-tests, it would suffice to copy the coverage data from Firebase Test Labs to the Bitrise VM before executing the other tests in context of the Bitrise VM. But I do not have a clue if that is even possible.

b) Not using the Virtual Device Testing step and execute the new Gradle tasks all inside the Bitrise VM. This is what I’ve done and what does not work.

Here’s a description how I implemented way b) and what happens on building:

Basically I based my workflow on the default primary workflow, but made some changes. You can see the full workflow in the screenshot:

The first step is the AVD Manager, then after the Install Missing Android SDK components, I call the Gradle task assemble for my test suite to prepare while the emulator might still be launching. Then I wait with the “Wait for Android emulator” step. In a following script step I perform the commands

adb shell settings put global window_animation_scale 0
adb shell settings put global transition_animation_scale 0
adb shell settings put global animator_duration_scale 0

in order to speed up test execution. Then I call the Gradle task jacocoTestReport. This is where my problems arise:

About 70% of my tests fail with the following message:

java.lang.RuntimeException: Waited for the root of the view hierarchy to have window focus and not request layout for 10 seconds. If you specified a non default root matcher, it may be picking a root that never takes focus. Root:
	Root{application-window-token=android.view.ViewRootImpl$W@4429e2b, window-token=android.view.ViewRootImpl$W@4429e2b, has-window-focus=false, layout-params-type=1, layout-params-string={(0,0)(fillxfill) sim={forwardNavigation} ty=BASE_APPLICATION wanim=0x10302f8

Environment:

Android & Docker, on Ubuntu 16.04

Involved steps: AVD Manager / Wait for Android Emulator / Script / Gradle UI test
-> last one fails

Reproducibility

  • a Rebuild (even without caches) does not help
  • the issue happens all the time, but it seems that different tests fail
  • all steps are used in the latest version
  • the issue was always there

Local reproduction: Linux / Android (docker based) stack builds

Have not set up local building yet.

0 Likes

Collect code coverage during virtual Android device testing
#2

Hi @b.kopiske!

Thanks for the report! Can you please include a build URL here and enable Support Access (in the Settitngs of the apps) so we can take an in-depth look? :slight_smile:

0 Likes

#3

Here it is:
https://app.bitrise.io/build/623f67e89a07eacd#?tab=log
(Support Access is enabled)

0 Likes

#4

Hi @b.kopiske,

Thank you for your patience.

I noticed you’ve had some green builds since posting this.

Are you still having issues with this behaviour?

Thanks and looking forward to hearing from you.

Kind regards,
Luna

0 Likes

#5

Unfortunately green builds are no indicator for success on SonarQube side. The scanner ignores missing xml reports which yields in empty code coverage stats on the SonarQube server.

As I’ve been in contact with your support team, we’ve agreed that it’d be best to use the Virtual Device Testing for this. So I’ve created a new topic here:

Would really be great to get help on that!

0 Likes