Collect code coverage during virtual Android device testing

markus.kallander · October 13, 2017, 12:55pm

Description of the feature request

Collect code coverage during virtual Android device testing. Make it available to a following script step.

Use case / for what or how I would use it

I would merge device test code coverage statistics with unit test statistics and send the result slack.

viktorbenei · October 17, 2017, 12:49pm

Could you please clarify this a bit? Is the issue that there’s absolutely no code coverage report, or just that it’s not exposed?

For generating code coverage reports I found this: https://stackoverflow.com/questions/40726208/how-to-get-code-coverage-reports-from-google-firebase-for-android-espresso-tests

dimitris-fishbrain · January 23, 2019, 3:57pm

right now there is no way to create a test coverage report for the tests running in the android virtual device step.
While we are able to do that for our unit tests by having a Gradle Runner step to execute the coverage report, we cannot do the same for the instrumentation tests as the gradle task you are running is not editable. createDevDebugCoverageReport could be a potential gradle task to run in case we want to both run the instrumentation tests and create the test coverage report for them

dimitris-fishbrain · January 24, 2019, 3:24pm

So to be clear @viktorbenei I am using the BETA version of Android device testing step and my question is how I can enable jacoco test reports. The SO solution you posted assumes that we are having full control over the Firebase test labs, which is not the case as far as i understand when we use Bitrise’s step

fehersanyi-bitrise · January 30, 2019, 7:51am

Hy there!

We created a card for the tooling team to find a way to collect code coverage reports from firebase as a default.
If the enchantment is done, we will post it here.

vintwala · May 14, 2019, 6:41pm

@fehersanyi-bitrise @viktorbenei Is there any update on this?

fehersanyi-bitrise · May 15, 2019, 8:44am

Hy there, I’m sorry but there is not yet.

jgironda · September 13, 2019, 7:54am

@fehersanyi-bitrise @viktorbenei Any update?

hernandazevedozup · November 7, 2019, 5:00pm

Hello!

Here is my solution to collect the code coverage using the Android Device Testing plugin:

gist.github.com

https://gist.github.com/hernandazevedo/4c19558db2f138c99e3c24838cccf307

bitrise.yml

---
format_version: 1.4.0
default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git

workflows:
  ci:
    steps:
      - activate-ssh-key@4.0.3:
          run_if: '{{getenv "SSH_RSA_PRIVATE_KEY" | ne ""}}'
      - git-clone@4.0.14: {}

This file has been truncated. show original

jacoco.gradle

apply plugin: 'jacoco'

jacoco {
    toolVersion = '0.8.5'
}

def fileFilter = ['**/R.class',
                  '**/R$*x.class',
                  '**/BuildConfig.*',
                  '**/Manifest*.*',

This file has been truncated. show original

jgironda · November 26, 2019, 5:26pm

Hello!
Thanks for your example.

However, I’m applying the same configuration and I’m pretty sure I’m not getting back the coverage file.

Once your pipeline is over, where can you check your .ec file? is it in the artifactories section or…?

Thanks!

jgironda · December 17, 2019, 3:16pm

Just to clarify, I connected to the build machine to check the VDTESTING_DOWNLOADED_FILES_DIR content after the instrumentation test step, and the expected coverage file is not there.

I’ve used exactly the same configuration for the device-testing step (virtual-device-testing-for-android@1.1.1) and checking your gradle file, I can see you are expecting the ec file to be in the VDTESTING_DOWNLOADED_FILES_DIR directory, but still no luck in my case, it has a lot of files (mostly logcat + mp4 + txt + ogg files), but never the .ec file.

Any idea would be welcome!
Thanks

jgironda · March 16, 2020, 8:28am

3 months (4 from my first comment) and still no answer. Any help?

b.kopiske · March 20, 2020, 12:08pm

At least I can tell you that I have exactly the same problem. I was thinking about executing the tests manually on Firebase Test Labs to get access to the files stored on the test machines. Maybe that way I can find out more what’s going on.

But I think it would be really helpful to get a statement here from @bitce or other bitrise staff members since this should be common problem when working with the new Virtual Device Testing feature. I’ve another post where I tried to work around that using the AVD Manager, but also without success (and I’d rather use the VDT instead of AVD in terms of conviencene and execution speed).

jgironda · March 20, 2020, 3:08pm

Thank you for your comment.

I spent some time trying to workaround this but didn’t succeed. Manually launching the test on Firebase Test Labs was also something I considered, but after playing around for a while, it came to my mind that we could reach the free tier limit execution time (something we don’t have to worry about through the Bitrise subscription I guess) and hence it would not be the final solution, so I gave up.

Some help from the Bitrise team would be welcome. I reached them also by chat some time ago, but no luck either.

suke · March 19, 2021, 12:42am

I also encountered this problem.

I’v create this issue and waiting for the response

github.com/bitrise-steplib/steps-virtual-device-testing-for-android

coverage.ec not found in downloaded directory

opened 12:55PM - 18 Mar 21 UTC

omuomugin

### Troubleshooting - [x] I've searched [discuss.bitrise.io](https://discuss.…bitrise.io/) for possible solutions. - Which version of the step is effected? 1.1.2 - Is the issue reproducible with the latest version? YES - Does the issue happen sporadically, or every time? EVERY TIME - Is the issue reproducible locally by following our [local debug guide](https://devcenter.bitrise.io/troubleshooting/debugging-your-build-on-your-own-machine/)? NOT APPLICABLE ### Useful information - [FAQ](https://devcenter.bitrise.io/faq/faq-index/) - [devcenter.bitrise.io](https://devcenter.bitrise.io/) ### Issue description I need exec file from the android testing result but it turned out that no `*.exec` or `*.ec` file were included in `$VDTESTING_DOWNLOADED_FILES_DIR` directory. According to the instruction guide from Firebase Test Lab it should generate `artifacts/coverage.ec` for coverage report with this command ``` gcloud firebase test android run \ --type instrumentation \ --app your-app.apk \ --test your-app-test.apk \ --device model=TestDevice,version=AndroidVersion \ --environment-variables coverage=true,coverageFile="/sdcard/coverage.ec" \ # I guess this is doing the trick --directories-to-pull /sdcard ``` see also https://firebase.google.com/docs/test-lab/android/command-line#code_coverage_reports_for_instrumentation_tests as I read this discussion https://discuss.bitrise.io/t/collect-code-coverage-during-virtual-android-device-testing/3002/12 it may be worked for some time but not now. my config is below ``` ... - virtual-device-testing-for-android@1: is_skippable: true inputs: - test_devices: Pixel2,29,ja,portrait - directories_to_pull: "/sdcard/" - download_test_results: 'true' - environment_variables: |- coverage=true coverageFile=/sdcard/coverage.ec - test_type: instrumentation ... ``` when I run ``` ls -R $VDTESTING_DOWNLOADED_FILES_DIR ``` I get ``` Pixel2-29-ja-portrait_artifacts_5eb3c240877fe387.uid Pixel2-29-ja-portrait_artifacts_hangouts_incoming_call.ogg Pixel2-29-ja-portrait_artifacts_hangouts_message.ogg Pixel2-29-ja-portrait_artifacts_sdcard_Android_data_com.google.android.youtube_cache_exo_5eb3c240877fe387.uid Pixel2-29-ja-portrait_artifacts_sdcard_Android_media_com.google.android.talk_Notifications_hangouts_message.ogg Pixel2-29-ja-portrait_artifacts_sdcard_Android_media_com.google.android.talk_Ringtones_hangouts_incoming_call.ogg Pixel2-29-ja-portrait_bugreport.txt Pixel2-29-ja-portrait_instrumentation.results Pixel2-29-ja-portrait_logcat Pixel2-29-ja-portrait_test_cases_0000_logcat Pixel2-29-ja-portrait_test_cases_0001_logcat Pixel2-29-ja-portrait_test_cases_0002_logcat Pixel2-29-ja-portrait_test_cases_0003_logcat Pixel2-29-ja-portrait_test_cases_0004_logcat Pixel2-29-ja-portrait_test_result_1.xml Pixel2-29-ja-portrait_video.mp4 ``` and no `*.exec` or `*.ec` found My question is - Is this a knowned issue? - Are there any work around that I can do to get `*.exec` or `*.ec` file ### Bitrise info - Build URL: _Please share the build URL with us._ - [Bitrise Support](https://devcenter.bitrise.io/troubleshooting/enabling-bitrise-support-user/) enabled: / NO_ - Log: NO_ <details> <pre> </pre> </details> ### Steps to reproduce 1. set up `[BETA] Virtual Device Testing for Android` workflow 2. set below as config ``` ... - virtual-device-testing-for-android@1: is_skippable: true inputs: - test_devices: Pixel2,29,ja,portrait - directories_to_pull: "/sdcard/" - download_test_results: 'true' - environment_variables: |- coverage=true coverageFile=/sdcard/coverage.ec - test_type: instrumentation ... ``` 3. won't see any `*.ec` or `*.exec` file for coverage report

Topic		Replies	Views
Report coverage Android vitual device testing Steps step , android	1	1151	October 3, 2019
Virtual Device Testing: gathering code coverage for reporting to SonarQube fails Tooling Issues	2	2319	May 20, 2020
Using AVD Manager to run test suite for reporting to SonarQube fails Builds emulator	5	1320	May 1, 2020
Environment Variables are not getting passed to virtual device test lab Tooling Issues android	4	1508	December 11, 2017
Virtual Device Testing [Android] Code Coverage Files Not Generated for Pixel 2 Tooling Issues	1	584	November 7, 2020

Collect code coverage during virtual Android device testing

Description of the feature request

Use case / for what or how I would use it

Related Topics