Collect code coverage during virtual Android device testing

codecov
android
#1

Description of the feature request

Collect code coverage during virtual Android device testing. Make it available to a following script step.

Use case / for what or how I would use it

I would merge device test code coverage statistics with unit test statistics and send the result slack.

0 Likes

#2

Hi @markus.kallander

Thanks for the #feature-request!

Could you please clarify this a bit? Is the issue that there’s absolutely no code coverage report, or just that it’s not exposed?

For generating code coverage reports I found this: https://stackoverflow.com/questions/40726208/how-to-get-code-coverage-reports-from-google-firebase-for-android-espresso-tests

0 Likes

#3

right now there is no way to create a test coverage report for the tests running in the android virtual device step.
While we are able to do that for our unit tests by having a Gradle Runner step to execute the coverage report, we cannot do the same for the instrumentation tests as the gradle task you are running is not editable. createDevDebugCoverageReport could be a potential gradle task to run in case we want to both run the instrumentation tests and create the test coverage report for them

0 Likes

#4

So to be clear @viktorbenei I am using the BETA version of Android device testing step and my question is how I can enable jacoco test reports. The SO solution you posted assumes that we are having full control over the Firebase test labs, which is not the case as far as i understand when we use Bitrise’s step

0 Likes

#5

Hy there!

We created a card for the tooling team to find a way to collect code coverage reports from firebase as a default.
If the enchantment is done, we will post it here.

0 Likes

#6

@fehersanyi-bitrise @viktorbenei Is there any update on this?

0 Likes

#7

Hy there, I’m sorry but there is not yet.

0 Likes

#8

@fehersanyi-bitrise @viktorbenei Any update?

0 Likes

#9

Hello!

Here is my solution to collect the code coverage using the Android Device Testing plugin:

0 Likes

#10

Hello!
Thanks for your example.

However, I’m applying the same configuration and I’m pretty sure I’m not getting back the coverage file.

Once your pipeline is over, where can you check your .ec file? is it in the artifactories section or…?

Thanks!

0 Likes

#11

Just to clarify, I connected to the build machine to check the VDTESTING_DOWNLOADED_FILES_DIR content after the instrumentation test step, and the expected coverage file is not there.

I’ve used exactly the same configuration for the device-testing step (virtual-device-testing-for-android@1.1.1) and checking your gradle file, I can see you are expecting the ec file to be in the VDTESTING_DOWNLOADED_FILES_DIR directory, but still no luck in my case, it has a lot of files (mostly logcat + mp4 + txt + ogg files), but never the .ec file.

Any idea would be welcome!
Thanks

0 Likes

#12

3 months (4 from my first comment) and still no answer. Any help?

0 Likes

#13

At least I can tell you that I have exactly the same problem. I was thinking about executing the tests manually on Firebase Test Labs to get access to the files stored on the test machines. Maybe that way I can find out more what’s going on.

But I think it would be really helpful to get a statement here from @bitce or other bitrise staff members since this should be common problem when working with the new Virtual Device Testing feature. I’ve another post where I tried to work around that using the AVD Manager, but also without success (and I’d rather use the VDT instead of AVD in terms of conviencene and execution speed).

0 Likes

#14

Thank you for your comment.

I spent some time trying to workaround this but didn’t succeed. Manually launching the test on Firebase Test Labs was also something I considered, but after playing around for a while, it came to my mind that we could reach the free tier limit execution time (something we don’t have to worry about through the Bitrise subscription I guess) and hence it would not be the final solution, so I gave up.

Some help from the Bitrise team would be welcome. I reached them also by chat some time ago, but no luck either.

0 Likes