Virtual Device Testing: gathering code coverage for reporting to SonarQube fails

Archive Tooling Issues
1

Description of the issue

I’ve created a jacocoTestReport and a sonarqube gradle task in my Android project. I can execute them locally on my machine and push a combined coverage report for the complete test suite, i.e. unit-, robolectric- and instrumented tests. The instrumented tests run on a locally spawned Android emulator. The psuh to SonarQube is successful and includes the complete coverage report.

When trying to execute the sequence jacocoTestReport and sonarqube on Bitrise, I end up with empty code coverage information on SonarQube. This is most certainly related to the following error I see in the logs:

Report doesn't exist: '/bitrise/src/app/build/reports/jacoco/jacocoTestReport/jacocoTestReport.xml'

I need to upload the xml report to SonarQube, since the latest versions reject ec/exec files due to their proprietary format.

My jacocoTestReport step most likely fails because it does not find the ec files from the instrumented tests – although I wonder why not least the exec file generated by the unit tests is used.

Since I am aware that the Virtual Device Testing step runs the tests on Firebase Test Labs, I guessed that the problem must be related to the ec files not being transferred back to the Bitrise VM. I found someone who claims to have solved this here: Bitrise configuration + Jacoco for Junit + Instrumentation tests + Vitual Device Testing · GitHub

So I tried using the environment variables coverage=true and coverageFile=/sdcard/coverage.ec for Firebase Test Labs, and adjusted my gradle files to look in the VDTESTING_DOWNLOADED_FILES_DIR directory, but without success. I found indications that I should better use coverageFilePath=/sdcard/ since the location might differ. But that did not work out either.

In the test run logs I found jacoco errors complaining about not having write access. According to java.io.FileNotFoundException: /jacoco.exec: open failed: EROFS (Read-only file system) (again, with solution) · Issue #968 · jacoco/jacoco · GitHub I created a jacoco-agent.properties with output=none to avoid fallbacks. But no luck, still no coverage files in the downloaded testing dir – only logcat-dumps, mp4 videos of the testing process, ogg sound recordings, some testsuite xmls (nothing jacoco related) and some other result/summary files (nothing jacoco related).

Here’s the current configuration I’'m using:

jacoco Gradle:

apply plugin: 'jacoco'

jacoco {
	toolVersion "0.8.5"
}

task jacocoTestReport(type: JacocoReport, dependsOn: "test${coverageFlavor}DebugUnitTest") {

	group = "Reporting"
	description = "Generate Jacoco coverage reports"

	reports {
		xml.enabled = true
		html.enabled = true
	}

	def fileFilter = [
		'**/R.class',
		'**/R$*.class',
		'**/BuildConfig.*',
		'**/Manifest*.*',
		'android/**/*.*',
		'**/Lambda$*.class',
		'**/Lambda.class',
		'**/*Lambda.class',
		'**/*Lambda*.class',
		'**/*Lambda*.*',
		'**/*Builder.*',
		'**/*_MembersInjector.class',
		'**/*_MembersInjector*.*',
		'**/*_*Factory*.*',
		'**/*Component*.*',
		'**/*Module*.*'
	]
	def debugTree = fileTree(dir: "${buildDir}/intermediates/app_classes/${coverageFlavor}Debug", excludes: fileFilter)
	def mainSrc = "${project.projectDir}/src/main/java"

	getSourceDirectories().setFrom(files([mainSrc]))
	getClassDirectories().setFrom(files([debugTree]))
	getExecutionData().setFrom(
		[fileTree(dir: project.buildDir, includes: [
			'jacoco/test${coverageFlavor}DebugUnitTest.exec',
			'outputs/code_coverage/${coverageFlavor}DebugAndroidTest/**/*.ec'
		])],
		[fileTree(dir: "${System.env.VDTESTING_DOWNLOADED_FILES_DIR}", includes: [
			'*.ec'
		])])
}

bitrise.yml [VDT step]

    - virtual-device-testing-for-android:
        inputs:
        - environment_variables: |-
            coverage=true
            coverageFilePath=/sdcard/
        - directories_to_pull: "/sdcard/"
        - download_test_results: 'true'
        - use_verbose_log: 'true'
        - test_type: instrumentation
        - inst_use_orchestrator: 'true'
        - test_devices: |-
            Pixel2,28,en,portrait
            Nexus6P,27,en,portrait
        - num_flaky_test_attempts: '1'

What am I doing wrong?

Environment:

Android & Docker, on Ubuntu 16.04

Involved steps: Virtual Device Testing / Gradle Tasks for jacocoTestReport and sonarqube

Reproducibility

  • the issue happens all the time
  • the issue was there from beginning
  • all steps are used in the latest version

Build log

2

Reach out to us via https://www.bitrise.io/bitrise-support. You will receive an answer much faster through email support than the forum.

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.