Virtual Device Testing | Why are permissions always granted before tests?

[PLN]

Virtual Device Testing seems to always grant permissions before running tests

Description of the issue

First of all congrats for this feature, it’s a game changer!

I have a question regarding permissions though: it seems that whatever I try, permissions requested by my app are always granted before running the tests. This prevents us to test the interactions that should happen when permissions are not yet granted:

junit.framework.AssertionFailedError: The permission was already granted, cannot test MainActivityNoPermission!

As you can see in our latest build, the permission apparently is always granted before running the tests:

if (ContextCompat.checkSelfPermission(mActivityRule.getActivity(), permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED) {
    fail("The permission was already granted, cannot test MainActivityNoPermission!")
}

This prevents us from testing part of our application’s behavior. It seems to be due to the Virtual Device Testing setup, as Instrumentation testing normally doesn’t grant any permission before running the tests (which is why the docs recommend using a GrantPermissionRule if one wants to actually have them granted before the tests run).

Do you have an idea what causes all permissions to be granted at install, even on an API26 device which should request them at runtime? Is this a hack you put in place to simplify testing, or could the issue come from Firebase Test Lab itself?

(I couldn’t find anyone mentioning the same issues online, neither with Bitrise nor Firebase Test Lab).

Thanks for your help!

Environment:

Where did the issue happen?

Bitrise.io | Docker with pre-cached Android setup, installed on Ubuntu 18.04.

Which build Step causes the issue and which version of the step?

[BETA] Virtual Device Testing for Android 1.0.3
E.g.: Git Clone v3.6.0

Reproducibility

• _Does a “Rebuild” help? NO
• _Does a rebuild without caches help? NO
• Does the issue happen sporadically, or every time? : every time
• Does upgrading the build Step to the latest version help? : NO (already latest)
• When did the issue start? : When we started using Virtual Device Testing ¯\_(ツ)_/¯

Local reproduction: Linux / Android (docker based) stack builds

• With a clean git clone, on my local devices, the permissions are not granted before running the app.
• Running the step with Bitrise CLI should reproduce the issue as it uses VDT instead of my connected device, but following the quick-repo-test instructions result in a android-lint failure:
  Failed to open project, error: lstat /tmp/__plugin__057501229/build.gradle: permission denied

Build log

Our build is private, but the logs don’t contain much.

• Here are the VDT logs:

at junit.framework.Assert.fail(Assert.java:50)
at com.algolia.instantsearch.voice.demo.MainActivityNoPermissionTest.clickInputButton_displaysPermissionOverlay(MainActivityNoPermissionTest.kt:55)
at java.lang.reflect.Method.invoke(Native Method)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at android.support.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:527)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at android.support.test.runner.AndroidJUnit4.run(AndroidJUnit4.java:101)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:384)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2068)

• Here’s the step log:

+------------------------------------------------------------------------------+
| (7) virtual-device-testing-for-android@1.0.3                                 |
+------------------------------------------------------------------------------+
| id: virtual-device-testing-for-android                                       |
| version: 1.0.3                                                               |
| collection: https://github.com/bitrise-io/bitrise-steplib.git                |
| toolkit: go                                                                  |
| time: 2018-09-28T08:56:50Z                                                   |
+------------------------------------------------------------------------------+
|                                                                              |
INFO[08:56:50] Start installing (golang) with apt-get       
INFO[08:56:50]  * [OK] Step dependency (go) installed, available. 
Configs:
- ApkPath: /bitrise/deploy/app-debug.apk
- TestTimeout: 900
- DirectoriesToPull: 
- EnvironmentVariables: 
- TestDevices:
---
Model         API Level   Locale     Orientation   
Nexus6P       26          portrait   en            
NexusLowRes   23          portrait   fr            
---
- AppPackageID: 
- TestType: instrumentation
- TestApkPath: /bitrise/deploy/app-debug-androidTest.apk
- InstTestPackageID: 
- InstTestRunnerClass: 
- InstTestTargets: 
- UseOrchestrator: true
Upload APKs
=> APKs uploaded
Start test
=> Test started
Waiting for test results
- Validating
- (2/2) running
- (1/2) running
- (0/2) running
=> Test finished
Test results:
Model         API Level   Locale   Orientation   Outcome              
Nexus6P       26          en       portrait      failure   
NexusLowRes   23          fr       portrait      failure   
|                                                                              |
+---+---------------------------------------------------------------+----------+
| x | virtual-device-testing-for-android@1.0.3 (exit code: 1)       | 161 sec  |
+---+---------------------------------------------------------------+----------+
| Issue tracker: ...se-steplib/steps-virtual-device-testing-for-android/issues |
| Source: ...thub.com/bitrise-steplib/steps-virtual-device-testing-for-android |
+---+---------------------------------------------------------------+----------+

[tamaspapik]

Hey @PLN!

We are not setting anything in TestLab to force granting permissions. I think it is a natural behavior from their side. The quickest and most straightforward way to solve your issue would be if you contact them on their public slack channel.

[PLN]

Hi @tamaspapik, thanks for your reply. I’ll check with them there and will report my findings here!

[PLN]

After asking on FTL’s Community Slack, I got the answer:

I don’t think so, the apks are installed with -g for grant all

To new readers: there is now an open issue on Firebase Test Lab’s Issue Tracker, please star it to add visibility if you have the same issue