Virtual Device testing for Android step doesn't work properly with Orchestrator

Archive Tooling Issues
1

Hi,

I am running android UI Tests with Espresso. I am using Android Test Orchestrator and have set clear app data between tests option in my gradle build file, however Bitrise doesn’t clear app data between tests.
When I run my tests locally via the Connect Android Test gradle task then data is cleared between tests and all tests pass, but in CI one of them fails because it preserves the state of previous test case.
Here’s the link to one of the failing builds (for more info go to Tests reports)

The following gradle configuration makes the Android Test Orchestrator run its “pm clear” command after each test invocation. This command ensures that the app’s state is completely cleared between tests. But it seems clearPackageData flag is being ignored

android {
    defaultConfig {
            ...
            testInstrumentationRunnerArguments clearPackageData: 'true'
        }

        testOptions {
            execution 'ANDROIDX_TEST_ORCHESTRATOR'
        }
 }
2

Is Use orchestrator step input set to true?

3

Yes @koral, I’ve tried with both true and false just in case. But none worked.

4

@koral, do you have any update?

5

Sorry but I’m not a Bitrise employee thus I don’t have access to logs.
Having this public info only I don’t have any more ideas.

6

Maybe @tamaspapik or @fehersanyi-bitrise know something about it

7

Hi @javier.marsicano!

Sorry for the huge delay. For future reference, we can be more promptly reached on-site on our direct support channels.

As for this issue, our Tooling team took a brief look but we’ve found the latest build to pass. Can you please advise what changed, if it’s solved or if you just changed your approach?

8

Hi Bence @bitce. Latest build is successful because it is a differen wokflow with different steps and params. This issue is still happening

9

Hello @javier.marsicano,
I believe this to be the same issue as described here: https://github.com/bitrise-steplib/steps-virtual-device-testing-for-android/issues/38

So you need to set this option as an environment which is then passed from the test runner:

 - virtual-device-testing-for-android:
        inputs:
        - test_devices: "$INSTRUMENTATION_DEVICE_LIST"
        - inst_use_orchestrator: 'true'
        - environment_variables: clearPackageData=true
        - test_type: instrumentation
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.