How-to run Android tests on Firebase TestLab

On Android stacks we have the gcloud and gsutil tools pre-installed, this guide is based on these tools. If you want to use it on other stacks, you will need to install these first.

###What you’ll need

  • a Google Cloud Platform service account and its JSON key file.

Read more about creating and configuring service accounts:
Service accounts  |  Compute Engine Documentation  |  Google Cloud

  • Google Cloud Testing API and Cloud Tool Results API enabled in your console.

Read more about how to enable and disable Google APIs:
Getting started  |  Cloud APIs  |  Google Cloud

  • the project to be linked to Firebase
    gsutil and gcloud use Google services, so the best is if you create the project on Google Cloud Platform Console, and then import this project into Firebase

Read more about adding projects to Firebase:

###Create your app as usual

Running Firebase TestLab tests is similar to the initial Android build workflow, but some modifications will be needed.

###Modify the workflow to use Firebase for tests (in this case: espresso instrumentation)

  1. Upload your service account’s JSON key file on Code Signing tab, under Generic File Storage: name it SERVICE_ACCOUNT_KEY and drop the file into the box, so the downloadable environment variable that contains the url will be: $BITRISEIO_SERVICE_ACCOUNT_KEY_URL

  2. Create a new variable under Env Vars tab. Name it GCP_PROJECT and set its value to the project ID used in Firebase Console and/or in Google Cloud Platform Console.

  3. In the Gradle Runner step
    We will need two APKs exported to BITRISE_DEPLOY_DIR: both unaligned, one test apk and the other a debug apk.

  • Change gradle_task to: "assembleDebug assembleDebugAndroidTest"
  • Remove everything from apk_file_exclude_filter, leave it empty
  • Set apk_file_include_filter to: "*-unaligned.apk"
  1. Add a script step after your gradle-runner step, and set it’s content to:
            set -ex

            #Download service account key
            curl -o /tmp/sacc_key.json $BITRISEIO_SERVICE_ACCOUNT_KEY_URL
            #Activate cloud client with the service account
            gcloud auth activate-service-account -q --key-file /tmp/sacc_key.json
            #Set the project's id used on Google Cloud Platform
            gcloud config set project $GCP_PROJECT


            APK="--app=$BITRISE_DEPLOY_DIR/app-debug-unaligned.apk --test=$BITRISE_DEPLOY_DIR/app-debug-androidTest-unaligned.apk"

            DEVICES="--device model=Nexus6,version=21,locale=en,orientation=portrait"

            gcloud firebase test android run $APK $DEVICES --type=$TYPE --results-dir=$RESULT_DIR/ >>$BITRISE_DEPLOY_DIR/gcloudlog.log 2>&1

            mkdir $SRCPTH

            #Download test results

            FB_BUCKET=$(grep -oP "(?<=browser\/).*(?<=\/$RESULT_DIR)" $BITRISE_DEPLOY_DIR/gcloudlog.log)

            gsutil -m cp -R gs://$FB_BUCKET $SRCPTH

            for file in $(find $SRCPTH -type f)
                if [[ -f $file ]]; then
                   f=$(echo ${file//$SRCPTH\/$RESULT_DIR\/})
                   mv $file $(echo $EXPPTH/${f//\//_})

If everything is configured correctly, you should see your test results among build artifacts.

If you encounter any issues or have a question, please drop a line here and we’ll figure it out as fast as possible.

:rocket: Happy Coding :tada:



I am trying to apply your tutorial but I am kind of stuck to the script step.
It seems that I cannot use “Cloud Tool Result API”.
Here is the log of my “script step” :

| (5) script@1.1.4 |
| id: script |
| version: 1.1.4 |
| collection: |
| toolkit: bash |
| time: 2017-07-14T15:19:29Z |
| |
+ curl -o /tmp/sacc_key.json 'https://concrete-userfiles-production.s3-us-west-
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2386 100 2386 0 0 3338 0 --:--:-- --:--:-- --:--:-- 3337

+ gcloud auth activate-service-account -q --key-file /tmp/sacc_key.json
Activated service account credentials for: [bitrise-developer@api-6348956192866480301-]
+ gcloud config set project cidemo-66b1d
Updated property [core/project].
++ printf %q android_test_automation
+ RESULT_BUCKET=gs://android_test_automation
+ RESULT_DIR=build-19
+ APK='--app=/bitrise/deploy/app-debug-unaligned.apk --test=/bitrise/deploy/app-debug-androidTest-unaligned.apk'
+ TYPE=instrumentation
+ DEVICES='--device model=Nexus6,version=21,locale=en,orientation=portrait --device 
+ gcloud firebase test android run --app=/bitrise/deploy/app-debug-unaligned.apk --test=/bitrise/deploy/app-debug-
androidTest-unaligned.apk --device model=Nexus6,version=21,locale=en,orientation=portrait --device 
model=Nexus7,version=19,locale=fr,orientation=landscape --type=instrumentation --results-
bucket=gs://android_test_automation --results-dir=build-19/

Have questions, feedback, or issues? Get support by visiting:

Creating results bucket [gs://android_test_automation] in project [cidemo-66b1d].
ERROR: ( Permission denied while creating bucket [android_test_automation]. Is billing enabled for project: [cidemo-66b1d]?

=> Do I have to pay to make it work ?


This issue is not related to the plan luckily. :wink:

Did you enabled these APIs in your console? :slightly_smiling_face:

1 Like

FYI Firebase has a Slack where the team can help with API issues. #test-lab channel

That error means you missed a step in the guide, it’s not related to bitrise. See "Enable required APIs"


Yes, I have been to the console and enabled both “Google Cloud Testing API and Cloud Tool Results API”.
To enable them I have just click on the “enable” blue button.
Is there something more to do ?

Not sure, I think you’d have better luck talking with Firebase support. This issue is unrelated to bitrise.

1 Like

No, it absolutely seem to be something to fix in the how-to. The reason of this error message is because I thought it is a good idea to set an own bucket for storage. But thats requires a blaze plan. So I am testing now using the built in firebase bucket to store results.

1 Like

First of all thanks for the feedback @VincentChannAndroid!

I’ve updated the first post, so the script now won’t use “your own FTL” buckets, so won’t need to subscribe for the blaze plan!

Please let me know if anything goes wrong with the new script!

@bootstraponline: thanks for the Firebase Slack channel and yes, if anyone have any Firebase related issue or question, feel free to ask them. :thumbsup:

1 Like

Thank you =) @tamaspapik,

It works like a charm.


Is it possible to get the test video or its URL from Firebase test lab ? @tamaspapik

This part of the code:

#Download test results

            FB_BUCKET=$(grep -oP "(?<=browser\/).*(?<=\/$RESULT_DIR)" $BITRISE_DEPLOY_DIR/gcloudlog.log)

            gsutil -m cp -R gs://$FB_BUCKET $SRCPTH

            for file in $(find $SRCPTH -type f)
                if [[ -f $file ]]; then
                   f=$(echo ${file//$SRCPTH\/$RESULT_DIR\/})
                   mv $file $(echo $EXPPTH/${f//\//_})

downloads everything from the bucket object. If there is a video, it should be downloaded also. :slightly_smiling_face:

The place where it will be downloaded is: EXPPTH=$BITRISE_DEPLOY_DIR

So simply put a Deploy to step if not already.

1 Like


I’m having trouble integrating, can you help me? Follow the log:

INFO[16:29:18] Step info not found in StepLib ( – Updating …
INFO[16:29:18] Update StepLib (…
8b6976d…1e87c4d master -> origin/master
Updating 8b6976d…1e87c4d
steps/apt-get-install/0.9.0/step.yml | 56 ++++++
steps/brew-install/0.9.0/step.yml | 56 ++++++
steps/cocoapods-install/1.7.2/step.yml | 89 +++++++++
…/2.11.0/step.yml | 214 +++++++++++++++++++++
steps/git-clone/3.5.3/step.yml | 138 +++++++++++++
steps/install-missing-android-tools/2.0.4/step.yml | 56 ++++++
steps/script/1.1.5/step.yml | 116 +++++++++++
steps/slack/2.6.3/step.yml | 177 +++++++++++++++++
8 files changed, 902 insertions(+)
create mode 100644 steps/apt-get-install/0.9.0/step.yml
create mode 100644 steps/brew-install/0.9.0/step.yml
create mode 100644 steps/cocoapods-install/1.7.2/step.yml
create mode 100644 steps/deploy-to-itunesconnect-deliver/2.11.0/step.yml
create mode 100644 steps/git-clone/3.5.3/step.yml
create mode 100644 steps/install-missing-android-tools/2.0.4/step.yml
create mode 100644 steps/script/1.1.5/step.yml
create mode 100644 steps/slack/2.6.3/step.yml

| (6) script@1.1.5 |
| id: script |
| version: 1.1.5 |
| collection: |
| toolkit: bash |
| time: 2017-10-31T16:29:21Z |
| |

Hey @CartolaFC!

Can’t see enough info from this log. Could you please contact us on our on-site chat with enabled Bitrise Support user and the app’s url? So we can have a closer look :slight_smile:

1 Like

Hello There,

I know it’s an old thread but this was the most relevant I could find. Following the steps described above I can indeed get the “gcloud run” instrumentation test goodies and save them as artifacts.

Using the “Export test results” step I can also publish the JUNIT XML into the Test Reports tab but it does not link up any of the other test run artifacts (logs, videos, etc).

I know there’s a dedicated step - virtual device testing - but we felt running the gcloud commands directly would give us more flexibility. So is there a way to publish the other test artifacts from Google Cloud into the Test Reports tab of a build?


1 Like

Hi @miklos.duma
You can use of “deploy to bitrise” step to upload the artifacts.


Hi, I’m running this script, but I’m getting an error at this step:

FB_BUCKET=$(grep -oP "(?<=browser\/).*(?<=\/$RESULT_DIR)" $BITRISE_DEPLOY_DIR/gcloudlog.log)

Here’s the relevant part of the log:

+ SRCPTH=/bitrise/deploy/test_results
+ EXPPTH=/bitrise/deploy
+ mkdir /bitrise/deploy/test_results
++ grep -oP '(?<=browser\/).*(?<=\/build-152125)' /bitrise/deploy/gcloudlog.log
grep: /bitrise/deploy/gcloudlog.log: No such file or directory

Hello @catherinechi :wave:

Can you please send us the URL of the related build and also enable Support Access on the Settings tab of the app ( Enabling the Bitrise Support user for your app | Bitrise DevCenter ) so that we can take a more in-depth look? :slight_smile: