How-to run Android tests on Firebase TestLab

how-to
firebase
emulator
android
#1

On Android stacks we have the gcloud and gsutil tools pre-installed, this guide is based on these tools. If you want to use it on other stacks, you will need to install these first.

###What you’ll need


  • a Google Cloud Platform service account and its JSON key file.

Read more about creating and configuring service accounts:
https://cloud.google.com/compute/docs/access/service-accounts

  • Google Cloud Testing API and Cloud Tool Results API enabled in your console.

Read more about how to enable and disable Google APIs:
https://support.google.com/cloud/answer/6158841?hl=en

  • the project to be linked to Firebase
    gsutil and gcloud use Google services, so the best is if you create the project on Google Cloud Platform Console, and then import this project into Firebase

Read more about adding projects to Firebase:
https://help.pandasuite.com/articles/create-project-firebase

###Create your app as usual


Running Firebase TestLab tests is similar to the initial Android build workflow, but some modifications will be needed.

###Modify the workflow to use Firebase for tests (in this case: espresso instrumentation)


  1. Upload your service account’s JSON key file on Code Signing tab, under Generic File Storage: name it SERVICE_ACCOUNT_KEY and drop the file into the box, so the downloadable environment variable that contains the url will be: $BITRISEIO_SERVICE_ACCOUNT_KEY_URL

  2. Create a new variable under Env Vars tab. Name it GCP_PROJECT and set its value to the project ID used in Firebase Console and/or in Google Cloud Platform Console.

  3. In the Gradle Runner step
    We will need two APKs exported to BITRISE_DEPLOY_DIR: both unaligned, one test apk and the other a debug apk.

  • Change gradle_task to: "assembleDebug assembleDebugAndroidTest"
  • Remove everything from apk_file_exclude_filter, leave it empty
  • Set apk_file_include_filter to: "*-unaligned.apk"
  1. Add a script step after your gradle-runner step, and set it’s content to:
           #!/bin/bash
            set -ex

            #Download service account key
            curl -o /tmp/sacc_key.json $BITRISEIO_SERVICE_ACCOUNT_KEY_URL
            #Activate cloud client with the service account
            gcloud auth activate-service-account -q --key-file /tmp/sacc_key.json
            #Set the project's id used on Google Cloud Platform
            gcloud config set project $GCP_PROJECT

            RESULT_DIR="build-$BITRISE_BUILD_NUMBER"

            APK="--app=$BITRISE_DEPLOY_DIR/app-debug-unaligned.apk --test=$BITRISE_DEPLOY_DIR/app-debug-androidTest-unaligned.apk"

            TYPE="instrumentation"
            DEVICES="--device model=Nexus6,version=21,locale=en,orientation=portrait"

            gcloud firebase test android run $APK $DEVICES --type=$TYPE --results-dir=$RESULT_DIR/ >>$BITRISE_DEPLOY_DIR/gcloudlog.log 2>&1

            SRCPTH=$BITRISE_DEPLOY_DIR/test_results
            EXPPTH=$BITRISE_DEPLOY_DIR
            mkdir $SRCPTH

            #Download test results

            FB_BUCKET=$(grep -oP "(?<=browser\/).*(?<=\/$RESULT_DIR)" $BITRISE_DEPLOY_DIR/gcloudlog.log)

            gsutil -m cp -R gs://$FB_BUCKET $SRCPTH

            for file in $(find $SRCPTH -type f)
            do
                if [[ -f $file ]]; then
                   f=$(echo ${file//$SRCPTH\/$RESULT_DIR\/})
                   mv $file $(echo $EXPPTH/${f//\//_})
                fi
            done

If everything is configured correctly, you should see your test results among build artifacts.

If you encounter any issues or have a question, please drop a line here and we’ll figure it out as fast as possible.

:rocket: Happy Coding :tada:

5 Likes

Android: Virtual Device Testing on multiple modules
Timed Out on Virtual Device Testing for Android
Google Firebase Android testing service step
Getting 405 response code when using firebase-testlab
Support for Google Test Cloud
How to run Android UI tests on virtual devices
#3

Hello,

I am trying to apply your tutorial but I am kind of stuck to the script step.
It seems that I cannot use “Cloud Tool Result API”.
Here is the log of my “script step” :

+------------------------------------------------------------------------------+
| (5) script@1.1.4 |
+------------------------------------------------------------------------------+
| id: script |
| version: 1.1.4 |
| collection: https://github.com/bitrise-io/bitrise-steplib.git |
| toolkit: bash |
| time: 2017-07-14T15:19:29Z |
+------------------------------------------------------------------------------+
| |
+ curl -o /tmp/sacc_key.json 'https://concrete-userfiles-production.s3-us-west-
2.amazonaws.com/project_file_storage_documents/uploads/4504/original/google_play_api_key.json?X-Amz-
Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKFZGFGFDF256G7J2W2TQ%2F20170714%2Fus-west-
2%2Fs3%2Faws4_request&X-Amz-Date=20170714T151601Z&X-Amz-Expires=86400&X-Amz-
SignedHeaders=host&X-Amz-
Signature=af5c077789217ace95b721ac2e8615653538ff2732f1423bb29150096a3dc49b' 
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 2386 100 2386 0 0 3338 0 --:--:-- --:--:-- --:--:-- 3337

+ gcloud auth activate-service-account -q --key-file /tmp/sacc_key.json
Activated service account credentials for: [bitrise-developer@api-6348956192866480301-
420807.iam.gserviceaccount.com]
+ gcloud config set project cidemo-66b1d
Updated property [core/project].
++ printf %q android_test_automation
+ RESULT_BUCKET=gs://android_test_automation
+ RESULT_DIR=build-19
+ APK='--app=/bitrise/deploy/app-debug-unaligned.apk --test=/bitrise/deploy/app-debug-androidTest-unaligned.apk'
+ TYPE=instrumentation
+ DEVICES='--device model=Nexus6,version=21,locale=en,orientation=portrait --device 
model=Nexus7,version=19,locale=fr,orientation=landscape'
+ gcloud firebase test android run --app=/bitrise/deploy/app-debug-unaligned.apk --test=/bitrise/deploy/app-debug-
androidTest-unaligned.apk --device model=Nexus6,version=21,locale=en,orientation=portrait --device 
model=Nexus7,version=19,locale=fr,orientation=landscape --type=instrumentation --results-
bucket=gs://android_test_automation --results-dir=build-19/

Have questions, feedback, or issues? Get support by visiting:

Creating results bucket [gs://android_test_automation] in project [cidemo-66b1d].
ERROR: (gcloud.firebase.test.android.run) Permission denied while creating bucket [android_test_automation]. Is billing enabled for project: [cidemo-66b1d]?

=> Do I have to pay to make it work ?

0 Likes

#4

Hey!

This issue is not related to the plan luckily. :wink:

Did you enabled these APIs in your console? :slightly_smiling_face:

1 Like

#5

FYI Firebase has a Slack where the team can help with API issues.
https://firebase.community #test-lab channel

That error means you missed a step in the guide, it’s not related to bitrise. See "Enable required APIs"
https://firebase.google.com/docs/test-lab/continuous

2 Likes

#6

Yes, I have been to the console and enabled both “Google Cloud Testing API and Cloud Tool Results API”.
To enable them I have just click on the “enable” blue button.
Is there something more to do ?

0 Likes

#7

Not sure, I think you’d have better luck talking with Firebase support. This issue is unrelated to bitrise.

1 Like

#8

No, it absolutely seem to be something to fix in the how-to. The reason of this error message is because I thought it is a good idea to set an own bucket for storage. But thats requires a blaze plan. So I am testing now using the built in firebase bucket to store results.

1 Like

#9

First of all thanks for the feedback @VincentChannAndroid!

I’ve updated the first post, so the script now won’t use “your own FTL” buckets, so won’t need to subscribe for the blaze plan!

Please let me know if anything goes wrong with the new script!

@bootstraponline: thanks for the Firebase Slack channel and yes, if anyone have any Firebase related issue or question, feel free to ask them. :thumbsup:

1 Like

#10

Thank you =) @tamaspapik,

It works like a charm.

2 Likes

#11

Is it possible to get the test video or its URL from Firebase test lab ? @tamaspapik

0 Likes

#12

This part of the code:

#Download test results

            FB_BUCKET=$(grep -oP "(?<=browser\/).*(?<=\/$RESULT_DIR)" $BITRISE_DEPLOY_DIR/gcloudlog.log)

            gsutil -m cp -R gs://$FB_BUCKET $SRCPTH

            for file in $(find $SRCPTH -type f)
            do
                if [[ -f $file ]]; then
                   f=$(echo ${file//$SRCPTH\/$RESULT_DIR\/})
                   mv $file $(echo $EXPPTH/${f//\//_})
                fi
            done

downloads everything from the bucket object. If there is a video, it should be downloaded also. :slightly_smiling_face:

The place where it will be downloaded is: EXPPTH=$BITRISE_DEPLOY_DIR

So simply put a Deploy to Bitrise.io step if not already.

1 Like

#13

Hello,

I’m having trouble integrating, can you help me? Follow the log:

INFO[16:29:18] Step info not found in StepLib (https://github.com/bitrise-io/bitrise-steplib.git) – Updating …
INFO[16:29:18] Update StepLib (https://github.com/bitrise-io/bitrise-steplib.git)…
From https://github.com/bitrise-io/bitrise-steplib
8b6976d…1e87c4d master -> origin/master
Updating 8b6976d…1e87c4d
Fast-forward
steps/apt-get-install/0.9.0/step.yml | 56 ++++++
steps/brew-install/0.9.0/step.yml | 56 ++++++
steps/cocoapods-install/1.7.2/step.yml | 89 +++++++++
…/2.11.0/step.yml | 214 +++++++++++++++++++++
steps/git-clone/3.5.3/step.yml | 138 +++++++++++++
steps/install-missing-android-tools/2.0.4/step.yml | 56 ++++++
steps/script/1.1.5/step.yml | 116 +++++++++++
steps/slack/2.6.3/step.yml | 177 +++++++++++++++++
8 files changed, 902 insertions(+)
create mode 100644 steps/apt-get-install/0.9.0/step.yml
create mode 100644 steps/brew-install/0.9.0/step.yml
create mode 100644 steps/cocoapods-install/1.7.2/step.yml
create mode 100644 steps/deploy-to-itunesconnect-deliver/2.11.0/step.yml
create mode 100644 steps/git-clone/3.5.3/step.yml
create mode 100644 steps/install-missing-android-tools/2.0.4/step.yml
create mode 100644 steps/script/1.1.5/step.yml
create mode 100644 steps/slack/2.6.3/step.yml
±-----------------------------------------------------------------------------+

| (6) script@1.1.5 |
±-----------------------------------------------------------------------------+
| id: script |
| version: 1.1.5 |
| collection: https://github.com/bitrise-io/bitrise-steplib.git |
| toolkit: bash |
| time: 2017-10-31T16:29:21Z |
±-----------------------------------------------------------------------------+
| |

0 Likes

#14

Hey @CartolaFC!

Can’t see enough info from this log. Could you please contact us on our on-site chat with enabled Bitrise Support user and the app’s url? So we can have a closer look :slight_smile:

1 Like