Android Emulator in 90 seconds

#Edit
For now it seems like, the new avdmanager breaks this how-to. Please follow the written below if you are still using emulator as executable to create and use your emulator.

#Summary
The two steps Bitrise.io Cache:Pull and Bitrise.io Cache:Push now helps a lot of users to speed up their build times, and caching emulator snapshots seems to be a good idea as it usually takes couple of minutes to boot up an arm Android emulator.

#What to expect
The last try to speed up android emulator was making it’s booting happen in the background while other steps can run with it paralell. (How to speed up Android emulator heavy workflows) It’s result was a great amount of time saved, in my example from the original 581 seconds I’ve managed to successfully decrease it to 432 seconds. Well, that’s not bad but, let’s try now with some caching!

+------------------------------------------------------------------------------+
|                               bitrise summary                                |
+---+---------------------------------------------------------------+----------+
|   | title                                                         | time (s) |
+---+---------------------------------------------------------------+----------+
| âś“ | cache-pull@0.9.2                                              | 11 sec   |
+---+---------------------------------------------------------------+----------+
| âś“ | script@1.1.3                                                  | 2.20 sec |
+---+---------------------------------------------------------------+----------+
| - | create-android-emulator@1.0.0                                 | 1.14 sec |
+---+---------------------------------------------------------------+----------+
| âś“ | start-android-emulator@1.1.1                                  | 43 sec   |
+---+---------------------------------------------------------------+----------+
| - | script@1.1.3                                                  | 0.73 sec |
+---+---------------------------------------------------------------+----------+
| - | cache-push@0.9.4                                              | 1.33 sec |
+---+---------------------------------------------------------------+----------+
| - | start-android-emulator@1.1.1                                  | 0.76 sec |
+---+---------------------------------------------------------------+----------+
| âś“ | git-clone@3.4.2                                               | 5.02 sec |
+---+---------------------------------------------------------------+----------+
| âś“ | nuget-restore@1.0.3                                           | 23 sec   |
+---+---------------------------------------------------------------+----------+
| âś“ | script@1.1.3                                                  | 183 sec  |
+---+---------------------------------------------------------------+----------+
| âś“ | xamarin-archive@1.3.2                                         | 23 sec   |
+---+---------------------------------------------------------------+----------+
| âś“ | wait-for-android-emulator@0.9.0                               | 11 sec   |
+---+---------------------------------------------------------------+----------+
| âś“ | calabash-android-uitest                                       | 98 sec   |
+---+---------------------------------------------------------------+----------+
| Total runtime: 403 sec                                                       |
+------------------------------------------------------------------------------+

It is about 30 seconds less again! :tada:

#More profit
The workflow in (How to speed up Android emulator heavy workflows) works very well, but only if your workflow average full build time is more than 4-5 minutes, because then there is enought time in the background for the emulator to be able to boot. With caching it doesn’t matter how long your workflow will run, android emulator’s boot time always will be the same.

#Workflow structure for caching
I’ve created an environment variable ANDROID_EMULATOR_NAME set it’s value to cachedEmulator and I used it in all the steps, to be easy to change emulator name later on.

Steps for example:

  • Bitrise.io Cache:pull
  • Script to check if avd exists in /Users/vagrant/.android/avd/$ANDROID_EMULATOR_NAME
  • If not exists, Create Android Emulator with an exra option: --snapshot
  • Start Android Emulator
  • _If not exists,_Script: sleep some (for ex.: 10 secs, let emulator to boot fully), then kill emulator (this is when snapshot images are witten)
  • If not exists, Bitrise.io Cache:push
  • If not exists, Start Android Emulator again, to have a running device after the first cache run also

After the first cache the script which checks if the emulator exists, won’t export an environment variable, so the steps starting with If not exists won’t run anymore. This way you always will have a fresh, empty and clean android emulator image. :smirk:

An example workflow (Best if you keep it separated workflow, and add it to your own workflow’s run_before):

---
format_version: 1.3.1
default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git
trigger_map:
- push_branch: "*"
  workflow: YourWorkflow
- pull_request_source_branch: "*"
  workflow: YourWorkflow
workflows:
  YourWorkflow:
    steps:
    - git-clone@3.4.2: {}
    before_run:
    - _startemulator
    after_run: 
  _startemulator:
    steps:
    - cache-pull@0.9.2:
        inputs:
        - workdir: "/Users/vagrant/.android/avd/"
    - script@1.1.3:
        inputs:
        - content: |-
            #!/bin/bash
            # fail if any commands fails
            set -e
            # debug log
            set -x

            if [ ! -f "/Users/vagrant/.android/avd/$ANDROID_EMULATOR_NAME.ini" ]; then
               envman add --key ANDROID_EMULATOR_EXISTS --value "false"
            fi
    - create-android-emulator@1.0.0:
        run_if: '{{enveq "ANDROID_EMULATOR_EXISTS" "false"}}'
        inputs:
        - name: "$ANDROID_EMULATOR_NAME"
        - options: "--snapshot"
        - custom_hardware_profile_content: ''
    - start-android-emulator@1.1.1:
        inputs:
        - emulator_name: "$ANDROID_EMULATOR_NAME"
    - script@1.1.3:
        run_if: '{{enveq "ANDROID_EMULATOR_EXISTS" "false"}}'
        inputs:
        - content: |-
            #!/bin/bash
            # fail if any commands fails
            set -e
            # debug log
            set -x

            sleep 10
            adb -s $BITRISE_EMULATOR_SERIAL emu kill
    - cache-push@0.9.4:
        run_if: '{{enveq "ANDROID_EMULATOR_EXISTS" "false"}}'
        inputs:
        - cache_paths: |-
            ./${ANDROID_EMULATOR_NAME}.avd/
            ./${ANDROID_EMULATOR_NAME}.ini
        - workdir: "/Users/vagrant/.android/avd/"
    - start-android-emulator@1.1.1:
        run_if: '{{enveq "ANDROID_EMULATOR_EXISTS" "false"}}'
        inputs:
        - emulator_name: "$ANDROID_EMULATOR_NAME"
    before_run: 
    after_run: 
app:
  envs:
  - opts:
      is_expand: true
    ANDROID_EMULATOR_NAME: cachedEmulator

#Warnings
I’ve seen so far…

  • Unexpected crash of the emulator can happen (I’ve read on sites, but didn’t happen in my tests)
  • The first run will take more time (When the emulator images are not cached yet)
  • WARNING: Force to use classic engine to support snapshot.
  • WARNING: Classic qemu does not support SMP. The hw.cpu.ncore option from your config file is ignored.
  • WARNING: opening audio output failed - Seems like audio device won’t be available

Feel free to ask any questions! :wink:

1 Like

@tamaspapik should we delete this guide? Re: https://github.com/bitrise-steplib/steps-create-android-emulator/issues/18#issuecomment-303428198

This still need further investigation. We should keep it, but I’ll mark now the OP.

1 Like

Perfect, thank you!

1 Like

Hello guys, i’m getting this error on my cache-push step:
$ cd /Users/vagrant/.android/avd/
/tmp/bitrise891227701/step_src/step.sh: line 12: cd: /Users/vagrant/.android/avd/: No such file or directory
I made my workflow based on @tamaspapik workflow to open emulator in 90 seconds
Anyone can help me please?

Hello there! Could you please send us the url of this build, so we can check out the logs for some more details?

Hi @fehersanyi-bitrise thanks for answer, here’s my url for this build: /build/32086b7a740af633
https://app.bitrise.io/build/32086b7a740af633

Hy there!

there are some issues with these steps, I would recommend you change the stac to the LTS ubuntu one, remove the emulator steps and add a script with this in it:

#!/usr/bin/env bash
# fail if any commands fails
set -e
# debug log
#set -x

cd $ANDROID_HOME/emulator

echo no | avdmanager create avd -n Nexus_5X_API_26 -k "system-images;android-26;google_apis;x86" --force

emulator -avd Nexus_5X_API_26 -no-window -no-audio -debug-init > /dev/null 2>&1 &

echo "Waiting emulator is ready..."

adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done; input keyevent 82'

echo "Emulator is ready!"

please get back with the results! :upside_down_face:

1 Like

hi fehersanyi,

TL;DR I solved AVD not booting by turning off the camera with option -camera-back none

Unfortunately this script did not work for me. I tried the Ubuntu LTS stack and the regular Ubuntu stack. I tried updating all the SDK components before running, and also tried downloading the latest Canary build of the emulator and running the new headless emulator. I tried pretty much every combination imaginable. The emulator was starting and visible to adb devices but never finished booting. Eventually I figured out that I could run adb logcat and see the debug output from the emulator, and I see the crash:

02-28 01:03:35.009  1726  1726 F HidlStatus: Status.cpp:149] Failed HIDL return status not checked: Status(EX_TRANSACTION_FAILED): 'DEAD_OBJECT: '
--------- beginning of crash
02-28 01:03:35.009  1726  1726 F libc    : Fatal signal 6 (SIGABRT), code -6 (SI_TKILL) in tid 1726 (surfaceflinger), pid 1726 (surfaceflinger)
02-28 01:03:35.289  1776  1776 I crash_dump32: obtaining output fd from tombstoned, type: kDebuggerdTombstone
02-28 01:03:35.303  1728  1728 I cameraserver: ServiceManager: 0xf231a700
02-28 01:03:35.303  1728  1728 I CameraService: CameraService started (pid=1728)
02-28 01:03:35.303  1728  1728 I CameraService: CameraService process starting
02-28 01:03:35.304  1728  1728 W BatteryNotifier: batterystats service unavailable!
02-28 01:03:35.304  1728  1728 W BatteryNotifier: batterystats service unavailable!
02-28 01:03:35.306  1750  1750 I /system/bin/tombstoned: received crash request for pid 1726
02-28 01:03:35.314  1776  1776 I crash_dump32: performing dump of process 1726 (target tid = 1726)
02-28 01:03:35.319  1776  1776 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
02-28 01:03:35.319  1776  1776 F DEBUG   : Build fingerprint: 'google/sdk_gphone_x86/generic_x86:9/PSR1.180720.075/5124027:userdebug/dev-keys'
02-28 01:03:35.319  1776  1776 F DEBUG   : Revision: '0'
02-28 01:03:35.319  1776  1776 F DEBUG   : ABI: 'x86'
02-28 01:03:35.320  1776  1776 F DEBUG   : pid: 1726, tid: 1726, name: surfaceflinger  >>> /system/bin/surfaceflinger <<<
02-28 01:03:35.320  1776  1776 F DEBUG   : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
02-28 01:03:35.320  1776  1776 F DEBUG   : Abort message: 'Status.cpp:149] Failed HIDL return status not checked: Status(EX_TRANSACTION_FAILED): 'DEAD_OBJECT: ''
02-28 01:03:35.321  1776  1776 F DEBUG   :     eax 00000000  ebx 000006be  ecx 000006be  edx 00000006
02-28 01:03:35.321  1776  1776 F DEBUG   :     edi 000006be  esi ec872048
02-28 01:03:35.321  1776  1776 F DEBUG   :     ebp ffaa5ff8  esp ffaa5f78  eip ecd6fb39

The full output of this build is here: https://app.bitrise.io/build/28f938e9ce79a101

This build was running on the Canary release of the emulator, but I got the same errors on the default emulator installed on the LTS and regular Ubuntu stack. This error seems to be related to the camera emulation. I saw all the options by using emulator -help-all and saw an option to turn off the camera emulation: -camera-front none. This fixed it! This is my working script:

#!/usr/bin/env bash
# fail if any commands fails 
set -e
# debug log
set -x

cd $ANDROID_HOME/emulator

echo no | avdmanager create avd -n $ANDROID_EMULATOR_NAME -k "system-images;android-28;google_apis;x86" --force

./emulator-headless -no-window -gpu swiftshader_indirect -no-audio -no-boot-anim -camera-back none @testEmulator > /dev/null 2>&1 &

echo "Waiting emulator is ready..."
adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 1; done; input keyevent 82'
echo "Emulator is ready!"

Note that this is using the Canary build of emulator, currently at v28.1.8 more details here: https://androidstudio.googleblog.com/2019/02/emulator-2818-canary.html - it has a special headless mode for running on CI. My script also works for me with the v27 emulator in the latest Ubuntu stack.

In order to install the Canary emulator I use this script:

#!/usr/bin/env bash
# fail if any commands fails
set -e
# debug log
set -x

cd $ANDROID_HOME

rm -rf $ANDROID_HOME/emulator

wget -q https://dl.google.com/android/repository/emulator-linux-5329922.zip
unzip -q emulator-linux-5329922.zip

ls -al $ANDROID_HOME/emulator

The only way I could find the build number was through the Android Studio SDK manager GUI, switching to Canary updates, then starting an emulator update through there.

I don’t know what is causing the crash with the camera in the emulator on the bitrise machines. It seems from reports that sometimes it does work, and for me occasionally it did actually finish booting with the crash (about 5% of the time). Maybe this bug report will help you identify what the issue is in the stack, but meantime hopefully the -camera-back none option will work for some people.

2 Likes

Correction: the wait for emulator script here does not work reliably. Instead I used the script here:

https://raw.githubusercontent.com/travis-ci/travis-cookbooks/master/community-cookbooks/android-sdk/files/default/android-wait-for-emulator

and that seems to work more reliably.

1 Like

Good news is that startup time for the latest Canary emulator-headless is about 20 seconds, which is really good!

1 Like

Thank you for this @gmaclennan, this is really great! I am very certain this will help out a huge number of people!

Great work! You can find the link to the latest canary here at least: https://aur.archlinux.org/packages/android-emulator-canary/

I’m trying to get detox e2e tests to run on android, but i can’t get it to finish installing the apk on bitrise, works fine on travis and locally: https://github.com/wix/Detox/issues/1194
https://github.com/msand/react-native-svg-e2e
https://app.bitrise.io/app/1a150b039e9b8fbd#/builds

Any suggestions would be greatly appreciated!

Managed to get x86 android working on mac, currently using “system-images;android-28;google_apis;x86” here: https://app.bitrise.io/build/f9c63b00ee41121f

1 Like

Thank you for the build log @msand , this is an awesome solution!

1 Like

Hi, Trying @msand solution, as well as others here. I cannot load an emulator, headless or not, on any of the mac environments, keeps giving this:

emulator: WARNING: Crash service did not start
emulator: ERROR: x86 emulation currently requires hardware acceleration!
Please ensure Intel HAXM is properly installed and usable.
CPU acceleration status: Android Emulator does not support nested virtualization.  Your CPU: 'GenuineIntel', VM host: 'VMwareVMware'

(for example, using ./emulator-headless -no-audio -camera-back none -camera-front none @emulator-5554)
only way to make it continue is to add -no-accel but then later app install hangs.

got a sample setup here - https://app.bitrise.io/build/7e5dbe62dd779e3c#?tab=log.
Problem is quite simple - just can’t load the emulator. either using older canary versions, or current canary version, or the default one provided with the mac setup.
Tried the linux setup, got hang on install too.
As such…no android testing for me :frowning:
I would appreciate any help or suggestions.

It looks like you are trying to start emulator on macOS stack.
Howver, as https://devcenter.bitrise.io/faq/android-x86-emulator/ says:

1 Like

Exactly the tip I needed. Thank you!
Thanks to your comment I gave up trying to run on Mac, and went back to Ubuntu stack.
As said before - It hung when I got to install. However, played a bit more. with this emulator setup i was finally successful:
./emulator-headless -no-window -gpu swiftshader_indirect -no-audio -camera-back none -camera-front none -no-snapshot @emulator-5554 > /dev/null 2>&1 & (note the -no-snapshot part…that’s the new one).

VDT (i.e. Firebase Test Lab) takes too long to run integration tests for us, so I’m trying to replace VDT with the new headless-emulator. I’ve combined all the information in this thread, and it appears to me that I’m close to success. I’ve already tried changing the stack to Ubuntu LTS (avdmanager step starts failing).

The connectedDebugAndroidTest step is failing with this is error:

> Task :app:packageQaDebugAndroidTest

Exception in thread "Device List Monitor" java.lang.NullPointerException
at com.android.ddmlib.EmulatorConsole.checkConnection(EmulatorConsole.java:319)
at com.android.ddmlib.EmulatorConsole.getConsole(EmulatorConsole.java:233)
at com.android.ddmlib.DeviceMonitor.queryAvdName(DeviceMonitor.java:246)
at com.android.ddmlib.DeviceMonitor.updateDevices(DeviceMonitor.java:218)\
at com.android.ddmlib.DeviceMonitor.access$400(DeviceMonitor.java:56)\
at com.android.ddmlib.DeviceMonitor$DeviceListUpdateListener.deviceListUpdate(DeviceMonitor.java:658)
at com.android.ddmlib.DeviceMonitor$DeviceListMonitorTask.processIncomingDeviceData(DeviceMonitor.java:843)
at com.android.ddmlib.DeviceMonitor$DeviceListMonitorTask.run(DeviceMonitor.java:777)
at java.lang.Thread.run(Thread.java:748)

> Task :app:connectedQaDebugAndroidTest

Here’s the relevant part of my configuration:

        - script:
        inputs:
        - content: |-
            #!/usr/bin/env bash

            # fail if any commands fails
            set -e

            # debug log
            set -x

            envman add --key CURRENT_DATE --value `date +%Y-%m-%d`
    - script@1.1.5:
        inputs:
        - content: |
            #!/usr/bin/env bash
            # fail if any commands fails
            set -e
            # debug log
            set -x

            cd $ANDROID_HOME/emulator
            echo no | avdmanager create avd -n Simple -k "system-images;android-28;google_apis;x86" --force
            cd -
    - script@1.1.5:
        inputs:
        - content: |
            #!/usr/bin/env bash
            # fail if any commands fails
            set -e
            # debug log
            set -x

            cd $ANDROID_HOME/emulator

            ./emulator-headless -no-window -gpu swiftshader_indirect -no-audio -camera-back none -camera-front none @Simple > /dev/null 2>&1 &
            cd -

            echo "Waiting for emulator....................."

            bootanim=""
            failcounter=0
            timeout_in_sec=600

            until [[ "$bootanim" =~ "stopped" ]]; do
              bootanim=`adb -e shell getprop init.svc.bootanim 2>&1 &`
              if [[ "$bootanim" =~ "device not found" || "$bootanim" =~ "device offline" || "$bootanim" =~ "no emulators found"
                || "$bootanim" =~ "running" ]]; then
                let "failcounter += 1"
                echo "Waiting for emulator to start"
                if [[ $failcounter -gt timeout_in_sec ]]; then
                  echo "Timeout ($timeout_in_sec seconds) reached; failed to start emulator"
                  exit 1
                fi
              fi
              sleep 1
            done

            echo "!!!! Emulator is ready !!!!"
            sleep 60
    - script@1.1.5:
        inputs:
        - content: |
            #!/usr/bin/env bash
            # fail if any commands fails
            set -e
            # debug log
            set -x

            adb devices
    - change-android-versioncode-and-versionname:
        inputs:
        - new_version_name: "$CURRENT_DATE-$BITRISE_BUILD_NUMBER"
    - install-missing-android-tools@2.3.5:
        inputs:
        - ndk_revision: '19'
    - gradle-runner@1.9.0:
        inputs:
        - gradle_task: |
            -PqaMaskedPhoneNumber=$QA_MASKED_PHONE_NUMBER
            connectedQaDebugAndroidTest

Any thoughts?