Actual Android emulator issues

##Android Emulator changes, issues…


In the last couple of weeks (since avdmanager released?) we’ve received user reports about freezing, slow emulators, and the main issue is usually that UI tests are failing, however in another environments they run successfully. The same thing we’ve successfully reproduced and seems like newer api versions are concerned.

###Workarounds, tips…


  • If you need api level 23, 24 or 25: seems like google_apis ABIs are broken. The emulator starts, wait-for-android-emulator step receives the signal of a successful boot, however after couple of seconds it freezes fully.

    • default ABIs reported to be working (still waiting for report of this, please share!)
  • Creating emulators using (currently latest) Create Android Emulator v1.1.5 seems like causing boot timeout in any emulator.

    • waiting for reports of this

###We are working on this issue already


But still need more info.

###Let’s discuss this


This topic is created for collecting global feedbacks and reports from you :wave:. If you have anything to say about Android emulators, please instantly share it with us!

Thanks! :rocket:

2 Likes

Currently, there is no default system image for api 25, only google_apis.

root@7871ff1d62c6:/bitrise/src#sdkmanager "system-images;android-25;default;armeabi-v7a"
Warning: File /root/.android/repositories.cfg could not be loaded.
Error: Failed to find package system-images;android-25;default;armeabi-v7a

Works for me: https://www.bitrise.io/build/6a71cea44eeda821

I have a bitrise app scheduled to build everyday with always latest steps and google_apis android-25 armeabi-v7a emulator. The source code is not changed. Usually, build succeeds but sometimes it fails due to emulator issues.

A few latest examples:

1 Like

Thanks for the feedback @koral.

We also have scheduled tests and I noticed this unstable behavior as well. On some emulators boot timeout happens, on others the step can read the boot status as booted, but after couple of seconds the emulator freezes and becomes unresponsive. The other mystery is that emulator boots, not freezing, but the tests that are working on a local emulator, doesn’t works on Bitrise. The log usually says timeout, or the test runs fully but for any reason the test results gets back as failed.

This is a summary of testing tons of emulators.

Need to figure out a scheme that works. Find android apis, ABIs, etc… that works more than 80% of a time.

Is that local emulator started with exactly the same parameters (-no-window etc.)? Do local machine have GPU?

Tried with multiple variation of flags, and using HAXM also. I think the test result should not depend on the speed of the device. If something can be managed on a local pc, that should be able to be managed on Bitrise as well, just a lot slower.

Isn’t it?

ANR (or something similar like exceeded timeout) may occur in some of system components after boot if emulation is slow enough.
I’m also collecting logs produced by emulators’ logcat in that app. Haven’t analyzed it deeply, but here is one example which appears if tests are failing due to unknown API level and is not present normally:

06-20 22:07:33.180 I/gralloc ( 70): using (fd=11)
06-20 22:07:33.180 I/gralloc ( 70): id =
06-20 22:07:33.180 I/gralloc ( 70): xres = 768 px
06-20 22:07:33.180 I/gralloc ( 70): yres = 1280 px
06-20 22:07:33.180 I/gralloc ( 70): xres_virtual = 768 px
06-20 22:07:33.180 I/gralloc ( 70): yres_virtual = 2560 px
06-20 22:07:33.180 I/gralloc ( 70): bpp = 16
06-20 22:07:33.180 I/gralloc ( 70): r = 11:5
06-20 22:07:33.180 I/gralloc ( 70): g = 5:6
06-20 22:07:33.180 I/gralloc ( 70): b = 0:5
06-20 22:07:33.182 I/gralloc ( 70): width = 118 mm (165.315247 dpi)
06-20 22:07:33.182 I/gralloc ( 70): height = 197 mm (165.035538 dpi)
06-20 22:07:33.182 I/gralloc ( 70): refresh rate = 101.72 Hz
06-20 22:07:33.214 E/SurfaceFlinger( 70): hwcomposer module not found
06-20 22:07:33.242 W/SurfaceFlinger( 70): no suitable EGLConfig found, trying a simpler query
06-20 22:07:33.243 W/SurfaceFlinger( 70): EGL_SLOW_CONFIG selected!
06-20 22:07:33.243 I/SurfaceFlinger( 70): EGL information:
06-20 22:07:33.243 I/SurfaceFlinger( 70): vendor : Android
06-20 22:07:33.243 I/SurfaceFlinger( 70): version : 1.4 Android META-EGL
06-20 22:07:33.243 I/SurfaceFlinger( 70): extensions: EGL_KHR_get_all_proc_addresses EGL_ANDROID_presentation_time EGL_KHR_swap_buffers_with_damage EGL_ANDROID_create_native_client_buffer EGL_ANDROID_front_buffer_auto_refresh EGL_KHR_image_base EGL_KHR_fence_sync EGL_ANDROID_image_native_buffer
06-20 22:07:33.243 I/SurfaceFlinger( 70): Client API: OpenGL_ES
06-20 22:07:33.244 I/SurfaceFlinger( 70): EGLSurface: 5-6-5-0, config=0x0
06-20 22:07:33.247 W/SurfaceFlinger( 70): no suitable EGLConfig found, trying a simpler query
06-20 22:07:33.248 W/SurfaceFlinger( 70): EGL_SLOW_CONFIG selected!
06-20 22:07:33.248 I/SurfaceFlinger( 70): EGL information:
06-20 22:07:33.248 I/SurfaceFlinger( 70): vendor : Android
06-20 22:07:33.248 I/SurfaceFlinger( 70): version : 1.4 Android META-EGL
06-20 22:07:33.248 I/SurfaceFlinger( 70): extensions: EGL_KHR_get_all_proc_addresses EGL_ANDROID_presentation_time EGL_KHR_swap_buffers_with_damage EGL_ANDROID_create_native_client_buffer EGL_ANDROID_front_buffer_auto_refresh EGL_KHR_image_base EGL_KHR_fence_sync EGL_ANDROID_image_native_buffer
06-20 22:07:33.248 I/SurfaceFlinger( 70): Client API: OpenGL_ES
06-20 22:07:33.248 I/SurfaceFlinger( 70): EGLSurface: 5-6-5-0, config=0x0
06-20 22:07:33.277 I/SurfaceFlinger( 70): OpenGL ES informations:
06-20 22:07:33.277 I/SurfaceFlinger( 70): vendor : Android
06-20 22:07:33.277 I/SurfaceFlinger( 70): renderer : Android PixelFlinger 1.4
06-20 22:07:33.277 I/SurfaceFlinger( 70): version : OpenGL ES-CM 1.0
06-20 22:07:33.277 I/SurfaceFlinger( 70): extensions: GL_EXT_debug_marker GL_OES_byte_coordinates GL_OES_fixed_point GL_OES_single_precision GL_OES_read_format GL_OES_compressed_paletted_texture GL_OES_draw_texture GL_OES_matrix_get GL_OES_query_matrix GL_OES_EGL_image GL_OES_EGL_sync GL_OES_compressed_ETC1_RGB8_texture GL_ARB_texture_compression GL_ARB_texture_non_power_of_two GL_ANDROID_user_clip_plane GL_ANDROID_vertex_buffer_object GL_ANDROID_generate_mipmap
06-20 22:07:33.278 I/SurfaceFlinger( 70): GL_MAX_TEXTURE_SIZE = 4096
06-20 22:07:33.278 I/SurfaceFlinger( 70): GL_MAX_VIEWPORT_DIMS = 4096
06-20 22:07:33.292 W/SurfaceFlinger( 70): no suitable EGLConfig found, trying a simpler query
06-20 22:07:33.292 W/SurfaceFlinger( 70): EGL_SLOW_CONFIG selected!
06-20 22:07:33.292 I/SurfaceFlinger( 70): EGL information:
06-20 22:07:33.293 I/SurfaceFlinger( 70): vendor : Android
06-20 22:07:33.293 I/SurfaceFlinger( 70): version : 1.4 Android META-EGL
06-20 22:07:33.293 I/SurfaceFlinger( 70): extensions: EGL_KHR_get_all_proc_addresses EGL_ANDROID_presentation_time EGL_KHR_swap_buffers_with_damage EGL_ANDROID_create_native_client_buffer EGL_ANDROID_front_buffer_auto_refresh EGL_KHR_image_base EGL_KHR_fence_sync EGL_ANDROID_image_native_buffer
06-20 22:07:33.293 I/SurfaceFlinger( 70): Client API: OpenGL_ES
06-20 22:07:33.293 I/SurfaceFlinger( 70): EGLSurface: 5-6-5-0, config=0x0
06-20 22:07:33.339 E/libEGL ( 70): called unimplemented OpenGL ES API
06-20 22:07:33.339 E/SurfaceFlinger( 70): Error while compiling shader:
(… bunch of other shader compilation errors)
06-20 22:07:33.396 E/SurfaceFlinger( 70): õĮ¶ß
06-20 22:07:33.396 E/SurfaceFlinger( 70): Error while linking shaders:
06-20 22:07:33.397 D/SurfaceFlinger( 70): shader cache generated - 24 shaders in 58.540848 ms
06-20 22:07:33.439 D/SurfaceFlinger( 70): Set power mode=2, type=0 flinger=0xa7632000

We’re seeing the same thing.
Testing create emulator v 1.1.3 launching emulator API 19 with play services, and it never boots.
The emulator starts but seems to get stuck on boot screen.

I wasn’t able to see how you guys check for emulator boot since that’s in your adbManager (which isn’t public?)
Here is the output from the moribund emulator

UPDATE:
Using same step versions, start-android-emulator still times out after 120 seconds using a default Emulator (API 19 with NO google_apis)

Here are the sources: https://github.com/bitrise-tools/go-android/blob/master/adbmanager/adbmanager.go

NDC Command {1 bandwidth enable} took too long (6467ms)

Timeout for this warning is 500ms so that one step is about 13 times slower than expected…

Maybe a good starting point is to find out if it is possible to gain some performance by AVD/emulator/docker/VM settings?

1 Like

Thanks @koral.
There must be some issue with the images. Locally querying the system properties, none of them return a desired value.
I’ll look to see if i can change some AVD settings.
Is there a way to verify if the images being used have changed?

1 Like

Tested around, and seems like android-24;armeabi-v7a;default working absolutely fine. Of course -no-window, -gpu off.

However, espresso UI test is a super sensitive stuff. On emulators without HAXM and KVM, the results are really random of a test, for example a click detected as long-click, and View events called before the layout is really inflated. So speed is really important in case of espresso UI tests. To confirm this I started an x86 emulator with HAXM enabled and ran multiple CPU heavy stuff on my PC and in the emulator itself as well to make it a bit slower. It was an easy test to confirm this, because the tests started to randomly fail.

Interesting… :thinking:

Any thoughts of this?

1 Like

Not sure if understand correctly but if you want to check if particular file has changed you can e.g. compare its hash (obtained using sha256sum or so) to the value calculated before performing any actions.

Is there something relevant printed to the logcat or emulator kernel log?

1 Like

Not really. Usually fails the test like the app doesn’t do what is should.

Espresso tests are listening for View events, sometimes it receives the event but I clearly can see from the emulator that the exact view is not even inflated. This results a NullReferenceException when the test engine want’s to ask a textView’s content from the received event’s view ID, but as it is not inflated, it cannot be reached yet. Not sure what’s causing this.

Actually I could not find a method to “slow down” espresso UI tests, any suggestion?

This is the test I am running as a reference.

1 Like

Tried this and it still times out. At this point I believe i’ve tried every emulator and none are working for us.
We need a resolution! a build server isn’t very useful if it’s not building.

In your case, boot timeout was caused because of the low timeout limit. Just set to 300 seconds and booted fine.

Like I said, UI tests are unstable at the moment. Still working on it, I will share as soon as I found a solution

1 Like

Found a back-door solution to all the issues. Follow this below:

###Workflow configuration

  • Create Android Emulator step:
- Platform: android-24
- Abi: armeabi-v7a
- Tag: default
  • Start Android Emulator step:
- Skin: 768x1280
- EmulatorOptions: -no-window -no-boot-anim -gpu off
- BootTimeout: 600 <-- usually ~300-~400, just to make sure...

###A little touch to the test code
Here is a commit with the delays I’ve tested.

https://github.com/trapacska/android-testing/commit/8cdf0b81bbac78472603285038ab8ecdedd74deb

As a short summary:

  • Add imports:
import android.os.SystemClock;
import android.support.test.espresso.DataInteraction;
import android.support.test.espresso.ViewInteraction;
  • Instead of an usual UI test implementation:
onView(withId(R.id.auto_complete_text_view))	
                 .perform(typeText("So"), closeSoftKeyboard());
onView(withText("South China Sea"))		
                 .inRoot(withDecorView(not(is(mActivity.getWindow().getDecorView()))))		 
                 .check(matches(isDisplayed()));	
         onView(withText("Southern Ocean"))		
                 .inRoot(withDecorView(not(is(mActivity.getWindow().getDecorView()))))	
                 .check(matches(isDisplayed()));
  • Separate calls, and put a little to be enough delay between them, like this:
        ViewInteraction x =  onView(withId(R.id.auto_complete_text_view));
        x.perform(typeTextIntoFocusedView("South "), closeSoftKeyboard());

        SystemClock.sleep(200);

        // Should be displayed
        x = onView(withText("South China Sea"));

        SystemClock.sleep(200);

        x.inRoot(withDecorView(not(is(mActivity.getWindow().getDecorView()))));

        SystemClock.sleep(200);

        x.check(matches(isDisplayed()));

        SystemClock.sleep(200);

        // Should not be displayed.
        x = onView(withText("Southern Ocean"));

        SystemClock.sleep(200);

        x.inRoot(withDecorView(not(is(mActivity.getWindow().getDecorView()))));

        SystemClock.sleep(200);

        x.check(doesNotExist());

This is the only way I’ve managed to make all the tests to be successful. The above is just an example, can be any sleep times, or implementation. The main reason for this is only to handle the delay between UI events and the actual UI view states which occurs on slower devices.

1 Like

Have you tried to decrease the screen density and or size?
With less number of pixels to process it should work faster, at least in theory.

I’ve tried few times with original code of the same sample project and with density set to 160. And success rate is higher.
Here is the bitrise app: https://www.bitrise.io/app/1d190f1c0e4c8486#/builds
and project source code: https://github.com/DroidsOnRoids/android-testing

I haven’t modified any tests, just a buildscript to get dependencies installed automatically and to set animation scales to 0.

1 Like

re: adding sleeps everywhere to tests

Do not put sleeps in Espresso tests. That is not a “solution.” Companies investing in automation require tests to pass consistently and quickly. Neither of those goals are possible with sleeps. Espresso as a framework is designed around the concept of idle resource listeners. The idea is the app tells Espresso when to wait or not, without having to hardcode this into the tests. The Android emulators provided by Google are not suitable for Espresso testing at scale. They are simply too flaky. Google’s Firebase Test Lab is the best option in the market today for virtual device testing on Android.

1 Like

Tried both, without success. Sometimes 1 test of 4 became successful, but it is still un-trustable.

Yep. There is the issue. These view events are fired before the view is inflated to be able to get it. = continuous nullrefexceptions. This is what I wanted to fix. As I can see the apps nowaday, developers have no goal of the user should be able to navigate through 6 screens, 20 UI element and couple of other features within 0.5 seconds. Rather they want to know if the components are working, not causing crashes, and shows what it should btw.

I understand that this cannot be a solution for everyone. The tests I used it doesn’t matter if the list is filled within 50ms or 200ms.

1 Like

Btw, UI test integration coming soon! :wink:

I definitely agree @bootstraponline - if you need stable UI / emulator / device tests go with Firebase Test Lab.

But, that should not prevent us from a bit of community knowledge sharing, debugging and searching for alternatives / workarounds :wink: