[Xamarin][Android] XamarinUITests on emulator

[GrzegorzTrawinski]

Hi Gents,

I am struggling with configuring a workflow that will fire Xamarin UI tests for both Android and iOS platform.
In this topic I would like to describe issue related to Android.

Assumptions:
I am using bitrise for about half a year and I have plently of working CI’s.
I have working Xamarin UI tests that I can fire locally.
I have configured everything (I hope so) correctly accordingly to: App Center Test - Visual Studio App Center | Microsoft Learn

What I know so far:
This is how looks the machine: https://github.com/bitrise-io/bitrise.io/blob/master/system_reports/osx-xamarin-stable.log so I know which simulators can I use and which android emulators can I create and launch.
Here are some tutorials: http://blog.bitrise.io/2016/02/29/xamarin-ci-with-bitrise-tutorial-part-1-getting-started.html
http://blog.bitrise.io/2016/03/05/xamarin-ci-with-bitrise-tutorial-part-2-testing.html however they are very not accurate, not so informative as I wish it could be, and there is no step by step tutorial actually. But I know I have to use environmental variables in my code.
Here is source of Xamarin iOS test step: GitHub - bitrise-steplib/steps-xamarin-ios-test
Here is source of Xamarin Android step: GitHub - bitrise-steplib/steps-xamarin-android-test
…but they have different inputs, so I assume different approaches, and NO DESCRIPTION at all!
(should I add that I am not familiar with GoLang, and can’t read this sources? )

I know that I have to have Create Android emulaotr and Start Android emulaotr steps before I launch tests.

First thing which is not clear and intuitive is:

1. Should I have Xamarin Archive step or Xamarin Builder step before running Xamarin Android Test step?
   I testes with both configuration… and not going into details; still I am not sure.
2. In Xamarin Android Test I do not provide APK file, I provide only Xamarin project configuration and Xamarin platform. But is it the configuration of the UI test project itself, or Mobile.Droid project?
   Finally what is strange, I tried various configurations and end up with this:

========== Configs ==========

• project: Mobile/Mobile.sln
• test_project: /Users/vagrant/git/Mobile/MobileUITests/MobileUITests.csproj
• configuration: STAGE
• platform: Any CPU
• clean_build: false
• emulator_serial: TestDevice

==> Building project: Mobile/Mobile.sln
xbuild “Mobile/Mobile.sln” /t:PackageForAndroid /p:Configuration=“STAGE” /p:Platform=“Any CPU” /p:OutputPath=“bin/Any CPU/STAGE/”
XBuild Engine Version 14.0
Mono, Version 4.6.2.0
Copyright (C) 2005-2013 Various Mono authors

Build started 2/1/2017 11:53:04 PM.

---

Project “/Users/vagrant/git/Mobile/Mobile.sln” (PackageForAndroid target(s)):
Target ValidateSolutionConfiguration:
Building solution configuration “STAGE|Any CPU”.
/Users/vagrant/git/Mobile/Mobile.sln: error : Target named ‘PackageForAndroid’ not found in the project.
Done building project “/Users/vagrant/git/Mobile/Mobile.sln”.-- FAILED

Build FAILED.
Errors:

/Users/vagrant/git/Mobile/Mobile.sln (PackageForAndroid) →

/Users/vagrant/git/Mobile/Mobile.sln: error : Target named ‘PackageForAndroid’ not found in the project.

0 Warning(s)
1 Error(s)

I had a lot of different errors before this one, but I think this one is outmost from begining.

If anybody is willing to help I would be grateful. If you need more details I can provide of course.
If somebody can provide example working workflow - that would be marvelous!

Cheers,
G

[viktorbenei]

Hi,

Can you please check this How To guide and let us know if you have any questions?

How to do Calabash UITesting on Bitrise

This guide introduces automated Calabash UITesting for Xamarin projects on Bitrise. Project setup The sample project, i will use in this guide is our xamarin-cross-platform sample. It contains iOS and Android projects, both with Calabash test. The test are placed into the project directories: for ios: REPO_ROOT/iOS/features for android: REPO_ROOT/Droid/fetaures and they are default generated/sample tests, read: Calabash Quickstart for Xamarin.iOS and Creating A Calabash Test for an APK gu…

[GrzegorzTrawinski]

This does not work for me 'cause I do not use Calabash, but Xamarin UI Test project instead.

However,
I will create Hello World example inside new solution with UI Test and verify it with clear state of art.
Will inform you soon.

[GrzegorzTrawinski]

Hi again,

So as I said I did; I created brand new solution for both Android and iOS with Xamarin.Forms + UITests project. Created dummy button and two tests, these are working perfectly locally without issues.

I git push them and created bitrise workflow for testing purposes.

The first issue I did previously is I didn’t read carefully. In Start Android Test first input is Path to Xamarin Project with default value $BITRISE_PROJECT_PATH…

But under default value I had path to solution. So I changed it to project path.

That one was tricky but did the work, finally I could observe how both projects were built properly.

Then I got the result from step:

Time Elapsed 00:02:05.2420170
(i) .dll path: /Users/vagrant/git/MobileTest/UISample/UITests/bin/AnyCPU/Debug/nunit.framework.dll

=> run unit test
NUnit Console Runner 3.4.1
Copyright (C) 2016 Charlie Poole

Runtime Environment
OS Version: MacOSX 16.3.0.0
CLR Version: 4.0.30319.42000

Test Files
/Users/vagrant/git/MobileTest/UISample/UITests/bin/AnyCPU/Debug/nunit.framework.dll

Errors and Failures

1. Invalid : /Users/vagrant/git/MobileTest/UISample/UITests/bin/AnyCPU/Debug/nunit.framework.dll
   WARNING: The runtime version supported by this application is unavailable.
   Using default runtime: v4.0.30319
   No suitable tests found in ‘/Users/vagrant/git/MobileTest/UISample/UITests/bin/AnyCPU/Debug/nunit.framework.dll’.
   Either assembly contains no tests or proper test driver has not been found.

Test Run Summary
Overall result: Failed
Test Count: 0, Passed: 0, Failed: 0, Inconclusive: 0, Skipped: 0
Start time: 2017-02-03 13:53:12Z
End time: 2017-02-03 13:53:12Z
Duration: 0.008 seconds

Results (nunit3) saved as TestResult.xml

result: <?xml version="1.0" encoding="utf-8" standalone="no"?>







No suitable tests found in ‘/Users/vagrant/git/MobileTest/UISample/UITests/bin/AnyCPU/Debug/nunit.framework.dll’.
Either assembly contains no tests or proper test driver has not been found.

/Library/Frameworks/Mono.framework/Versions/Current/bin/mono /Users/vagrant/bitrise/xamarin/nunit/3.4.1/bin/nunit3-console.exe /Users/vagrant/git/MobileTest/UISample/UITests/bin/AnyCPU/Debug/nunit.framework.dll – failed

Egh… Why the hell it uses nunit.framework.dll as dll with tests? It should be UISample.UITests.dll instead!
Does he takes first dll from /bin/ folder? and nunit… beats UISample… due to alphabetical order? (Will check it).
This one is killed me.

Edit:
Yeah just checked different workflow (my primary (and the reason of affair)) and in this case I have such issue:

Errors and Failures

1. Invalid : /Users/vagrant/git/Mobile/MobileUITests/bin/AnyCPU/STAGE/Acr.UserDialogs.dll
   No suitable tests found in ‘/Users/vagrant/git/Mobile/MobileUITests/bin/AnyCPU/STAGE/Acr.UserDialogs.dll’.
   Either assembly contains no tests or proper test driver has not been found.

So he took the first dll that name starts with capital A (why the hell he uses Acr.UserDialogs in UI Tests, btw.?).

Need to comment it - who and why did such implementation of the step!?

[GrzegorzTrawinski]

Hi Gents,

Finally I got it work.
First of all apologize for my irritation in previous messages. I was struggling with this for about 3-4 days and I was little disappointed that first time in my bitrise carrier something is not straightforward, although, eventually, it happens that mostly it was my fault.

So here is the summary of my thoughts, maybe it will help someone in the future:

1. In Create Android Emulator step you will get $BITRISE_EMULATOR_NAME variable, use in:
2. Start Android Step as Emulator to boot, here you will get at the end $BITRISE_EMULATOR_SERIAL, this one use
3. Xamarin android Test step in Emulator serial to boot step.
4. In Xamarin android Test remember that first two fields: Path to Xamarin Project and Path to Xamarin UITest Project requires to have path to .csproj file; not a .sln file!
5. In Xamarin android Test you should use Release configuration in Xamarin project configuration field. Otherwise you will get error about Mono Shared Runtime: “Mono Shared Runtime is not supported. This can be resolved by changing the project configuration or using a Release build”
6. After build your UITest assembly must be presented first in /bin/ folder, 'cause step will take first library from directory. This one is actually a bug for me or at least misconception. I will raise an issue in this step github. As a workaround you can provide Assembly name in Output tab under Project Options.
7. Finally, add Internet permission to you Android app, otherwise you’ll get: “System.Exception : App does not contain permission ‘android.permission.INTERNET’, please include this permission”. I don’t suppose that somebody has an app without this permission; but I did dummy app with a simple button for test purposes and I needed to add this permission. Not sure why.

Last thing to add, Xamarin android Test step will try to run tests for both platforms (you can experience same behaviour in Xamarin iOS Test step). Not sure if this is “by design” behaviour, or it is a bug…

=> run unit test
NUnit Console Runner 3.4.1
Copyright (C) 2016 Charlie Poole

Runtime Environment
OS Version: MacOSX 16.3.0.0
CLR Version: 4.0.30319.42000

Test Files
/Users/vagrant/git/MobileTest/UISample/UITests/bin/AnyCPU/Release/AAA.UISample.UITests.dll

=> UISample.UITests.Tests(Android).SecondTestToCheckIfMultipleTestsAreWorking
Full log file: /var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T/uitest/log-2017-02-06_00-51-11-010.txt
Skipping IDE integration as important properties are configured. To force IDE integration, add .PreferIdeSettings() to ConfigureApp.
Android test running Xamarin.UITest version: 2.0.5
Initializing Android app on device emulator-5554 with apk: /Users/vagrant/git/MobileTest/UISample/Droid/bin/AnyCPU/Release/com.smt.uisample.uisample.apk
Skipping local screenshots. Can be enabled with EnableScreenshots() when configuring app.
Signing apk with Xamarin keystore.
Waiting for element matching Marked(“Hello World!”).
Using element matching Text(“Click me!”).
Tapping coordinates [ 383, 684 ].
Using element matching Text(“Click me!”).
Tapping coordinates [ 383, 684 ].
Using element matching Text(“Click me!”).
Tapping coordinates [ 383, 684 ].
Waiting for element matching Marked(“You click it!”).
=> UISample.UITests.Tests(Android).WelcomeTextIsDisplayed
Full log file: /var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T/uitest/log-2017-02-06_00-53-00-345.txt
Skipping IDE integration as important properties are configured. To force IDE integration, add .PreferIdeSettings() to ConfigureApp.
Android test running Xamarin.UITest version: 2.0.5
Initializing Android app on device emulator-5554 with apk: /Users/vagrant/git/MobileTest/UISample/Droid/bin/AnyCPU/Release/com.smt.uisample.uisample.apk
Skipping local screenshots. Can be enabled with EnableScreenshots() when configuring app.
Signing apk with Xamarin keystore.
Skipping installation: Already installed.
Waiting for element matching Marked(“Hello World!”).
Using element matching Text(“Click me!”).
Tapping coordinates [ 383, 684 ].
Waiting for element matching Marked(“You click it!”).
=> UISample.UITests.Tests(iOS).SecondTestToCheckIfMultipleTestsAreWorking
Full log file: /var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T/uitest/log-2017-02-06_00-54-24-372.txt
iOS test running Xamarin.UITest version: 2.0.5
Skipping IDE integration as important properties are configured. To force IDE integration, add .PreferIdeSettings() to ConfigureApp.
Skipping local screenshots. Can be enabled with EnableScreenshots() when configuring app.
Sim check: App not installed.
1 - LaunchTestAsync:
deviceId: D0602EF4-CCAA-4169-A034-97AEEEB66952
1 - Launching simulator if not already running
=> UISample.UITests.Tests(iOS).WelcomeTextIsDisplayed
Full log file: /var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T/uitest/log-2017-02-06_00-55-34-901.txt
iOS test running Xamarin.UITest version: 2.0.5
Skipping IDE integration as important properties are configured. To force IDE integration, add .PreferIdeSettings() to ConfigureApp.
Skipping local screenshots. Can be enabled with EnableScreenshots() when configuring app.
Sim check: App not installed.
7 - LaunchTestAsync:
deviceId: D0602EF4-CCAA-4169-A034-97AEEEB66952
7 - Launching simulator if not already running

Errors and Failures

1. SetUp Failed : UISample.UITests.Tests(iOS)
   One or more child tests had errors

Test Run Summary
Overall result: Failed
Test Count: 4, Passed: 2, Failed: 2, Inconclusive: 0, Skipped: 0
Failed Tests - Failures: 0, Errors: 2, Invalid: 0
Start time: 2017-02-06 08:51:10Z
End time: 2017-02-06 08:55:56Z
Duration: 285.345 seconds

Results (nunit3) saved as TestResult.xml

result: <?xml version="1.0" encoding="utf-8" standalone="no"?>

































<![CDATA[ at Xamarin.UITest.iOS.iOSAppLauncher.EnsureCalabashRunning (Xamarin.UITest.iOS.ICalabashConnection connection) [0x0000e] in :0 at Xamarin.UITest.iOS.iOSAppLauncher.LaunchApp (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration, Xamarin.UITest.Shared.Http.HttpClient httpClient, Xamarin.UITest.TestCloud.TestCloudiOSAppConfiguration testCloudAppConfiguration, Xamarin.UITest.Shared.Http.HttpClient testCloudWsClient, Xamarin.UITest.Shared.Http.HttpClient xtcServicesClient, System.Boolean testCloudUseDeviceAgent) [0x00091] in :0 at Xamarin.UITest.iOS.iOSApp..ctor (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration) [0x00302] in :0 at Xamarin.UITest.Configuration.iOSAppConfigurator.StartApp (Xamarin.UITest.Configuration.AppDataMode appDataMode) [0x00017] in :0 at UISample.UITests.Tests.BeforeEachTest () [0x00067] in <1e6e2a1aacd649268d505512ea3015eb>:0 at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in <8f2c484307284b51944a1a13a14c0266>:0 ]]>





<![CDATA[ at Xamarin.UITest.iOS.iOSAppLauncher.EnsureCalabashRunning (Xamarin.UITest.iOS.ICalabashConnection connection) [0x0000e] in :0 at Xamarin.UITest.iOS.iOSAppLauncher.LaunchApp (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration, Xamarin.UITest.Shared.Http.HttpClient httpClient, Xamarin.UITest.TestCloud.TestCloudiOSAppConfiguration testCloudAppConfiguration, Xamarin.UITest.Shared.Http.HttpClient testCloudWsClient, Xamarin.UITest.Shared.Http.HttpClient xtcServicesClient, System.Boolean testCloudUseDeviceAgent) [0x00091] in :0 at Xamarin.UITest.iOS.iOSApp..ctor (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration) [0x00302] in :0 at Xamarin.UITest.Configuration.iOSAppConfigurator.StartApp (Xamarin.UITest.Configuration.AppDataMode appDataMode) [0x00017] in :0 at UISample.UITests.Tests.BeforeEachTest () [0x00067] in <1e6e2a1aacd649268d505512ea3015eb>:0 at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in <8f2c484307284b51944a1a13a14c0266>:0 ]]>

/Library/Frameworks/Mono.framework/Versions/Current/bin/mono /Users/vagrant/bitrise/xamarin/nunit/3.4.1/bin/nunit3-console.exe /Users/vagrant/git/MobileTest/UISample/UITests/bin/AnyCPU/Release/AAA.UISample.UITests.dll – failed

Actually, I am happy that it is working and for sure it is a nice alternative for Xamarin UI Cloud.
By the way, I could place somewhere my dummy Xamarin UI Tests POC along with bitrise proper configuration for future reference. Do we have such place somewhere? Or just give a link to github somewhere.

Cheers guys,
G

[viktorbenei]

@GrzegorzTrawinski

By the way, I could place somewhere my dummy Xamarin UI Tests POC along with bitrise proper configuration for future reference. Do we have such place somewhere?

Awesome, would be really fantastic!

The right place is the How To category here on discuss.bitrise.io, we try to keep that category clean and move questions into Question & Answer and only leave real How To guides there.