[Xamarin][iOS][XamarinUITests on iPad Simulator]

GrzegorzTrawinski · February 2, 2017, 8:00am

Hi Gents,

I am struggling with configuring a workflow that will fire Xamarin UI tests for both Android and iOS platform.
In this topic I would like to describe issue related to iOS.

Assumptions:
I am using bitrise for about half a year and I have plently of working CI’s.
I have working Xamarin UI tests that I can fire locally.
I have configured everything (I hope so) correctly accordingly to: App Center Test - Visual Studio App Center | Microsoft Learn

What I know so far:
This is how looks the machine: https://github.com/bitrise-io/bitrise.io/blob/master/system_reports/osx-xamarin-stable.log so I know which simulators can I use and which android emulators can I create and launch.
Here are some tutorials: http://blog.bitrise.io/2016/02/29/xamarin-ci-with-bitrise-tutorial-part-1-getting-started.html
http://blog.bitrise.io/2016/03/05/xamarin-ci-with-bitrise-tutorial-part-2-testing.html however they are very not accurate, not so informative as I wish it could be, and there is no step by step tutorial actually. But I know I have to use environmental variables in my code.
Here is source of Xamarin iOS test step: GitHub - bitrise-steplib/steps-xamarin-ios-test
Here is source of Xamarin Android step: GitHub - bitrise-steplib/steps-xamarin-android-test
…but they have different inputs, so I assume different approaches, and NO DESCRIPTION at all!
(should I add that I am not familiar with GoLang, and can’t read this sources? )

First thing which is not clear and intuitive is:

Should I have Xamarin Archive step or Xamarin Builder step before running Xamarin iOS Test step?
I testes with both configuration… and not going into details; still I am not sure.
In Xamarin iOS Test I do not provide IPA file, I provide only Xamarin project configuration and Xamarin platform. But is it the configuration of the UI test project itself, or Mobile.iOS project?
Finally what is strange, I tried various configurations and end up with this:

/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/iOS/Xamarin.iOS.Common.targets: error : No installed provisioning profiles match the installed iOS signing identities.

What the heck? I copied workflow with exactly same configuration and provisioning profile, somehow I manage to build my solution withing Xamarin iOS Test step, but I get such strange error.

If anybody is willing to help I would be grateful. If you need more details I can provide of course.
If somebody can provide example working workflow - that would be marvelous!

Cheers,
G

viktorbenei · February 2, 2017, 10:06am

Hi,

Can you please check this How To guide and let us know if you have any questions?

GrzegorzTrawinski · February 2, 2017, 2:10pm

This does not work for me 'cause I do not use Calabash, but Xamarin UI Test project instead.

godreikrisztian · February 2, 2017, 4:23pm

Hi @GrzegorzTrawinski,

Regarding to your questions:

You do not need any build/archive step, the Xamarin iOS Test step manages to build your solution.

Since the step manages to build your solution, you do not need to provide .app file path, it is detected by the step, based on the provided configuration and platform.

The configuration and platform are your solution’s configuration and platform and select the ones, which maps to the desired project configuration and platform.

The desired project configuration and platform is the one, which will generate .app file (most of the times this selects platform: iPhoneSimulator) and launches the Xamarin Test Cloud Agent (most of the times this selects platform: Debug)).

So if your desired project config is: Debug + iPhoneSimulator, by default your solution will have this configs as well, so you can go with this ones. (Most of the times solution config Debug + Any CPU maps to iOS project’s Debug + iPhoneSimulator config, so this is also a good choice).

That is right, do not forget to modify your BeforeEachTest method:

...
			string appBundlePath = Environment.GetEnvironmentVariable("APP_BUNDLE_PATH");

			if (appBundlePath != null && appBundlePath != "")
			{
				// In case of Bitrise step: steps-xamarin-ios-uitest.
				app = ConfigureApp
					.iOS
					.AppBundle(appBundlePath)
					.StartApp();
			}
			else {
				// In case of Bitrise step: steps-xamarin-testcloud-for-ios or running in the IDE.
				app = ConfigureApp
					.iOS
					.StartApp();
			}
...

Please take a look on our code-singing guide.

Please let me know if you need any more informations!

GrzegorzTrawinski · February 3, 2017, 3:02pm

Hi again,

I know that I follow two different topics with almost same question, but issues must treated individually,definitely.
So as I mentioned previously… I created brand new solution for both Android and iOS with Xamarin.Forms + UITests project. Created dummy button and two tests, these are working perfectly locally without issues.

I git push them and created bitrise workflow for testing purpose

Let me kindly ignore previous message as I experience different issues right now.
I assume that issues related to provisioning profile is connected to wrong configuration in my prime project that I can fix obviously. Later.

With the new project I am able to build both iOS app and Test dll, but I cannot get working tests. This is the result of the Xamarin iOS Tests:

Testing (UISample.UITests) against (UISample.iOS)
test dll: /Users/vagrant/git/MobileTest/UISample/UITests/bin/Debug/UISample.UITests.dll
app: /Users/vagrant/git/MobileTest/UISample/iOS/bin/iPhoneSimulator/Debug/UISample.iOS.app

Running Xamarin UITest
$ “/Library/Frameworks/Mono.framework/Versions/Current/Commands/mono” “/Users/vagrant/bitrise/xamarin/nunit/3.4.1/bin/nunit3-console.exe” “/Users/vagrant/git/MobileTest/UISample/UITests/bin/Debug/UISample.UITests.dll” “–result” “/Users/vagrant/deploy/TestResult.xml”

NUnit Console Runner 3.4.1
Copyright (C) 2016 Charlie Poole

Runtime Environment
OS Version: MacOSX 16.3.0.0
CLR Version: 4.0.30319.42000

Test Files
/Users/vagrant/git/MobileTest/UISample/UITests/bin/Debug/UISample.UITests.dll

=> UISample.UITests.Tests(iOS).SecondTestToCheckIfMultipleTestsAreWorking
Full log file: /var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T/uitest/log-2017-02-03_06-05-19-589.txt
iOS test running Xamarin.UITest version: 2.0.5
Skipping IDE integration as important properties are configured. To force IDE integration, add .PreferIdeSettings() to ConfigureApp.
Skipping local screenshots. Can be enabled with EnableScreenshots() when configuring app.
Sim check: App not installed.
1 - LaunchTestAsync:
deviceId: D0602EF4-CCAA-4169-A034-97AEEEB66952
1 - Launching simulator if not already running
=> UISample.UITests.Tests(iOS).WelcomeTextIsDisplayed
Full log file: /var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T/uitest/log-2017-02-03_06-06-35-056.txt
iOS test running Xamarin.UITest version: 2.0.5
Skipping IDE integration as important properties are configured. To force IDE integration, add .PreferIdeSettings() to ConfigureApp.
Skipping local screenshots. Can be enabled with EnableScreenshots() when configuring app.
Sim check: App not installed.
7 - LaunchTestAsync:
deviceId: D0602EF4-CCAA-4169-A034-97AEEEB66952
7 - Launching simulator if not already running

Errors and Failures

SetUp Failed : UISample.UITests.Tests(Android)
One or more child tests had errors

SetUp Failed : UISample.UITests.Tests(iOS)
One or more child tests had errors

Test Run Summary
Overall result: Failed
Test Count: 4, Passed: 0, Failed: 4, Inconclusive: 0, Skipped: 0
Failed Tests - Failures: 0, Errors: 4, Invalid: 0
Start time: 2017-02-03 14:05:19Z
End time: 2017-02-03 14:06:56Z
Duration: 97.572 seconds

Results (nunit3) saved as /Users/vagrant/deploy/TestResult.xml
Test failed, error: exit status 4

Well, I did enabled Calabash agent obviously. I even commented EABLE_TEST_CLOUD directive to double check it is always included. It didn’t helped.

In step I configure that I wish to use iPhone 6 Plus + latest iOS (which is 10.2). And I see such info in log:

Collecting simulator info...
Simulator (iPhone 6s Plus), id: (EBD5719F-BB37-478E-AEC9-448EC2C03E26), status: Shutdown

I can even check that this id is correct accordingly to: https:// [github.com] /bitrise-io/bitrise.io/blob/master/system_reports/osx-xamarin-stable.log

but during the tests it starts to run

deviceId: D0602EF4-CCAA-4169-A034-97AEEEB66952

which is iPhone SE iOS 10.2. Btw. this one starts on my mac either as default if I do not provide any DeviceIdentifier and set UITests as startup project.
Although, I didn’t ask for this device in step configuration .

Additionaly:

SetUp Failed : UISample.UITests.Tests(Android)
One or more child tests had errors

Why he tries to launch Android tests?

Finally, one thing bothers me. Here: http://blog.bitrise.io/2016/03/05/xamarin-ci-with-bitrise-tutorial-part-2-testing.html you mention that you have to embed IOS_SIMULATOR_UDID environment variable usage.

string deviceUDID = Environment.GetEnvironmentVariable(“IOS_SIMULATOR_UDID”);

But I didn’t see the usage of IOS_SIMULATOR_UDID neither in Xamarin iOS Test step source code nor somewhere else, actually. So maybe should I set it manually? It didn’t help either…

Gents, any suggestions?

viktorbenei · February 4, 2017, 11:12am

We’ll try to answer here, just a friendly reminder: could you please create a new topic/question in the future when you face a new / separate issue? That would make it much easier to follow through the discussion and help you.

Thank you!

GrzegorzTrawinski · February 6, 2017, 10:34am

Well ViktorBenei, I cannot agree with you. It it still all related to the same issue: Xamarin iOS tests, I am just in different point on this long, painfull path to make it working.

I still get the issue with not linked Calabash. And my only guess is - it starts different iOS simulator or it does not do it at all.

Collecting simulator info…
Simulator (iPhone SE), id: (D0602EF4-CCAA-4169-A034-97AEEEB66952), status: Shutdown

This status Shutdown make my worried. Should I start simulator manually somehow? I do not see such step anywhere.

and I am still wondering about this one:

string deviceUDID = Environment.GetEnvironmentVariable(“IOS_SIMULATOR_UDID”);

Where do you set this parameter?

godreikrisztian · February 6, 2017, 1:59pm

Hi @GrzegorzTrawinski, regarding to:

The previous version does not handle correctly the specified simulator. I fixed it, the new version (2.1.1) is availabe in the steplib.

I recommend the following test setup to support both testing on bitrise and locally:

			iOSAppConfigurator iosConfigurator = ConfigureApp.iOS;

			string appBundlePath = Environment.GetEnvironmentVariable("APP_BUNDLE_PATH");
			if (!string.IsNullOrEmpty(appBundlePath))
			{
				iosConfigurator.AppBundle(appBundlePath);
			}

			string simulatorUDID = Environment.GetEnvironmentVariable("IOS_SIMULATOR_UDID");
			if (!string.IsNullOrEmpty(simulatorUDID))
			{
				iosConfigurator.DeviceIdentifier(simulatorUDID);
			}

			return iosConfigurator.StartApp();

The step’s new version will export the selected simulator’s UDID into IOS_SIMULATOR_UDID .

If you are using cross platform Xamarin.UITest both android and ios tests are specified in the same UITest project, you need to tell to the step which test to run, related input: Test name to run.

You can figure out the test name based on your IDE’s test pad. In our sample’s case, it looks like:

and the iOS test name to run is: Multiplatform.UItest.Tests(iOS)

GrzegorzTrawinski · February 7, 2017, 3:52pm

Hi @godreikrisztian,

Finally I did it.

Firstly, after your update proper iPhone Simulator launched + it was correctly used:

Collecting simulator info…
Simulator (iPhone 6s Plus), id: (EBD5719F-BB37-478E-AEC9-448EC2C03E26), status: Shutdown
…
Sim check: App not installed.
1 - LaunchTestAsync:
deviceId: EBD5719F-BB37-478E-AEC9-448EC2C03E26
1 - Launching simulator if not already running
6 - Launching simulator if not already running
6 - InstallApp:
pathToBundle: /Users/vagrant/git/MobileTest/UISample/iOS/bin/iPhoneSimulator/Debug/UISample.iOS.app

Secondly, Indeed when I passed iOS test name properly to Test name to run field, e.g. “UISample.UITests.Tests(iOS)” It used only iOS tests.

Finally, here http://blog.bitrise.io/2016/03/05/xamarin-ci-with-bitrise-tutorial-part-2-testing.html you mentioned that you have to provide bundleId

    string deviceUDID = Environment.GetEnvironmentVariable("IOS_SIMULATOR_UDID");
    **string bundleID = "your.applications.bundle.id";**

  ConfigureApp
      .iOS
      **.InstalledApp(bundleID)**
      .DeviceIdentifier(deviceUDID)
      .StartApp();

Proposal from previous post was better because it worked:

  	string appBundlePath = Environment.GetEnvironmentVariable("APP_BUNDLE_PATH");
  	if (!string.IsNullOrEmpty(appBundlePath))
  	{
  		iosConfigurator.AppBundle(appBundlePath);
  	}

So maybe you should update your article ;), currently is misleading? ( http://blog.bitrise.io/2016/03/05/xamarin-ci-with-bitrise-tutorial-part-2-testing.html )

By the way I keep finger crossed about issues raised in Xamarin Android Test step https:// [github.com] /bitrise-steplib/steps-xamarin-android-test/issues/15 → it would be nice to have mirrored functionality here with selecting tests to run.

Don’t you own me a beer, do you?
Cheers,
G

Topic		Replies	Views
[Xamarin][Android] XamarinUITests on emulator Question & Answer xamarin	5	5430	February 6, 2017
How to link with app center and run the xamarin test suits in BitRise Question & Answer ios , android	4	1119	May 18, 2018
Xamarin Archive ios & Android Builds	6	1176	November 25, 2018
Error: No simulators found for OS version - Xamarin IOS Test Tooling Issues xamarin , ios	1	743	January 9, 2020
How to configure UI tests in the Xcode Test plugin? Question & Answer ios , testing , xcode , ui	3	1578	May 30, 2018

[Xamarin][iOS][XamarinUITests on iPad Simulator]

Related topics