Xamarin iOS UITest step breaks when entering text

Archive Tooling Issues
, , ,
1

Using the Bitrise step for running Xamarin UITests for iOS, entering text looks to be broken, from what I can tell.

Here’s the error:
Calabash.XDB.Core.Exceptions.DeviceAgentException: DeviceAgent enter_text failed: Timed out waiting for key event to complete

This is most likely related to the hardware keyboard being connected to the simulator, which could be disabled by doing this:
defaults write com.apple.iphonesimulator ConnectHardwareKeyboard -bool NO

There’s a couple of Calabash issues that mention this exception too:

2

So it turns out this is only happening when testing in an iPhone SE simulator, so it’s probably the Calabash issue after all. Seems to work fine on iPhone 5s/6s.

1 Like
3

Good catch, thanks for reporting @Dids! :blush:

4

Unfortunately the workaround I mentioned isn’t a workaround after all, and the test still fails from time to time.

5

With the same timeout error?

Also, did you try to increase the timeout a bit?

6

Same timeout when entering text, yes, but no, I haven’t tried increasing timeouts, so doing that now. :slight_smile:

7

Let us know how that goes! :wink:

8

Unfortunately it’s still happening quite often, even with a 5 minute timeout. :frowning:

9

Did you try to upgrade the tools you use, primarily Calabash?

10

It’s a Xamarin.Forms project, so I’m technically using UITest instead of Calabash.

That said, I am using an old version of NUnit (2.6.4), since Xamarin/Xamarin.Forms recommends using that, but I could try updating.

Unfortunately it’s hard to tell if it’s this specific project/app, UITest/NUnit, resource usage/latency etc. that’s causing the actual issue, since it does work sometimes, but lately it hasn’t worked for most of the time.
I did see web request errors/timeouts when it fails though, so I guess it could also be that the connection to the device agent times out?

Definitely want to get to the bottom of this, even if it takes me a lifetime, haha.

11

Hopefully it won’t :smiley:

Can you share an example error log snippet?

12

Here you go:

Tapping element matching Class("UIButtonLabel").Text("Lähetä") at coordinates [ 286.5, 293 ].
Exceptions while waiting: 
System.Net.Http.HttpRequestException: An error occurred while sending the request ---> System.Net.WebException: Error: ConnectFailure (Connection refused) ---> System.Net.Sockets.SocketException: Connection refused
  at System.Net.Sockets.Socket.Connect (System.Net.EndPoint remoteEP) [0x000d1] in <f8255d9ef0594d18ae2c0d97286b9a80>:0 
  at System.Net.WebConnection.Connect (System.Net.HttpWebRequest request) [0x0019b] in <f8255d9ef0594d18ae2c0d97286b9a80>:0 
   --- End of inner exception stack trace ---
  at System.Net.HttpWebRequest.EndGetRequestStream (System.IAsyncResult asyncResult) [0x00043] in <f8255d9ef0594d18ae2c0d97286b9a80>:0 
  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x00014] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0004e] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x0002e] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x0000b] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Net.Http.HttpClientHandler+<SendAsync>c__async0.MoveNext () [0x002f7] in <6994115ed0ee47caa3f39ecebd680a4c>:0 
   --- End of inner exception stack trace ---
  at System.Net.Http.HttpClientHandler+<SendAsync>c__async0.MoveNext () [0x00486] in <6994115ed0ee47caa3f39ecebd680a4c>:0 
--- End of stack trace from previous location where exception was thrown ---
  at Xamarin.UITest.Shared.Http.HttpClient.SendData (System.String endpoint, System.String method, System.Net.Http.HttpContent content, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable`1[T] timeOut) [0x000ea] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.Shared.Http.HttpClient.Post (System.String endpoint, System.String arguments, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable`1[T] timeOut) [0x00017] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.Shared.Http.HttpClient.Post (System.String endpoint, System.Object arguments, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable`1[T] timeOut) [0x00019] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.iOS.HttpCalabashConnection.Condition (System.Object condition) [0x00000] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.iOS.iOSGestures+<WaitForNoneAnimatingOrElapsed>c__AnonStorey5.<>m__0 () [0x00031] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.Shared.WaitForHelper.WaitFor (System.Func`1[TResult] predicate, System.String timeoutMessage, System.Nullable`1[T] timeout, System.Nullable`1[T] retryFrequency, System.Nullable`1[T] postTimeout) [0x00023] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
Exceptions while waiting for any: 
System.Net.Http.HttpRequestException: An error occurred while sending the request ---> System.Net.WebException: Error: ConnectFailure (Connection refused) ---> System.Net.Sockets.SocketException: Connection refused
  at System.Net.Sockets.Socket.Connect (System.Net.EndPoint remoteEP) [0x000d1] in <f8255d9ef0594d18ae2c0d97286b9a80>:0 
  at System.Net.WebConnection.Connect (System.Net.HttpWebRequest request) [0x0019b] in <f8255d9ef0594d18ae2c0d97286b9a80>:0 
   --- End of inner exception stack trace ---
  at System.Net.HttpWebRequest.EndGetRequestStream (System.IAsyncResult asyncResult) [0x00043] in <f8255d9ef0594d18ae2c0d97286b9a80>:0 
  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x00014] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0004e] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x0002e] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x0000b] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <dbb16e0bacdc4a0f87478e401bc29b6c>:0 
  at System.Net.Http.HttpClientHandler+<SendAsync>c__async0.MoveNext () [0x002f7] in <6994115ed0ee47caa3f39ecebd680a4c>:0 
   --- End of inner exception stack trace ---
  at System.Net.Http.HttpClientHandler+<SendAsync>c__async0.MoveNext () [0x00486] in <6994115ed0ee47caa3f39ecebd680a4c>:0 
--- End of stack trace from previous location where exception was thrown ---
  at Xamarin.UITest.Shared.Http.HttpClient.SendData (System.String endpoint, System.String method, System.Net.Http.HttpContent content, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable`1[T] timeOut) [0x000ea] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.Shared.Http.HttpClient.Post (System.String endpoint, System.String arguments, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable`1[T] timeOut) [0x00017] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.Shared.Http.HttpClient.Post (System.String endpoint, System.Object arguments, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable`1[T] timeOut) [0x00019] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.iOS.HttpCalabashConnection.Map (System.Object arguments) [0x00000] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.iOS.iOSGestures.InternalQuery[T] (Xamarin.UITest.Queries.AppQuery query, System.Object[] args, System.String methodName) [0x00046] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.iOS.iOSGestures.Query (Xamarin.UITest.Queries.AppQuery query) [0x0000d] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.iOS.iOSGestures+<QueryGestureWait>c__AnonStorey3.<>m__0 () [0x00000] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.Shared.WaitForHelper.WaitForAnyOrDefault[T] (System.Func`1[TResult] query, T[] defaultValue, System.Nullable`1[T] timeout, System.Nullable`1[T] retryFrequency, System.Nullable`1[T] postTimeout) [0x00023] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
Error while performing Tap(Class("UINavigationButton"))
Exception: System.Exception: Unable to tap element. Query for Class("UINavigationButton") gave no results.
  at Xamarin.UITest.iOS.iOSApp+<Tap>c__AnonStoreyA.<>m__0 () [0x0004f] in <b2dbb1bd4b224d63be0561fef2070e3b>:0 
  at Xamarin.UITest.Utils.ErrorReporting.With (System.Action func, System.Object[] args, System.String memberName) [0x00006] in <b2dbb1bd4b224d63be0561fef2070e3b>:0

There’s quite a lot more of information available, if you need it. Some of it mentions errors codesigning (which is weird, since it’s clearly running at least one of the tests up to a certain point), and failures to install the app.
It might be worth noting that each of my tests perform a new install of the app, some using a clean install and some using an existing install.

Note: theses tests work locally, and I can verify that it’s “seeing” the correct button from the screenshot it generates.

EDIT: Could the Bitrise test runner be running the tests simultaneously? If so, that might be what’s causing the issues, since they rely on a specific order etc. (I’ve named them so they’re running in specific order)

13

I’ve more or less nailed it down to keyboard input timeouts (EnterText), as all the other exceptions are either fixable with a few build configuration changes on my side, or entirely ignorable.

I’ve used UITest screenshotting to confirm that the input field is visible, and so is the keyboard, it’s just not entering anything, and times out:

DeviceAgent enter_text failed: Timed out waiting for key event to complete

This could be due to the simulator using the “hardware” keyboard, unless you guys are disabling this specifically in the step?

I also wonder if this could be caused by the Xamarin iOS Test step building using xbuild instead of msbuild?
I highly doubt it, but not sure what’s going on, since 1/10 times the build goes through fine.

14

Strange. We don’t, everything’s on default, no modifications at all.

Did you try to reset the Simulator on your Mac and run it in a clean simulator?

Did you try to increase the timeout for the input/key event?

15

I’ve actually been unable to replicate the issue locally on my Mac, which is why it’s been tricky to troubleshoot, since I need to run new builds on Bitrise after every little change.

Xamarin’s UITest doesn’t actually have a separate timeout parameter for the EnterText function, so I’ve been specifying the WaitTimes() property specifically for Bitrise, so it should be globally waiting 5 minutes before timing out.

Now to the good news: I think I’ve actually solved this! :smiley: So far I’ve had a few successful builds/tests in a row, so I’m keeping my fingers crossed.

I ended up sleeping the entire thread for Bitrise builds, right after each test is started, so at the beginning of each test I’m simply doing this:

System.Threading.Thread.Sleep(10 * 1000);

I might still need to sleep the thread here and there, in case I run in to similar issues, but I noticed that it often happened at the beginning of a test, or very close to the beginning.

Before you ask, yes, I’m waiting for the input to appear before entering the text, but that didn’t seem to be enough, not sure why.

1 Like
16

Hmm… That’s really strange, why that 10 sec (?) sleep solves the issue…

One more thing worth a try is to upgrade all the related packages. Xamarin 15.2 rolled out a ton of changes, and we just deployed 15.2.2 which should resolve the regressions in 15.2; it might be that a package upgrade could solve this timeout issue too (e.g. Xamarin.Forms had upgrades too).

In any case thank you for posting the details & your solution/workaround here, I’m sure it’ll help others too! :wink: