Since 2016-12-15 Xamarin Build Times Increased - Stable/Beta Channels

I have a fairly simple Xamarin Forms app that had been building without issue in about 7-8 minutes.

Starting 2016-12-15 I noticed the same builds were getting aborted (over the 10 minute limit) without any changes to the app itself.

@viktorbenei has been very helpful in diagnosing the issue, but still don’t seem to have a resolution.

I’ve tested this with the latest Xamarin Stable/Beta/Alpha channels and they are all the same. I even removed the Android build, and just did iOS and it still isn’t compiling under 10 minutes.

I feel like this also potential problem going forward. Especially with new users with a small Xamarin App (like myself) that just wants to test out Bitrise. If build times are now being increased due to Xamarin/Xcode/etc. this is a problem.

I’m up for suggestions and trying different things if anyone has thoughts on why this might be. Thanks guys!

Hi,

Thanks for reporting!

So, in general we saw significantly slower builds for some projects with Xamarin + Xcode 8.2
We worked with other users to see if there’s anything we could do, but it seems to be a Xamarin issue, where Xamarin + Xcode 8.1 compiled the same project significantly faster than Xamarin + Xcode 8.2, although in general Xcode 8.2 is faster for native iOS projects.

This might or might not be an issue in your case, as it doesn’t affect all projects.

Things to try

Managed Linker configuration

One thing might worth a try is to change the “Managed Linker” configuration in your Xamarin project settings (http://blog.bitrise.io/2016/09/10/weekly-vm-updates.html)

  1. In Xamarin Studio’s project navigator open your iOS project’s Project Options (by double clicking on the project/rigth click Options).
  2. Select iOS Build in the side menu
  3. Change the Linker behavior (in the Code Generation & Runtime section)

Switch the Xamarin Stack to another Xamarin channel stack

Try to build on the Xamarin Stable, Xamarin Beta, Xamarin Alpha stacks, we have a stack for all three Xamarin channels, updated every week to the latest update of that channel.

Did you try to change the Managed Linker configuration @aherrick?

The thing about this is that the issue seems to be related to some kind of project settings (in your Xamarin project), but so far we could not figure out which project config.

Our own sample projects (the ones we use for testing/monitoring the system/performance) were not affected at all, we did not see any significant build time change.

Another note: we worked with another user who reported that the build got slower on their Macs too after the upgrade from Xcode 8.1 to Xcode 8.2. Did you see something similar @aherrick?

A new Xamarin beta was also just released (https://releases.xamarin.com), we’ll update the Beta stack with it this weekend - it’d help a lot if you could test with the update and let us know if that helps!

One more thing: can anyone supply a test project for us, one that we could use to reproduce this issue? As I mentioned before we don’t have this issue with our sample projects, which makes it really hard to test changes which might help to resolve this issue, or to pinpoint the project configuration which might cause this issue.

Hi Vik,

I’m not sure exactly if it was from 8.1 to 8.2. I just know that after December 15th the time started increasing. Does that time correlate to when 8.2 went live?

One side note is for testing I have the exact same App setup in MSFTs Mobile platform (http://mobile.azure.com) and it’s building in the same time it was on Bitrise prior the December 15th. About 7.5-8 minutes.

That’s exactly when the Xcode 8.2 upgrade happened
http://blog.bitrise.io/2016/12/17/weekly-vm-updates-bitrise-cli-1-5-1-xamarin-android-fastlane.html

And all the other reports (~3) mention the same time for this issue.

Re:Mobile Azure: is that on Xcode 8.1 or 8.2?

Related to https://github.com/xamarin/AndroidSupportComponents/issues/32 ?

1 Like

I don’t think that would be related, the issue seems to be iOS related so far.

Hey Vik,

Yes Mobile Azure is on Xcode 8.2

But also for Android there is a serious delay because of the fact building from an empty system.
If you use Xamarin.Forms there is the following at a certain moment:

Downloading https://dl-ssl.google.com/android/repository/android_m2repository_r38.zip into /Users/vagrant/.local/share/Xamarin/zips

This is a pretty large file and depending on the internet speed this takes a long time. Furthermore, after this step, it starts unpacking.

Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Compat/24.2.1/content
-> contains 6013 files

Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Compat/24.2.1/content/m2repository/com/android/support/support-compat/24.2.1/support-compat-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Compat/24.2.1/embedded
Extracted 1 of 14 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Core.UI/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Core.UI/24.2.1/content/m2repository/com/android/support/support-core-ui/24.2.1/support-core-ui-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Core.UI/24.2.1/embedded
Extracted 1 of 10 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Core.Utils/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Core.Utils/24.2.1/content/m2repository/com/android/support/support-core-utils/24.2.1/support-core-utils-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Core.Utils/24.2.1/embedded
Extracted 1 of 8 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Media.Compat/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Media.Compat/24.2.1/content/m2repository/com/android/support/support-media-compat/24.2.1/support-media-compat-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Media.Compat/24.2.1/embedded
Extracted 1 of 19 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Fragment/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Fragment/24.2.1/content/m2repository/com/android/support/support-fragment/24.2.1/support-fragment-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Fragment/24.2.1/embedded
Extracted 1 of 9 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.RecyclerView/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.RecyclerView/24.2.1/content/m2repository/com/android/support/recyclerview-v7/24.2.1/recyclerview-v7-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.RecyclerView/24.2.1/embedded
Extracted 1 of 13 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Vector.Drawable/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Vector.Drawable/24.2.1/content/m2repository/com/android/support/support-vector-drawable/24.2.1/support-vector-drawable-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Vector.Drawable/24.2.1/embedded
Extracted 1 of 7 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Animated.Vector.Drawable/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Animated.Vector.Drawable/24.2.1/content/m2repository/com/android/support/animated-vector-drawable/24.2.1/animated-vector-drawable-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Animated.Vector.Drawable/24.2.1/embedded
Extracted 1 of 7 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.AppCompat/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.AppCompat/24.2.1/content/m2repository/com/android/support/appcompat-v7/24.2.1/appcompat-v7-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.AppCompat/24.2.1/embedded
Extracted 57 of 573 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Design/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Design/24.2.1/content/m2repository/com/android/support/design/24.2.1/design-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.Design/24.2.1/embedded
Extracted 5 of 56 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.MediaRouter/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.MediaRouter/24.2.1/content/m2repository/com/android/support/mediarouter-v7/24.2.1/mediarouter-v7-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.MediaRouter/24.2.1/embedded
Extracted 50 of 507 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v4/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v4/24.2.1/content/m2repository/com/android/support/support-v4/24.2.1/support-v4-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v4/24.2.1/embedded
Extracted 1 of 7 files
Extracting /Users/vagrant/.local/share/Xamarin/zips/29EB87B0770A1740E6766163D3BFCB8B.zip to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.CardView/24.2.1/content
Extracted 601 of 6013 files
Extracting /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.CardView/24.2.1/content/m2repository/com/android/support/cardview-v7/24.2.1/cardview-v7-24.2.1.aar to /Users/vagrant/.local/share/Xamarin/Xamarin.Android.Support.v7.CardView/24.2.1/embedded
Extracted 1 of 12 files

A lot of files extracted. And this proces takes around 4 to 5 minutes extra.

I tried to use the Cache, but if I specify /Users/vagrant/.local/share/Xamarin the cache is added as expected, but upon the next build you will see that only the first zip download is skipped by Xbuild and the extraction is still there. Of course this already cuts building time with at least some minutes but maybe you could find out where the files are going to and where xbuild tries to resolves the files.

running this against the current alpha stack redeuces the time further.
Xamarin just pushed a new RC in the beta channel, so when you update these vm’s in the weekend maybe there is some gain in build time!

In addition add /Users/vagrant/.local/share/Xamarin to the cach push step.
There seemed to be an issue with Xamarin that it did not correctly recognized the cached files. That is fixed in this alpha release (and in the meantime beta release). I brought my 10min build to 1m30s.

1 Like

The thing is that one of the purpose of a CI service is to do full clean builds, to prevent false negatives and false positives caused by previous builds’ results.

It’s of course possible and whether you want to do it depends on your project, but that’s why we have the build-cache system of course :wink:

Btw @stefandevogelaere

Awesome! You might want to consider adding this as a build-cache guide, you can mostly copy paste the template for a How To post from here How to cache CocoaPods dependencies I think :wink: It’d definitely help others / give them an idea about Xamarin related build caching.

Thanks for sharing your insights! :wink:

Wow!!!

You should definitely consider writing a build-cache How To, or send us the details if you can so that we can write one, this is quite impressive!!

Definitely worth a try, it will be part of this weekend’s Xamarin Beta stack update of course :wink:

P.S.: just made the related stack updates changelog public: Weekly Virtual Machine / Stack updates - 2017.01.14

@aherrick can you please send us the logs from there, in email / onsite chat? We’d be interested to check it, maybe we can spot something, e.g. a mapping difference or similar.

1 Like

Thanks again, here too! :wink:

Hi

Any updates on this issue from your side?

My IOS builds are still taking 15mins to complete, no matter what I try.