Xcode 15 unit test time increases 5x

Hi,

After upgrading to stack to osx-xcode-15.0.x
Our projects unit test time increases 5x (~60 mins) compared to Xcode 14 stack (~15 min)
We are unable to reproduce this on our mac (M1 Pro), unit tests runs smoothly on our side with same Xcode version

Haven’t been able to create a project to reproduce this issue yet, but my friend who is also using Bitrise Xcode 15 stack is also having this issue.

Any thoughts? Thank you!

2 Likes

@JohnnyTseng that looks odd. Can you provide two build urls to compare with support access enabled please?

Same here, my tests never finish now. Im running unit + ui tests. Builds are timing out.
Some comparison:
Xcode 15:
:heavy_check_mark: testAddingItemsToTheBagFromWishlistCarouselWhenSizeIsNotSelected (71.753 seconds)
:heavy_check_mark: testAddingItemsToTheBagFromWishlistCarouselWhenSizeIsSelected (223.265 seconds)
:heavy_check_mark: testAddingItemToTheBagFromPDP (0.268 seconds)
:heavy_check_mark: testAddPromoCodeIsShownAtTheTopWhenFeatureFlagIsOn (326.061 seconds)
:heavy_check_mark: testAddPromoCodeIsShownInTheListWhenFeatureFlagIsOff (13.230 seconds)
:heavy_check_mark: testApplyingWrongPromoCode (152.068 seconds)
:heavy_check_mark: testBagShowsCorrectPricingInformation (0.286 seconds)
:heavy_check_mark: testDeliveryInformationCanBeDisplayed (71.072 seconds)
:heavy_check_mark: testDonateToCharityAddOnDoesNotAppearWhenFeatureFlagDisabled (87.873 seconds)

Xcode 14.3:
:heavy_check_mark: testAddingItemsToTheBagFromWishlistCarouselWhenSizeIsNotSelected (5.816 seconds)
:heavy_check_mark: testAddingItemsToTheBagFromWishlistCarouselWhenSizeIsSelected (5.779 seconds)
:heavy_check_mark: testAddingItemToTheBagFromPDP (0.061 seconds)
:heavy_check_mark: testAddPromoCodeIsShownAtTheTopWhenFeatureFlagIsOn (4.411 seconds)
:heavy_check_mark: testAddPromoCodeIsShownInTheListWhenFeatureFlagIsOff (4.682 seconds)
:heavy_check_mark: testApplyingWrongPromoCode (7.832 seconds)
:heavy_check_mark: testBagShowsCorrectPricingInformation (0.039 seconds)
:heavy_check_mark: testDeliveryInformationCanBeDisplayed (4.598 seconds)
:heavy_check_mark: testDonateToCharityAddOnDoesNotAppearWhenFeatureFlagDisabled (4.063 seconds)

Using:
xcode15
Computer Name: vm-osx-ventura-15-g2-m1.4core-d050cfd2-80c7-41c5-a25a-4c394782ef94

Locally its fine

We’re seeing an increase as well but it’s 2.5-3x for our project.

Xcode 14.2

Xcode 15

Hi, @CraigSiemens the links you shared lead me to sign in page.

Only a bitrise employee would be able to view those.

I’m from Bitrise actually. For some reasons when I click or hover on copy the link url I can only see Bitrise - Mobile Continuous Integration and Delivery - iOS & Android Build Automation

Maybe it’s because discuss forum obscures Bitrise links. Please create a support request https://support.bitrise.io/hc/en-us/requests/new and pass us the ticket number.

We are also experiencing increased build times when updating to Xcode 15

Here is a build log with regular build time ~10 mins (link)
And here is a build log with Xcode 15

Best,
Killian

@silvercast.nguyen I’ve created ticket #42995

I also updated the link formatting above so the forum doesnt obscure it.

1 Like

Chiming in as another data point. We’re having the same issue with a 2x increase of test runtimes from Xcode 14.3 to Xcode 15.0.

Support ticket: https://support.bitrise.io/hc/en-us/requests/43085

Xcode 14.3 build: Bitrise - Mobile Continuous Integration and Delivery - iOS & Android Build Automation
Xcode 15.0 build: Bitrise - Mobile Continuous Integration and Delivery - iOS & Android Build Automation

I have found that changing the Preferred Capture Format on your test plan from video to screenshot make it faster. Still not as quick but I managed to get a full run of ci without timing out.

Our build+test times have increased from 4-5 minutes to 7-12 minutes. One thing I noticed was that the default “-destination” device in Bitrise was “iPhone 8 Plus”, which doesn’t work with iOS17 builds, so I had to change it. I arbitrarily decided on “iPhone 14”, and I’m wondering if that is part of the reason for increased build times?

I expect this problem will be fixed in Xcode 15.0.1

Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)