Hi,
After upgrading to stack to osx-xcode-15.0.x
Our projects unit test time increases 5x (~60 mins) compared to Xcode 14 stack (~15 min)
We are unable to reproduce this on our mac (M1 Pro), unit tests runs smoothly on our side with same Xcode version
Haven’t been able to create a project to reproduce this issue yet, but my friend who is also using Bitrise Xcode 15 stack is also having this issue.
Any thoughts? Thank you!
@JohnnyTseng that looks odd. Can you provide two build urls to compare with support access enabled please?
Same here, my tests never finish now. Im running unit + ui tests. Builds are timing out.
Some comparison:
Xcode 15:
testAddingItemsToTheBagFromWishlistCarouselWhenSizeIsNotSelected (71.753 seconds)
testAddingItemsToTheBagFromWishlistCarouselWhenSizeIsSelected (223.265 seconds)
testAddingItemToTheBagFromPDP (0.268 seconds)
testAddPromoCodeIsShownAtTheTopWhenFeatureFlagIsOn (326.061 seconds)
testAddPromoCodeIsShownInTheListWhenFeatureFlagIsOff (13.230 seconds)
testApplyingWrongPromoCode (152.068 seconds)
testBagShowsCorrectPricingInformation (0.286 seconds)
testDeliveryInformationCanBeDisplayed (71.072 seconds)
testDonateToCharityAddOnDoesNotAppearWhenFeatureFlagDisabled (87.873 seconds)
Xcode 14.3:
testAddingItemsToTheBagFromWishlistCarouselWhenSizeIsNotSelected (5.816 seconds)
testAddingItemsToTheBagFromWishlistCarouselWhenSizeIsSelected (5.779 seconds)
testAddingItemToTheBagFromPDP (0.061 seconds)
testAddPromoCodeIsShownAtTheTopWhenFeatureFlagIsOn (4.411 seconds)
testAddPromoCodeIsShownInTheListWhenFeatureFlagIsOff (4.682 seconds)
testApplyingWrongPromoCode (7.832 seconds)
testBagShowsCorrectPricingInformation (0.039 seconds)
testDeliveryInformationCanBeDisplayed (4.598 seconds)
testDonateToCharityAddOnDoesNotAppearWhenFeatureFlagDisabled (4.063 seconds)
Using:
xcode15
Computer Name: vm-osx-ventura-15-g2-m1.4core-d050cfd2-80c7-41c5-a25a-4c394782ef94
Locally its fine
Hi, @CraigSiemens the links you shared lead me to sign in page.
Only a bitrise employee would be able to view those.
I’m from Bitrise actually. For some reasons when I click or hover on copy the link url I can only see Bitrise - Mobile Continuous Integration and Delivery - iOS & Android Build Automation
Maybe it’s because discuss forum obscures Bitrise links. Please create a support request https://support.bitrise.io/hc/en-us/requests/new and pass us the ticket number.
We are also experiencing increased build times when updating to Xcode 15
Here is a build log with regular build time ~10 mins (link)
And here is a build log with Xcode 15
Best,
Killian
@silvercast.nguyen I’ve created ticket #42995
I also updated the link formatting above so the forum doesnt obscure it.
Chiming in as another data point. We’re having the same issue with a 2x increase of test runtimes from Xcode 14.3 to Xcode 15.0.
Support ticket: https://support.bitrise.io/hc/en-us/requests/43085
Xcode 14.3 build: Bitrise - Mobile Continuous Integration and Delivery - iOS & Android Build Automation
Xcode 15.0 build: Bitrise - Mobile Continuous Integration and Delivery - iOS & Android Build Automation
I have found that changing the Preferred Capture Format on your test plan from video to screenshot make it faster. Still not as quick but I managed to get a full run of ci without timing out.
Our build+test times have increased from 4-5 minutes to 7-12 minutes. One thing I noticed was that the default “-destination” device in Bitrise was “iPhone 8 Plus”, which doesn’t work with iOS17 builds, so I had to change it. I arbitrarily decided on “iPhone 14”, and I’m wondering if that is part of the reason for increased build times?
I expect this problem will be fixed in Xcode 15.0.1
Fixed: Executing Unit/UI tests from Xcode on the iOS Simulator takes an extended time to launch on first run of the suite. (115187363) (110330776) (FB12237092)
We are still seeing this issue on Xcode 15.1
Anyone having same issue?
I’ve read whole topic on CircleCI forum → Severe performance problems with Xcode 15 - #52 by lukeredpath - Mobile - CircleCI Discuss
Based on the comment from the CircleCI dev team the issue seems to be resolved with the introduction of macOS 14 Sonoma + Xcode 15.X
Since currently, the Xcode 15.X+ images from Bitrise uses macOS 13 Ventura, we were forced to downgrade our UI Tests workflow to Xcode 14.3. On Xcode 15+ all of our UI tests are failing randomly and execution time is really long.
We need an image that would include macOS Sonoma - this probably resolves the issues.
When can we expect for those stacks to be available?
Hi p.walczak,
You can try out the 15.3 with edge updates. It’s using macOS 14.1.2 and Xcode 15.