Crashlytics DSYM upload issue

Hi,
We use Bitrise to upload DSYM for crashlytics tool during our deployment process.
We have all the steps set to always use the latest version.

Over the last month we have noticed that the DSYMs are uploaded to Crashlytics by the Bitrise step and Crashlytics doesn’t complain about any missing DSYMs.
But when we do actually have a crash, it comes out as hidden.
I tried manually building and uploading the DSYMs and that shows the crash debug info correctly.

Also, when we upload the DSYMs we get the following message in the build logs:

Successfully submitted symbols for architecture arm64 ...
warning: skipping Apple dSYM: ...

Please advise if we are doing something wrong.

Thank you,
Mukund

Hidden crash syntax

Hi @mukund.agarwal,

Do you have the crash in an AppStore / TestFlight build?

From the Crashlytics docs:

There can be certain situations however, when dSYM uploads fail because of unique project configurations or if you’re using Bitcode in your app.

https://docs.fabric.io/apple/crashlytics/missing-dsyms.html

In short, if you use bitcode Apple re-compiles the app on their servers, which means the original dSYM won’t work anymore. You need the dSYMs from the App Store / iTunes Connect directly.

1 Like

Hi @viktorbenei,

We distribute the app via Fabric Beta and yes, we are using Bitcode in our app.
I investigated it further and it seems the issue is with the Bitrise build upload route.

If I upload the build using Fabric client we see the crash info correctly e.g.http://crashes.to/s/dec435af57b

But if we use the latest https://github.com/bitrise-io/steps-deploy-to-bitrise-io and https://github.com/bitrise-io/steps-fabric-crashlytics-beta-deploy steps then the crash info is hidden e.g. http://crashes.to/s/1143e0b4d26

I can’t figure out what has changed in those steps which could be causing this issue as our workflow hasn’t been changed. Please advise and I have also shared the project for “Bitrise support” if you need access.

Thank you,
Mukund

Thanks for the infos @mukund.agarwal!

Do you have Fabric integrated into your iOS app? E.g. as a Pod?

Hi @viktorbenei,

I have added the latest Fabric.framework manually as we are not using cocoapods and they dont have a Carthage based implementation.

Thank you,
Mukund

Is it committed into the repo?

Yes, it is committed and we can see that Bitrise steps are generating the DSYMs.

Perfect!

Can you please try to run the ./Pods/Fabric/upload-symbols command from a Script step? See e.g. https://github.com/fastlane/fastlane/issues/9840 - When running upload-symbols directly I am getting the following error message:

./Pods/Fabric/upload-symbols -a xxxx -p ios /Users/chris/Development/TheApp/TheApp/The App.app.dSYM.zip

Also, can you please copy paste a related bitrise.io build’s URL?

Hi @viktorbenei,

I added the script step but it failed when I used the pods command.
I then changed the “Pods” to “Fabric.framework” as we are not using Cocoapods and it still failed with:

"/var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T/bitrise697172187/step_src/._script_cont: line 8: ./Fabric.framework/upload-symbols: No such file or directory"

The existing step “fabric-crashlytics-beta-deploy” is using the following path:

Submitting DSYM...
  /var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T/bitrise697172187/step_src/Fabric/upload-symbols -a "03313f28417cd1cd3595b819fd900244f0279465" -p ios "/Users/vagrant/deploy/NinjaAppEnterprise.dSYM.zip"

This uploads the DSYMs with the following warning:

2018-01-18 07:15:32.550 upload-symbols[9506:35342] Successfully submitted symbols for architecture arm64 with UUID 72f3b285570033cbb1d8f0ae5dd6751b in dSYM: /var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T///Users/vagrant/deploy/NinjaAppEnterprise.dSYM.zip.unzipped/__dsyms__121541944/72F3B285-5700-33CB-B1D8-F0AE5DD6751B.dSYM
warning: skipping Apple dSYM: /var/folders/90/5stft2v13fb_m_gv3c8x9nwc0000gn/T///Users/vagrant/deploy/NinjaAppEnterprise.dSYM.zip.unzipped/__dsyms__121541944/032500CD-693D-3842-8CF7-4857DACE9C5C.dSYM
w

Is this warning expected?
Also, is the fabric-crashlytics-beta-deploy step using upload-symbols script from the framework I checked in or is it using its own version of the script?
If it’s the later, is it possible that the fabric-crashlytics-beta-deploy step is outdated?

Build page: https://www.bitrise.io/build/b188ad1ae3402567

Thank you,
Mukund

It uses its own, the one in the Step’s repo.

That’s exactly what I suspect to be the case here.

Can you please check the content of the framework (right click -> show content, or similar) on your Mac and see where the upload-symbols is, and try to run the script with that again?

Hi @viktorbenei,

I spent some time looking for upload-symbols script inside the checked in frameworks. Apparently, its not part of the iOS frameworks instead they have another script called uploadDSYMs.
It’s part of the MacOS app. I extracted it out and checked it in with my project and added a script step with the following command:

./FabricMacApp/upload-symbols -a "$FABRIC_API_KEY" -p ios "$BITRISE_DSYM_PATH"

The step did run and uploaded the DSYMs but, Fabric is still unable to decode the crash.
If I run the same upload-sumbols script locally on my machine then Fabric is able to decode the crash.

This means that the DSYMs being generated via the Bitrise route are not right and hence we see the

warning: skipping Apple dSYM:

during DSYMs upload.

I checked the project settings and the highest selectable stack in Bitrise is

Xcode 9.2.x (including betas), installed on macOS 10.12 (Sierra).

All of our machines in our office are on macOS 10.13.2 (High Sierra) running Xcode 9.2 (9C40b).

Could this be causing the incorrect DSYM generation on Bitrise?

Thank you,
Mukund

I guess there’s a chance, but I’d be surprised if OS version would be the issue.

Did you try to run that script on your Mac and see if the warning is present there?

Hi @viktorbenei,

When I run the script on my Mac, I don’t see the warning lines.
I checked the upload-symbols command and they are exactly the same i.e. no further debug argument being sent.

I was checking all the workflow steps and this one:

https://github.com/bitrise-io/steps-xcode-archive/releases

is the one which has been updated frequenty. I am just guessing but may be the new updates in step is causing the DSYM variation.

Do you think that could be the case considering OS version is less likely?

Thank you,
Mukund

Quite unlikely, as that Step (the Xcode Archive one) does not perform any transformation / modification, neither on the code nor on the ipa; all it does is it runs Xcode (xcodebuild) with the params appropriate for its input (Scheme, project file, etc.).


You say there’s no warning if you run the upload from your Mac - that’s a good lead!

Could you please download the dSYM from bitrise.io, from the build’s page, and try to upload that from your Mac? Just to see what happens with the exact same dSYM.

Hi @viktorbenei,

Thanks for the explanation.
I downloaded the Bitrise generated DSYMs and uploaded them via the local script and I do see the same warning.

warning: skipping Apple dSYM: /var/folders/s6/8fb6jx9s6vd7jblm4sd8dhsm0000gp/T//Archive.zip.unzipped/C59C6DE1-A843-30F6-9FDD-21DBA994A038.dSYM
2018-01-19 14:38:44.455 upload-symbols[11935:396348] Successfully submitted symbols for architecture arm64 with UUID 2aabb75482c63ca1bcab03fa1b22a499 in dSYM: /var/folders/s6/8fb6jx9s6vd7jblm4sd8dhsm0000gp/T//Archive.zip.unzipped/NinjaTracking.framework.dSYM

When Xcode generates DSYMs on my machine, I end up with one DSYM per library.
The one generated by Bitrise step seems to be way more than that. And the additional ones seemed to be named with a UUID.
Do I need to change some setting to clean up the DSYM folder?

Thank you,
Mukund

That’s a good lead, so it’s the dSYM itself!

How do you generate the dSYM on your own Mac?

Also, can you please link a related bitrise.io build’s URL? I’d check its logs to see if I can spot something, if that’s possible.

Hi @viktorbenei,

On Mac they are generated when I run the app under the same build scheme.

Please see below link to a related build: https://www.bitrise.io/build/f90998f03363a4bd

Thank you,
Mukund

1 Like

Are you sure that you Archived the same config? Same project file & same Scheme on your Mac?

Did you try to run the build locally with the Bitrise CLI ( How to experiment with Bitrise configs locally, on your Mac/Linux )? If you did not, can you please try to run it?

Hi @viktorbenei

Sorry, for the delay in response.
When running Bitrise CLI locally what is the correct way to make the code signing files available?
I tried downloading the *.mobileprovision and the *.p12 files and placing them in the same directory as the bitrise.yml but, still ran into the following error:

2018-02-07 17:57:04 +0000 [MT] Step failed: <IDEDistributionSigningAssetsStep: 0x7ff55ea76ba0>: Error Domain=IDEDistributionSigningAssetStepErrorDomain Code=0 "Locating signing assets failed." UserInfo={NSLocalizedDescription=Locating signing assets failed., IDEDistributionSigningAssetStepUnderlyingErrors=(

Thank you,
Mukund