The Test Reports Add-on always displays an incorrect duration for my iOS tests

Archive Tooling Issues
1

Whenever looking over the duration of my iOS test cases in the Test Reports add-on, regardless if I’m looking at the whole test execution duration or the individual test case duration, it’s always incorrect.
The whole test run takes about 30 mins according to the Bitrise logs, yet Test Reports add-on says it took 2 mins to execute. Regarding the individual test cases duration, each test is reported to take a few seconds, even though the logs reflect that the test case took much more to be executed.
Did I set something wrong or has anyone any idea how to solve this issue?

2

I’m having the same issue.

3

Hi @stursunov @rbaraboi
Thank you for reaching out to Bitrise Support .
Could you please let me know which step you are using for your iOS Testing?
Also, if you could send me the build URL.

Thanks
Chaitanya

4

Hi @chaitanya.tanna,
I’m using XCode Test for iOS step, set to always use the last version.
Here is a link to my build URL.

5

The same. Looks like the gem which you do use to parse result, doesn’t map value correctly for Test Reports.

6

Hi @rbaraboi @stursunov
And is it possible if the test reports was configured correctly to match the test from builds?

Thanks
Chaitanya

7

Hi @chaitanya.tanna
I’m not sure I understand your question. If you’re suggesting that the test reports may actually match the real duration of the tests, note that this is not the case. None of the displayed durations in the Test Reports add-on are correct.

Thanks,
Razvan

8

Everything is fine. Setup test case, download Xcode test result file and then compare time from it with Test Reports. You will see mismatch. You’re using Gem or 3rd party tool to parse xcode test result file and generate Test Reports, at this step something is wrong. It doesn’t taking correct executed time. If you would you like we can set up conference call and I will do demo for you.

9

Hey! I’m sorry to see so many folks are having issues with this. I’ve been asked to investigate.

@rbaraboi - Could you please enable support access on your account and send me a few links to builds which are reflecting incorrectly in test reports?

10

@matthew.jones support access is now enabled. Below are some links you requested:

11

Hey! Could you try those links again? All of them just point to the Bitrise sign-in page. :slight_smile:

Matthew
Senior Customer Engineer

13

@matthew.jones it seems that whenever I paste a link for a build, it will redirect to the sign-in page by default. I will only paste the build data, which should work if entering https://app.bitrise.io/ before. Please check these ones:

  • build/dd6e720d9822a085

  • build/df259e8c81ef1c62

  • build/401efc7fd7a1776d

14

Yikes! Okay, that’s definitely a bug and I’m filing a report.

Also, I didn’t realize that Discuss redirected the build links. I apologize for the inconvenience on that and will keep it in mind when asking for builds in the future.

Matthew
Senior Customer Engineer

15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.