Tracking your mobile app’s failing tests is important to identify development process blockers. When tests fail frequently, it indicates that there is a problem with the process that needs to be addressed. Tracking mobile app’s failing tests is crucial for identifying areas where people often wait and spend time debugging, fixing, and retrying tests and builds. This article explains why tracking frequently failing mobile tests is important and how Bitrise Insights can help teams track and analyze their mobile app’s test failure rate. By identifying the most frequently failing tests, teams can improve their testing process, minimize time spent on resolving test failures, and increase their overall efficiency and productivity.
You can find the video version of this content at: https://www.youtube.com/watch?v=hQtcjPhjdHA
Why should you be tracking frequently failing mobile tests?
Similar to tracking build failure rate, it’s important to track your mobile app’s failing tests to identify areas where people often have to wait during the development process.
When your mobile app’s tests fail in a continuous integration (CI) environment, it typically indicates that the build has also failed. This means that someone will need to make code changes or fixes and then retry the build and test it again, leading to a cycle of iterations. Even if someone switches to other tasks while the test and build are running, there will be a context switch when they have to return to the task once the test and build are done.
A high mobile app test failure rate suggests that people frequently spend time debugging, fixing, and then retrying — and waiting for — tests and builds.
Failure rate is especially critical for long tests and builds, as engineers will have to wait even more when they do a fix and then run a new build and test.
By tracking and analyzing where and why your mobile app’s tests fail the most, teams can improve their testing process and make it more resilient over time. Meaning, the time and effort spent on resolving test failures are minimized and the overall efficiency and productivity of your team are increased.
The main goal of tracking your mobile app’s failing tests is to reduce the wait time throughout the development process. If you have tests that fail frequently, it means that sooner or later you’ll have to wait for that test and the related build, fix the issue, and then try it again. As part of this cyclical process, you’ll likely lose time either due to 1) context switching or, 2) because you can’t do anything useful while you’re waiting for the results of that test and build.
How to track frequently failing tests in Bitrise Insights
When you open Bitrise Insights you can start with the Build failure rate chart on the Overview page to find the most frequently failing apps and builds (see: Build failure rate - why and how to find your frequently failing builds? ).
If you’re only interested in tests, you can switch to the Tests page under the Explore section in the left sidebar:
After you open the Tests Explore page switch to the Failure rate tab:
On all of the Tests metric pages you’ll see two charts.
The upper chart always shows you data for the filters you set. The lower chart is called the breakdown chart and it shows data “one level deeper” than what you filtered on.
If you have only a single app in your workspace then Insights will automatically select that application when you open any of the Explore pages. However, if you have multiple apps in your workspace, when you open the Tests Explore page the upper chart it will show you the overall test failure rate of the workspace — based on Test Suite failure rates in all of your apps in the workspace.
The lower chart, in this example, shows you the same data — but on a per-application basis. From this you can see which app’s Test Suites are failing the most:
Here’s a working example.
On the breakdown chart, you can see the application that has the most frequently failing Test Suites in the workspace. When filtering down into that application, you’ll find the per Test Suite breakdown:
The upper chart now shows what is filtered — so in this case, the selected app’s test suite failure rate is being displayed. On the lower, breakdown chart, you can see which test suite failed most frequently:
Let’s filter down to that test suite:
The upper chart now reflects this filtering, and the breakdown chart switched over to per branch failure rate.
Under the graphs, you can see the Test Cases list, which is filtered based on the filters that you set at the top and also on the time range that you set in the top right corner.
UPDATE: the
Test casestab can now be found right above theBreakdown, under the upper chart. If you’re interested why we moved it you can read about it here: Complex Breakdown Charts Improvements
Using this Test cases table, you can find which test case is failing most frequently (Failure rate) or which test case failed the most amount of times (Failure count) in the selected time period, app, and test suite.
Select the most frequently failing test case to continue your investigation:
This will set the relevant filters for you to focus only on this specific test case. As a result, the charts at the top will now only show the failure rate trend of this specific test case:
Under the charts, the Insights filter automatically switched over to the Related test case runs tab:
Using the Related test case runs table, you can see all the previous runs of this specific test case, how long that run was, and whether it was successful or not. Using the buttons on the right side you can quickly jump to the relevant build’s page and to the relevant test report’s page and then continue your investigation there:
Keeping an eye on and improving your test failure rate helps you to reduce wait time during the app development process and to increase the efficiency and productivity of your team.
If you’d have any questions or feedback, please let us know using the Give feedback button in the bottom left corner on any of the Bitrise Insights pages:
Happy Building!
You can find the rest of the “What, Why and How to track” learning series at: what-why-how-track