Testing steps now automatically populate BITRISE_FLAKY_TEST_CASES when retries are enabled, identifying which specific tests are behaving inconsistently.
What this fixes
When tests fail intermittently, you need to know which specific test cases are unstable. Previously, you could only see overall pass/fail results - not identify individual flaky tests that might be causing reliability issues in your test suite.
How it works
When any of these steps run with retries enabled, they automatically track individual test results:
- Xcode Test for iOS (ID: xcode-test)
- Android Unit Test (ID: android-unit-test)
- iOS Device Testing (ID: virtual-device-testing-for-ios)
- Virtual Device Testing for Android (ID: virtual-device-testing-for-android)
These steps populate BITRISE_FLAKY_TEST_CASES with tests that failed at least once but also passed at least once during retry attempts.
Environment variable format
Contains a newline-separated list of flaky test identifiers:
- TestTarget_1.TestClass_1.TestMethod_1
- TestTarget_1.TestClass_1.TestMethod_2
- TestTarget_1.TestClass_2.TestMethod_1
- TestTarget_2.TestClass_1.TestMethod_1
When to use this
- fail builds on flaky tests: Check if the variable is non-empty and fail the build
- Send alerts: Notify your team when flaky tests are detected
- Generate reports: Log flaky test names to track patterns over time
- Prioritise fixes: Use the output to identify which tests need stabilisation
Usage examples
Fail build if flaky tests detected:
steps:
...
- script:
run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'
inputs:
- content: |-
#!/bin/env bash
echo "Build failed due to flaky tests:"
echo "$BITRISE_FLAKY_TEST_CASES"
exit 1`
Send Slack notification:
steps:
...
- slack:
run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'
inputs:
- webhook_url: $SLACK_WEBHOOK_URL
- text: |-
⚠️ Flaky tests detected in build $BITRISE_BUILD_NUMBER:
$BITRISE_FLAKY_TEST_CASES
Create a GitHub issue automatically:
steps:
...
- script:
run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'
inputs:
- content: |-
#!/bin/env bash
ISSUE_BODY="Flaky tests detected in build $BITRISE_BUILD_NUMBER:\n\n\`\`\`\n$BITRISE_FLAKY_TEST_CASES\n\`\`\`"
curl -X POST \
-H "Authorization: token $GITHUB_TOKEN" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/owner/repo/issues \
-d "{\"title\":\"Flaky tests - Build $BITRISE_BUILD_NUMBER\",\"body\":\"$ISSUE_BODY\"}"
Log to analytics service:
steps:
...
- script:
run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'
inputs:
- content: |-
#!/bin/env bash
echo "$BITRISE_FLAKY_TEST_CASES" | while read -r test_case; do
curl -X POST https://analytics.example.com/events \
-H "Content-Type: application/json" \
-d "{\"event\":\"flaky_test\",\"test\":\"$test_case\",\"build\":\"$BITRISE_BUILD_NUMBER\"}"
Save to build artifacts:
steps:
...
- script:
run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'
inputs:
- content: |-
#!/bin/env bash
echo "$BITRISE_FLAKY_TEST_CASES" > flaky_tests_$BITRISE_BUILD_NUMBER.txt
cp flaky_tests_$BITRISE_BUILD_NUMBER.txt $BITRISE_DEPLOY_DIR/
Count and categorize by test target:
steps:
...
- script:
run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'
inputs:
- content: |-
#!/bin/env bash
echo "flaky test summary:"
echo "$BITRISE_FLAKY_TEST_CASES" | cut -d'.' -f1 | sort | uniq -c | while read count target; do
echo " $target: $count flaky tests"
Set environment variable for downstream steps:
steps:
...
- script:
run_if: '{{getenv "BITRISE_FLAKY_TEST_CASES" | ne ""}}'
inputs:
- content: |-
#!/bin/env bash
FLAKY_COUNT=$(echo "$BITRISE_FLAKY_TEST_CASES" | wc -l)
envman add --key FLAKY_TEST_COUNT --value "$FLAKY_COUNT"
echo "found $FLAKY_COUNT flaky tests - available as \$FLAKY_TEST_COUNT"
Prerequisites
- Retries must be enabled in the testing step
- Available immediately after the testing step completes
Happy building 