Surface Xcode Test Results as Bitrise Output Variable

Feature Requests Steps
1

EDITED:
Feature request to have number of tests passing vs. failing as an output variable for Xcode tests. Or at least the names of the tests that failed.

2

Thanks for the Feature Requests @juliancmg!

How exactly would that look like? Can you provide an example?

Wouldnā€™t that be a ā€œtoo longā€ list? E.g. what if you have 100+ tests?

But to see why exactly that test failed youā€™d still have to open the related bitrise.io page, right?
I mean, from only ā€œTest X failedā€ you wonā€™t have enough information why it failed.

P.S.: you can include the build URL in your slack message, with $BITRISE_BUILD_URL

3

Hi Viktor, noted - I didnā€™t think through the implications of if ALL the tests were reported as in my original feature request. Iā€™ve updated the summary to match maybe more of a regular use case for Slack alerts (high level).

4

Thanks! :slight_smile:

Still, do you have any example idea?
All I can think of wonā€™t solve that you still have to go to the buildā€™s page to check the logs/details, which donā€™t really solve the problem (I guess the problem is that you donā€™t even want to check the build page, only Slack).

5

Would be nice at least to get numbers of tests ran, how many passed / failedā€¦ Other CI platforms Iā€™ve used like Jenkins make that available. Currently BITRISE_XCODE_TEST_RESULT just outputs ā€œfailedā€ which doesnā€™t tell me much. Yes I can go into the build on Bitrise then click into the test results but itā€™s nice to have the numbers there in Slack for my team to see.

6

Have you found a solution to the problem?

7

Happy to contribute and add support for this.

To add some technical details it seems relatively straightforward by using xcresulttool to parse the results to output json:

xcrun xcresulttool get --path ./Test.xcresult --format json > result.json

We can parse the output and set the environmental vars.
Hereā€™s some code as example:

    // Get test results using the test summary ID
    cmd = exec.Command("xcrun", "xcresulttool", "get", "--path", xcresultPath, "--id", testSummaryId, "--format", "json")
    output, err = cmd.Output()
    if err != nil {
        fmt.Printf("Error running xcresulttool for test results: %v\n", err)
        os.Exit(1)
    }

    // Parse JSON output to get the test results
    var results TestResults
    err = json.Unmarshal(output, &results)
    if err != nil {
        fmt.Printf("Error parsing JSON: %v\n", err)
        os.Exit(1)
    }

    // Set environment variables for the number of tests passed and failed
    passedCount := results.Summary.TestsPassedCount.Value
    failedCount := results.Summary.TestsFailedCount.Value

    os.Setenv("BITRISE_XCODE_TESTS_PASSED_COUNT", fmt.Sprintf("%d", passedCount))
    os.Setenv("BITRISE_XCODE_TESTS_FAILED_COUNT", fmt.Sprintf("%d", failedCount))
8