Surface Xcode Test Results as Bitrise Output Variable

Feature Requests Steps
1

EDITED:
Feature request to have number of tests passing vs. failing as an output variable for Xcode tests. Or at least the names of the tests that failed.

2

Thanks for the Feature Requests @juliancmg!

How exactly would that look like? Can you provide an example?

Wouldn’t that be a “too long” list? E.g. what if you have 100+ tests?

But to see why exactly that test failed you’d still have to open the related bitrise.io page, right?
I mean, from only “Test X failed” you won’t have enough information why it failed.

P.S.: you can include the build URL in your slack message, with $BITRISE_BUILD_URL

3

Hi Viktor, noted - I didn’t think through the implications of if ALL the tests were reported as in my original feature request. I’ve updated the summary to match maybe more of a regular use case for Slack alerts (high level).

1 Like
4

Thanks! :slight_smile:

Still, do you have any example idea?
All I can think of won’t solve that you still have to go to the build’s page to check the logs/details, which don’t really solve the problem (I guess the problem is that you don’t even want to check the build page, only Slack).

5

Would be nice at least to get numbers of tests ran, how many passed / failed… Other CI platforms I’ve used like Jenkins make that available. Currently BITRISE_XCODE_TEST_RESULT just outputs “failed” which doesn’t tell me much. Yes I can go into the build on Bitrise then click into the test results but it’s nice to have the numbers there in Slack for my team to see.

3 Likes
6

Have you found a solution to the problem?

7

Happy to contribute and add support for this.

To add some technical details it seems relatively straightforward by using xcresulttool to parse the results to output json:

xcrun xcresulttool get --path ./Test.xcresult --format json > result.json

We can parse the output and set the environmental vars.
Here’s some code as example:

    // Get test results using the test summary ID
    cmd = exec.Command("xcrun", "xcresulttool", "get", "--path", xcresultPath, "--id", testSummaryId, "--format", "json")
    output, err = cmd.Output()
    if err != nil {
        fmt.Printf("Error running xcresulttool for test results: %v\n", err)
        os.Exit(1)
    }

    // Parse JSON output to get the test results
    var results TestResults
    err = json.Unmarshal(output, &results)
    if err != nil {
        fmt.Printf("Error parsing JSON: %v\n", err)
        os.Exit(1)
    }

    // Set environment variables for the number of tests passed and failed
    passedCount := results.Summary.TestsPassedCount.Value
    failedCount := results.Summary.TestsFailedCount.Value

    os.Setenv("BITRISE_XCODE_TESTS_PASSED_COUNT", fmt.Sprintf("%d", passedCount))
    os.Setenv("BITRISE_XCODE_TESTS_FAILED_COUNT", fmt.Sprintf("%d", failedCount))
8