Test Reports wasn't configured correctly

[steveroy]

Hi! I’m evaluating Bitrise and set up a macOS project using the assistant. The project builds successfully except for the Test Reports add-on that gives the message “Test Reports wasn’t configured correctly. Please make sure you configure it in the Workflow Editor.”

I verified the workflow and the steps against the documentation as well as the video showing how to set it up. Everything seems as it should be.

The build log gives a clue as to something going wrong. It says it ran 12 tests:

Executed 12 tests, with 0 failures (0 unexpected) in 0.164 (0.184) seconds
▸ Test Succeeded

But then in the deploy step, it says it uploaded 0 of them:

Upload test results
- uploading (0) test results
Success

The steps used in my workflow are:

• Xcode Test for Mac, version 1.4.1
• Deploy to Bitrise.io, version 1.10.0

Is it possible that the test step is not generating artifacts as the deploy step expects them? Or maybe the deploy step doesn’t support macOS project tests?

[Roland-Bak]

Hi there @steveroy

Could you please send me a build URL of the related build and enable Support Access on the Settings tab of the app (Enabling the Bitrise Support user for your app | Bitrise DevCenter), so that I may take a closer look?

[steveroy]

Hi @Roland-Bak and thanks for following up!

I turned on Support Access and here’s a link to the latest build. You should be able to see in the log that the Deploy step uploads 0 test results even though 12 tests ran successfully above.

https://app.bitrise.io/build/a0f945c828b5326e#?tab=log

[Roland-Bak]

Thanks for your patience, @steveroy!

By Default, the Test Reports plugin works with the following steps:

• Xcode Test for iOS .
• Android Unit Test .
• iOS Device Testing .
• Virtual Device Testing for Android .
• Flutter Test.

Since you are using Xcode Test for Mac, you need to do a bit of working around to make it available for Test Reports.
Have you tried exporting your test results via this guide: Exporting to Test Reports from any Step | Bitrise DevCenter ?

Seems like including an Export test results to the Test reports add-on step should help

[steveroy]

Thanks @Roland-Bak! That makes perfect sense. I gave that a go and still am not getting anything in Test Reports. It’s probably a configuration. In particular I suspect the value of Path where custom test results reside is incorrect. I tried using BITRISE_XCODE_TEST_RESULT, which is what’s output by the Xcode Test for Mac step. Still no go. Is there a variable that points to the derived data dir holding my project build output and xctest results?

Here are the relevant bits from my current yml:

- xcode-test-mac@1:
    inputs:
    - project_path: "$BITRISE_PROJECT_PATH"
    - scheme: "$BITRISE_SCHEME"
- custom-test-results-export@0:
    is_always_run: false
    inputs:
    - search_pattern: "*"
    - base_path: "$BITRISE_XCODE_TEST_RESULT"
    - test_name: Unit Tests
- deploy-to-bitrise-io@1: {}

[Roland-Bak]

Hi there @steveroy!

The catch is that the step does not output anything else than a flag indicating if all tests have succeeded.
-resultBundlePath is the way to go, you should specify a path there that is accessible by the Export test results to the Test reports add-on
$ BITRISE_DEPLOY_DIR seems perfect for the purpose

[steveroy]

Hey @Roland-Bak,

I’m sorry but I cannot for the life of me get this to work.

• As you suggested, I configured the Xcode Test for Mac step to include the -resultBundlePath option for xcodebuild.
• Then I configured the Export Test Results step to look for test results at that path.

Giving me this yml:

- xcode-test-mac@1:
    inputs:
    - project_path: "$BITRISE_PROJECT_PATH"
    - xcodebuild_options: "-resultBundlePath $BITRISE_DEPLOY_DIR/TestResults"
    - scheme: "$BITRISE_SCHEME"
- custom-test-results-export@0:
    is_always_run: false
    inputs:
    - search_pattern: "*"
    - base_path: "$BITRISE_DEPLOY_DIR/TestResults.xcresult"
    - verbose_log: 'yes'
    - test_name: Unit Tests
- deploy-to-bitrise-io@1: {}

I see the TestResults.xcresult bundle being created at the path above. However it’s a macOS bundle, which is a directory, and the Export Test Results step appears to only work with a single file because it outputs “Pattern matched more than one file, will use the first match”, which is the directory, and then deploy fails because it can’t deploy a directory.

Am I missing something obvious? Is this not supposed to work with a macOS build? I keep searching for an example of doing exactly this but none of the samples I find are for a macOS app build.

You can see the verbose output of my last attempt here:
https://app.bitrise.io/build/d0ff0d1dae35da79

[bitce]

Hi @steveroy!

Have you tried providing the path to the .plist file insbtead of the whole xcresult?

[steveroy]

Thanks for the suggestion, @bitce. That itself did not work but I dug some more and figured it out. The issue was the path I had given to -resultBundlePath. I had omitted the .xcresult extension:

- xcodebuild_options: "-resultBundlePath $BITRISE_DEPLOY_DIR/TestResults"

As a result, Xcode outputted the test results at TestResults.xcresult but then made a symlink at TestResults since that’s what I had asked for. It went downhill from there as the symlink had no extension so looked like a plain directory to the rest of the process. After adding the .xcresult extension to -resultBundlePath, now the export step works and I see my tests in the Test Reports add-on.

For future reference, in case anyone else runs into this issue and searches this forum, here’s my yml:

- xcode-test-mac@1:
    inputs:
    - project_path: "$BITRISE_PROJECT_PATH"
    - xcodebuild_options: "-resultBundlePath $BITRISE_DEPLOY_DIR/TestResults.xcresult"
    - scheme: "$BITRISE_SCHEME"
- custom-test-results-export@0:
    inputs:
    - search_pattern: "*"
    - base_path: "$BITRISE_DEPLOY_DIR/TestResults.xcresult"
    - test_name: Unit Tests
- deploy-to-bitrise-io@1: {}

Thanks @Roland-Bak for bearing with me!