Multiple custom-test-results-export steps duplicates test results

Question & Answer Steps
1

I’ve got a pipeline that runs three sets of iOS UI tests (different test plans)

If I have a custom-test-results-export step after each test run I end up with 6 test results listed on the Test Reports page. The first test run gets listed thrice, the second run twice and the third run once, so it’s uploading prior tests each time. This is a little odd as I’m clearing the test results folder between test runs, so I guess the step must be caching the prior results somewhere and including them in subsequent uploads.

I tried with just a single custom-test-results-export step after all the test runs but that results in just one of the results being selected as the step only wants to handle one set of test results (Allow for multiple matches in step-custom-test-results-export step could resolve this specific issue)

Example of the step yml (bits in angle brackets redacted for simplicity):

    - script@1.2.0:
        title: "UI Tests"
        inputs:
        - content: |-
            # Delete the previous logs (to prevent duplicates being uploaded)
            rm -rf <log path>
            <run test plan>
    - custom-test-results-export@0:
        title: "Upload test results"
        inputs:
        - base_path: "$BITRISE_SOURCE_DIR/<log path>"
        - search_pattern: "*"
        - test_name: <test name>
    - deploy-to-bitrise-io@2:
        title: "Archive xcresult"
        inputs:
        - deploy_path: "$BITRISE_SOURCE_DIR/<log path>"
        - is_compress: true
        - is_enable_public_page: false
        - notify_user_groups: none
        - zip_name: <test name>_xcresult
2

It also seems to have completely passed me by but all the duplicated Test Reports page entries are actually the same, just with a different name, so each time it’s uploading the same test report multiple times :man_shrugging:

3

I’ve got a workaround that handles the issue:

    - script@1.2.0:
        title: "UI Tests"
        inputs:
        - content: |-
            # Delete the previous logs (to prevent duplicates being uploaded)
            rm -rf <log path>
            # Delete the previous exports from prior custom-test-results-export steps
            rm -rf $BITRISE_TEST_DEPLOY_DIR/*    <---------  new workaround!
            <run test plan>
    - custom-test-results-export@0:
        title: "Upload test results"
        inputs:
        - base_path: "$BITRISE_SOURCE_DIR/<log path>"
        - search_pattern: "*"
        - test_name: <test name>
    - deploy-to-bitrise-io@2:
        title: "Archive xcresult"
        inputs:
        - deploy_path: "$BITRISE_SOURCE_DIR/<log path>"
        - is_compress: true
        - is_enable_public_page: false
        - notify_user_groups: none
        - zip_name: <test name>_xcresult

The custom-test-results-export step puts export files in $BITRISE_TEST_DEPLOY_DIR and doesn’t clear them out, so subsequent uses of it seems to favour the first file in there (the first test run’s export).

Obviously only a workaround that works while the step’s behaviour remains the same, so it would be great to get the step updated to clean up after itself or identify the files that need uploading.

EDIT: the workaround is better if you do the rm -rfing in a separate unskippable step before the tests run as otherwise any failures will result in all the export steps running still (you don’t want to skip them if their test run step has failed) and uploading cloned/duplicated test reports

1 Like