How to have a separate cache per workflow?

Question & Answer
1

What I want is to have two workflows each with a different cache.

Using printenv on a macOS box I found $BITRISE_CACHE_DIR and $BITRISE_CACHE_API_URL, also some scripts use $BITRISE_CACHE_INFO_PATH but it’s not clear whether those variables can be changed to point to separate caches.

This is how I’ve setup bitrise.yml in my repository:

format_version: "4"

default_step_lib_source: https://github.com/bitrise-io/bitrise-steplib.git

project_type: other

#trigger_map:
#  - push_branch: '*'
#    workflow: primary
#  - pull_request_source_branch: '*'
#    workflow: primary

app:
  envs:
    - SCONS_CACHE: $HOME/.scons_cache
    - SCONS_CACHE_LIMIT: 1024
    - CC: clang
    - CXX: clang++
    - GODOT_TARGET: osx
    - TOOLS: yes

workflows:
#  primary:
#    steps:
#      - activate-ssh-key@3.1.1:
#          run_if: '{{getenv "SSH_RSA_PRIVATE_KEY" | ne ""}}'
#      - git-clone@3.6.2: {}
#      - script@1.1.5:
#          title: Do anything with Script step
#      - deploy-to-bitrise-io@1.3.9: {}

  build:
    steps:
      - cache-pull:
          inputs:
            - cache_paths: |-
                $SCONS_CACHE
      - script@1.1.5:
          title: Build script
          inputs:
            - content: |-
                echo $BITRISE_CACHE_DIR
                echo $BITRISE_CACHE_API_URL
                echo $BITRISE_CACHE_INFO_PATH
                mkdir -p $SCONS_CACHE
                misc/bitrise/scons-local-osx.sh
                scons -j 2 CC=$CC CXX=$CXX platform=$GODOT_TARGET TOOLS=$TOOLS verbose=yes progress=no;
      - cache-push:
          inputs:
            - cache_paths: |-
                $SCONS_CACHE
2

Hi @rraallvv,

Thanks for the question! :slight_smile:

What I want is to have two workflows each with a different cache.

Unfortunately that’s not possible. Cache storage is allocated for branches, not workflows. It’s automatic, every branch when pushes a cache to the API will create its own cache (will not overwrite another branch’s cache), but it’s always allocated by branch.

Related docs: Redirecting… - Bitrise Docs

If you’d have any questions just let us know! :slight_smile:

3

That said, checking your example bitrise.yml in your question, are you sure you need/want per workflow cache? It should definitely work with just per-branch cache, at least in the example I don’t see any reason why it might not.

If you have a specific error please copy paste the related build’s bitrise.io URL and we’ll check it ASAP! :slight_smile:

P.S.: in the example you have cache_paths in the “Cache:Pull” step too; that step does not have that input, the cache:pull step simply moves the caches to the original location, the one which was defined in the Cache:Push step.

In short you control the cache config (what to cache) in the Cache:Push step, which stores these infos in the cache archive; the Cache:Pull step simply reads that meta-info and uncompresses the cache archive to the original locations/paths.

4

I was hoping maybe I could have separate caches for macOS and iOS for the same branch. I’ve seen parallel builds are not possible out of the box but I’ll try the workaround described by @koral. Regarding the separate caches issue, I’ll have to use separate folders for each build then. I’m going to remove the input in cache-pull. Thanks a lot.

1 Like
5

Indeed, not yet available as a built in feature, but definitely planned for the near future. Please vote & comment at Parallel Builds: trigger multiple workflows (builds) at the same time - build stages - fan-out fan-in support - #20 by viktorbenei to bump its priority (although at this point it definitely is planned for early next year).

If you have any questions just let us know! :slight_smile:

6

@viktorbenei thanks for the info.

1 Like
7

Hey,

I would like to bump this thread with more sophisticated usage pattern:

I am using job with multiple workflows. The entry workflow is triggering few others via build-router-start. That is one per ios and one per android. I would love to be able to have separate caches per workflow. For instance: I want to cache Pods for ios workflow but I don’t care about pods in android flow.

Any way to achieve this today? Perhaps something changed since previous answer?

8

Hi @max!

There weren’t any changes to the main feature, the link posted above by @viktorbenei is up to date, so you can check out the current detailed behaviour here: https://devcenter.bitrise.io/caching/about-caching/

But in short cache does not work per workflow, it does per branch, so you could only achieve this if you were to separate your ios and android code you build per branch too.

9

I believe caching per Branch will not be super useful, in our case we create branches for each feature and and we make builds from those branches (without pull request)
if we have to download again from scratch for every new branch we create before having cache available, then it will not really save us much time.
having cache per workflow or by App would save lot of time and build resources
Is there at least a possibility to define in cache pull step where the cache should be pulled from (ex pulling from develop’s cache while building another branch)?

10

Hi @AlaaEddineCharbib! This is not possible at the moment, but thanks for your feedback! I’ve made sure we take a note of it.

11

This feature is highly requested for us, but we’re actually imagining it slightly more granular. If you could simply name a cache, and thus push and pull named caches, you’d solve the issue, at least according to our needs. We currently want to have a cache for our base workflow with npm packages, a cache for our iOS workflow and a cache for our Android workflow. I realized that scoping it per workflow would work perfectly fine here as well, but I think the naming would provide a large amount of flexibility to the caching solution.

Thanks for listening to the feedback though!

3 Likes
12

+1 I think having the ability to push/pull cache by a specific key and be able to use it through different branches would make caching way more flexible :slight_smile: (this also how it’s normally handled by some other other CI/CD platforms that offers cashing solutions)

13

Isn’t it extremely common to have branches for every feature separately? What use case would this caching system be helpful for? First merging and then testing?