Cache flags

Will be good to have the possibility to have some flags from the cache addons.
I imagine a workflow like this:

On cache:pull I specify the same configuration of cache:push, where I say which lock files use:

node_modules -> package.json
bower_components -> bower.json

After the pull I will get some environment variables like:

BITRISE_CACHE_NODE_MODULES
BITRISE_CACHE_BOWER_COMPONENTS

each of these variables contains a simple 1 or 0, where a 1 means “something is changed” and a 0 is “nothing is changed”.
So if I pull node_modules, and nothing is changed I don’t need to run npm install and I can skip that expensive step.

if [ BITRISE_CACHE_NODE_MODULES == 1 ]; then
  npm install
fi
1 Like

Awesome idea, thanks for creating the #feature-requests @redant!

A technical note: I’d personally rather implement a mechanism like the one we did in the Carthage step, where the step itself can detect whether the caches are available and sufficient, and simply skip the “install/bootstrap” command when not needed.

In this case I’d add this logic to the NPM step (GitHub - bitrise-steplib/steps-npm: Bitrise step for running npm commands), in case the command is “install” (similar to the “bootstrap” command in case of the Carthage step)

In short, the step would work the way mentioned by @redant:

But instead of checking an environment variable, it should inspect the required files to determine whether an npm install is required or not. E.g. in case of the Carthage step, “cache facts” are stored by the step into a file, including the resolve/lock file’s content or hash, as well as tool versions (e.g. install might be required if the NPM version changes since the cached version of the deps).

For the cached npm install there’s only one downside.
When there’s a postinstall script.

If I choose to not run npm install because a valid node_modules exists, I may want to run another script, like npm run postinstall that usually contains something like bower install or some action that I want to do using one or more packages I’ve previously installed (like gulp for example).

I think that a general cache step can be useful, maybe a “cache:if”

1 Like

Great idea!

The problem with an environment variable like BITRISE_CACHE_NODE_MODULES is how you would define this variable, or what would define this variable?

The Build Cache can be configured to cache any directory. Maybe we should add one more “special item” to the format, like the cache indicator file (/what/to/cache -> /only/if/this/file/changed)? Something like

/what/to/cache -> /only/if/this/file/changed => ENV_TO_SET_IF_CACHED

?

In case of the Carthage step it does handle this internally, so you don’t have to configure anything else, but it only skips the command if the cache is available, it does not perform another / alternative command instead.

Maybe the npm step could do the same, with an optional “run this command if cache available:” input? That way you could set install as the default command, and e.g. run postinstall for the “if cache available” input.

WDYT @redant?

Well the idea was to use BITRISE_CACHE_${FOLDER_NAME}

But with a syntax where you can specify the name of the variable is even better

1 Like

I have made a script that will do nearly what cache:if is supposed to:

#!/bin/bash

# Automatic exit on error
set -e

# Run a cached NPM install
# Arguments
# 1: Directory of the package.json
# 2: Cache file
function runCachedNPMInstall {
    originalDir=$(pwd)
    cd $1

    # Create a unique object with the keys of dependencies and devDependencies of the package.json
    currentPackageJSONVariables=$(jq -cM '.dependencies + .devDependencies' package.json)

    # Read the dependencies from the cache
    cachedPackageJSONVariables=''
    if [ -f "$2" ]; then
       echo "cached npm file exists"
       cachedPackageJSONVariables=$(cat $2)
    fi


    # If any dependency is changed on package.json
    if [ "${currentPackageJSONVariables}" != "${cachedPackageJSONVariables}" ]; then
        # Install ignoring root and postinstall
        npm config set loglevel warn
        echo "Installing package.json deps"
        npm install --unsafe-perm --ignore-scripts
        echo "${currentPackageJSONVariables}" > $2
    else
        echo "Using package.json cache"
    fi

    cd ${originalDir}
}

runCachedNPMInstall "${BITRISE_SOURCE_DIR}" "${BITRISE_SOURCE_DIR}/cached.npm.json"

What the script does is simple:

  1. Extract and merge the fields dependencies and devDependencies from the package.json file (this means that if you make a dependency as a devDependency the cache will not be invalidated)
  2. Check with a file that comes from the cache if the “new” deps are equal to the “old” deps
  3. In case are different run npm install in silent mode, and disabling postinstall scripts

For use this script on cache:push I put:

$BITRISE_SOURCE_DIR/node_modules -> $BITRISE_SOURCE_DIR/package.json
$BITRISE_SOURCE_DIR/cached.npm.json

The script requires jq that can be installed using:

DEBIAN_FRONTEND=noninteractive apt-get install -qq -y jq > /dev/null

I got a speed-up of 2 minutes (originally npm install required ~150 seconds):

+---+---------------------------------------------------------------+----------+
| âś“ | cache-pull@0.9.2                                              | 26 sec   |
+---+---------------------------------------------------------------+----------+
| âś“ | Check cache and install deps                                  | 1.79 sec |
+---+---------------------------------------------------------------+----------+
| âś“ | cache-push@0.9.4                                              | 3.19 sec |
+---+---------------------------------------------------------------+----------+

WDYT @viktorbenei

3 Likes

Awesome, I love it!

Note: instead of DEBIAN_FRONTEND=noninteractive apt-get install -qq -y jq > /dev/null you can declare jq as a dependency for a step: http://devcenter.bitrise.io/tips-and-tricks/install-additional-tools/#advanced-option-use-deps-in-bitriseyml

deps:
  brew:
  - name: jq
  apt_get:
  - name: jq

You can add brew as well, to make it work on Mac too :wink:

Btw this is quite similar to what the Carthage step does, but instead of getting the dependency list it uses the full “resolve” file as the cache indicator source, combined with Swift version.