Output from multiple instances of the same step

[eliot_pear]

Is there a good pattern for differentiating between outputs of multiple instances of the same step? I’m working on a step right now to extract the version for any given node.js package.json so I was trying to figure out how to make it more generic/modular. Right now it uses envman to export a single output variable. But what if you had more than one package you wanted to inspect? Is there a good way to hang on to seperate outputs for each of them?

I had a few ideas but I’m not really in love with any of them:

1. The exported envvar could have a configurable name, probably using some nasty bash eval hack. This would make the outputs defined in the step.yml misleading at best, assuming it even works.
2. The exported envvar could be a serialized data structure that is appended to on subsequent runs (My first thought was to use a bash array.) But this would not work very well for the purpose of having envvars that can be easily routed as inputs to other scripts.

[viktorbenei]

Great question @eliot_pear!

There are two things worth to discuss here:

1. How to expose a list of outputs from a Step, and consume that in other steps
2. How to run the same step multiple times, and have access to the output of all the same steps, instead of overwriting each others

How to expose a list of outputs from a Step

For the first point we think this should be the official way to do it: https://github.com/bitrise-io/bitrise/blob/master/_docs/step-development-guideline.md#inputs-which-can-accept-a-list-of-values

TL;DR;

You should postfix the input ID with _list (e.g. input_path_list), and expect the values to be provided as a pipe character separated list (e.g. first value|second value)

We’ll soon start to test this solution through our core steps, so far this solution seems to be the best way to handle lists of outputs, e.g. if a Gradle Runner step generates more than one .apk file.

How to reference outputs of separate steps which expose outputs with the same environment variable key

The solution for this right now, which works but not exactly a clean solution, is to use a Script step, right after the step which generated the output which would be overwritten later, and in the Script step “copy” the value to a new environment variable. Docs: Environment Variables - Bitrise Docs

This could be released as a new step too, see: [Step] "bridge" an environment variable - assign the value of one Step's output to another Environment Variable

What we plan to do, is to add this as a built in feature in the Bitrise CLI / bitrise.yml . In short, you could do something like what you described:

@eliot_pear

The exported envvar could have a configurable name

but instead of doing any eval hack, and hopefully without making this “misleading”, we’d propose a new item/syntax for step outputs. In bitrise.yml right now you can’t overwrite outputs (it makes sense, as you can’t change how a step generates an output), but we would add an “alias” or “name” syntax, so that you could define an alternative name for the environment variable.

E.g.:

- git-clone:
    inputs: []
    outputs:
    - GIT_CLONE_COMMIT_HASH: SAVE_IT_INTO_THIS_ENV_VAR

This would indicate for the Bitrise CLI that you want to store the GIT_CLONE_COMMIT_HASH output of the Git Clone step into the SAVE_IT_INTO_THIS_ENV_VAR environment variable, instead of into GIT_CLONE_COMMIT_HASH. Of course if you don’t provide an output item for the step, then GIT_CLONE_COMMIT_HASH will be generated and it won’t be modified, like the way it works today.

This would be a pretty similar solution to what I described above as “The solution for this right now”, but built into the Bitrise CLI and into the bitrise.yml format specs, so that you don’t have to use a separate step for this.

Whether this is a copy or a rename of the GIT_CLONE_COMMIT_HASH env var is still up for discussion (whether GIT_CLONE_COMMIT_HASH should still be populated in this case, or only SAVE_IT_INTO_THIS_ENV_VAR). Personally I think “rename” should be enough, if you define an alternative name there’s no need to keep the value in the original output as well.

WDYT @eliot_pear?

[eliot_pear]

I like the solution of incorporating this functionality into the bitrise.yml. “Rename” seems like the best option here as the final value of the original exported variable might be misleading as it will be dependent on the order of which step executed last. I also like that this solution doesn’t require additional overhead on the part of the step, and will work without having to update existing steps.

In addition it might be nice to get a warning in the logs if envman tries to set an envvar that is already set as a clue that the output of a step should be re-routed by the consumer. (perhaps with a way for the script to configure envman to suppress the warning in case this behavior is by design.)

Also, if the idea is to support multiple steps that can interact with “list” inputs and outputs, perhaps a tool can be added to the bitrise cli to ease interaction with these rather than expecting each script to roll its own. I don’t have a use case for this right now but something to think about. This is more of a problem in bash as its collection handling is rather primitive, I’d expect higher-level languages to mostly have easy-to-use split and join functions. I guess if this were only needed by bash scripts, this could be as simple as a versioned, shared library which is part of the cli environment that can be source'd by any script and have a function that can be called to parse a list string into a bash array, as well as one that will convert a bash array back into a string. I took a crack at writing these routines myself, feel free to use them.

It would also be nice to put some other common helper functions into this library such as validate_required_input, echo_fail, etc. rather than having to copy-paste them into every step script.

[viktorbenei]

@eliot_pear

“Rename” seems like the best option here as the final value of the original exported variable might be misleading as it will be dependent on the order of which step executed last.

I agree, I think we’ll go with “rename” and see if there’s any use case for “copy”.

@eliot_pear

In addition it might be nice to get a warning in the logs if envman tries to set an envvar that is already set as a clue that the output of a step should be re-routed by the consumer

Good idea! ;)[quote=“eliot_pear, post:3, topic:538”]
if the idea is to support multiple steps that can interact with “list” inputs and outputs, perhaps a tool can be added to the bitrise cli to ease interaction
[/quote]

We thought about this, but so far we couldn’t find a better solution. If you have any idea please let us know![quote=“eliot_pear, post:3, topic:538”]
This is more of a problem in bash as its collection handling is rather primitive, I’d expect higher-level languages to mostly have easy-to-use split and join functions.
[/quote]

The solution for this right now is to use a different language for the step (preferably Go as it has the best CLI support), which has proper support for parsing and processing these data. I’d say that if you have to process a list of inputs then you’ll eventually end up writing the step in a more advanced language than Bash anyway, especially if you have to handle errors, retries, etc. For example, while you might be able to write a simple Bash step for uploading an app to e.g. HockeyApp, if you have to do that for a list of inputs and preferably handle upload errors and retry properly, that can be quite a challenge in Bash. But again, if you have an idea, we’re always happy to discuss

@eliot_pear

It would also be nice to put some other common helper functions into this library such as validate_required_input, echo_fail, etc. rather than having to copy-paste them into every step script.

We have that, for Go - GitHub - bitrise-io/go-utils: Common, utility packages for Go

[eliot_pear]

@viktorbenei One step ahead of you on the bash implementation… did you miss the link on my previous post?

Certainly bash is not the right hammer for certain nails (especially for more complex scenarios, ex. requiring parallelism like you mentioned above) but I think shell scripts also have unrivaled expressiveness for interacting with other command-line tools. I think it’s a good idea to provide some step interop facilities and then allow the implementor to decide which language is best for the job.

My pure bash 3.x+ implementation is below, feel free to use it (although if you do take my suggestion, I’d reccomend giving the ‘bash library’ a versioned path/name so that new versions could be released without having to worry about maintaining backwards compatibility):

https://gist.github.com/fadookie/4a08de38784dfdaa1f31952c1792d4dd

[viktorbenei]

@eliot_pear

Certainly bash is not the right hammer for certain nails (especially for more complex scenarios, ex. requiring parallelism like you mentioned above) but I think shell scripts also have unrivaled expressiveness for interacting with other command-line tools.

Ohh, I definitely agree with you! The only thing is, why we don’t plan to provide a bash utility library (but that should not stop anyone else to do so!!) is that we could not properly maintain it (as we always use Go for more complex steps, and we do have a common Go library actively maintained by us), and that we do think that in most more complex cases Bash is simply not the right tool (e.g. error handling is way too complicated once your bash script grows over a couple of tens of lines of bash code).

But again, this should not stop anyone from creating a utility bash lib/code!

Thanks for sharing your util code @eliot_pear (looks pretty neat btw, I don’t think I could have written it the way you did - you most likely have more Bash experience than me ;)), I’m sure it’ll help other step devs!

[viktorbenei]

Ohh, one more thing: if you want to feel free to create a “proper” repository for these Bash util scripts, we can highlight it in the docs & here on discuss.bitrise.io for other step devs if you want to. Just make sure it’s simple to integrate into a new step (quick guide about how to include it).

[eliot_pear]

@viktorbenei Thanks! I’m curious if you have thoughts on how such a lib could be included in the step without violating the “Do not use submodules, or require any other resource downloaded on-demand” guideline. People could just copy this file into their step repo, but how will they know there is an update? And is the update process to just copy-paste the whole library again into their step repo?

That’s why I was thinking it might make sense to be a part of the CLI environment and function somewhat like system-provided C headers. So the script could do something like source "$BITRISE_CLI_SHELL_LIBS/latest/bash/list.sh" or source "$BITRISE_CLI_SHELL_LIBS/0.0.1/bash/list.sh". (Not sure if having a “latest” symlink would be a great idea though as new bitrise cli releases could cause regressions on existing steps.)

[viktorbenei]

@eliot_pear

I’m curious if you have thoughts on how such a lib could be included in the step without violating the “Do not use submodules, or require any other resource downloaded on-demand” guideline.

We usually use our depman tool for this. It’s really really simple, a revision is long overdue, but it can do this perfectly. Basically you depman init, which generates a dependency definition file. You fill that out by specifying the git clone URLs and the relative path where it should be stored, and any time you want to update you run depman update, which will git clone the latest version from the repos and move the files (without .git) to the specified relative path.

[viktorbenei]

@eliot_pear

I like the solution of incorporating this functionality into the bitrise.yml. “Rename” seems like the best option here as the final value of the original exported variable might be misleading as it will be dependent on the order of which step executed last. I also like that this solution doesn’t require additional overhead on the part of the step, and will work without having to update existing steps.

I’m happy to announce that this is now available in the latest CLI version, v1.6.0!! Finally, I personally waited for this for a really long time now, but priorities…

Anyway, v1.6.0 was just released, it can now be installed locally, and will be deployed on the bitrise.io VMs during the next stack updates this weekend, as usual