Arrays in step outputs


#1

Is there any convention or recommended way how to deal with cases when output need to be an array?
For example in official gradle-runner step there is an BITRISE_APK_PATH_LIST output which seems to be just file paths separated by pipe char |.

So when one of the paths contain | character (eg. app flavor name contains it) it becomes non-trivial to distinguish pipes in filenames from separators.
Isn’t it better to use for example a YAML/JSON array there?


#2

Hey @koral! :wink:

Definitely would be, but you can’t do that with environment variables unfortunately. Env Vars are simple strings, no other data type is supported as an environment variable as far as I know.

Of course you can serialize data into string, e.g. serialize these infos as JSON and store that in an env var, then read and deserialize it in another step. If you have your own steps this is definitely possible, but that makes handling of these variables quite a bit more complex, e.g. handling JSON in Bash is not trivial at all unfortunately.

Why we chose the pipe char is because pipe should not be in the path. Pipes in shells are used for connecting commands, and AFAIK it’s considered as a really bad practice to include it in a file’s name (due to the special meaning the char has in most shells).

I think the best solution might be to add a second output which would include the data in a JSON serialized form. That way simple steps which are only interested in the path list can be written in Bash, while the JSON output could include more info (e.g. mapping of paths) but would require a bit more complex logic to parse that output (json decoding).

WDYT @koral & @godreikrisztian?


#3

Yeah, that’s true. They are plain strings at low level: http://pubs.opengroup.org/onlinepubs/9699919799/functions/getenv.html

Maybe JSON as whole isn’t bash-friendly, but arrays only seem to be simple, for example with jq:

$ JSON_ARRAY='["foo|\"", ".", "|", "bar"]'
$ BASH_ARRAY=($(echo $JSON_ARRAY | jq '.[]' -r))
$ echo ${BASH_ARRAY[0]}
foo|"
$ echo ${BASH_ARRAY[1]}
.
$ echo ${BASH_ARRAY[2]}
|
$ echo ${#BASH_ARRAY[*]}
4

OK, understand.

Sounds good.

Just curious, are there any other official steps which use arrays as input and/or output?


#4

Sure!

Output: https://github.com/bitrise-io/steps-gradle-runner/blob/master/step.yml#L147

Input: https://github.com/bitrise-steplib/steps-google-play-deploy/blob/master/step.yml#L70


#5

It seems that second one will fail if there is a pipe in APK filename: https://github.com/bitrise-steplib/steps-google-play-deploy/blob/master/main.go#L163-L165


#6

Yes it will. How did you generate an APK file name like that?


#7

I explained it here:

Here is the source code: https://github.com/DroidsOnRoids/android-bitrise-test/blob/master/lib/build.gradle#L23
and bitrise build having this “issue” https://www.bitrise.io/build/a7f6486fff96fd23


#8

Ahh, so it’s the name of your flavor which includes the pipe char, not a “default” android thing.

We’ll try to think about a better solution (I like the idea of a “rich” output encoded into JSON more and more). In the meantime I’d suggest you to use a different separator for your file name, if possible.


#9

[quote=“viktorbenei, post:8, topic:2200”]
In the meantime I’d suggest you to use a different separator for your file name, if possible.
[/quote]:slight_smile:

This is not a real project but only playground to test our bitrise steps.
I wanted to consume BITRISE_APK_PATH_LIST in another step and realized that pipes may cause problems so I created this topic.


#10

Ahh, got it :wink:

Sure, edge cases have to be covered and that’s part of why we have this forum & this sub category :wink:

The thing here is that there’s pretty much no char which is not allowed in a filename on a modern Unix system (AFAIK the only one is /), so we had to choose one which is unlikely to be in a path or any other value. Of course it’s impossible to select one which works for every case, pipe seems to be a “sensible default” as a separator (but it’s not hardcoded, just a suggestion to use |, every step is free to choose any separator if required).