Limit concurrency on specified (deploy) branches, or on project/app level (only allow X concurrency for app Y)

From Viktor Benei [bitrise staff] on 2016/01/14 14:09:23 +0000

There might be a case when you don’t want to allow multiple (usually deploy) builds to run concurrently. Even if we have more than 1 concurrency we might want to require one or more branch to only run one build at a time - for example when deploying a web-app.

Copied from original feature request: http://bitrise.uservoice.com/forums/235233-general/suggestions/11444958-limit-concurrency-on-specified-deploy-branches

From Ara Juljulian on 2016/09/13 13:21:02 +0000

We have an iOS team and an Android team. We have two corresponding apps on Bitrise (in the process of setting them up). Currently, even though we have multiple concurrencies, I find that I am constantly having to wait for the Android integrations to complete before I'm able to run an iOS integration. Ideally, I would allocate 2 or 3 concurrencies to each team and not have them step on each others' toes. Thanks.

From Sam G on 2016/05/05 20:20:09 +0000

Also useful if you have a large project with frequent commits. I assumed this was standard CI behavior (Jenkins does this).

Just a note/workaround: now that we have Organizations, if you have e.g. an Android app&team, and an iOS one, you can create two separate Organizations, one for the Android and one for the iOS app & team, and manage the concurrency count for both separately.

We still plan to provide a more customizable solution, where you could set max concurrencies for projects, so that e.g. the Android project can only use 2 concurrencies from the available 3, as well as the iOS one -> with this config both would get 1 “guaranteed” concurrency, and would share one concurrency/build spot which would be assigned automatically, depending on which project needs more builds at that time.

If you’re interested in this more customizable solution please keep voting on this feature request!

4 Likes

Another use case is some kind of limited external resource.
For example a private device test farm.
One may want to test app #1 on all the devices, then app #2 etc. Even if there are bunch of concurrencies available workflows with tests should be serialized.

On jenkins such configuration may be achieved by disabling Execute concurrent builds if necessary and enabling Restrict where this project can be run with single agent specified.

Maybe a little bit more flexible solution is to add a “concurrency label” attribute to workflows. So that you can set that label on workflows and then set max concurrencies per label. Given label may be assigned to workflows in multiple apps.

1 Like

@koral Definitely a great idea and there are related things on our roadmap :wink:

Please implement this feature, otherwise we are stuck in a infinite queue per project

1 Like

Why? If you have infinite builds limiting the running build count won’t help - can you please share a bit more details/context @neobeppe ?

We have 4 projects, 2 iOS and 2 Android
concurrency x 2 is ok, but without concurrency limit on project we are experiencing something like 8-9 builds in queue on a single project

1 Like

Makes sense, thanks for the details @neobeppe! :slight_smile:

I would also love this feature! But instead of branches, would be awesome if we can limit concurrency for specific apps or workflows.

In my case, we currently have two type of workflow: automated test workflow (which triggers on every new commits), and build workflow (which gets triggered daily, or manually during testing period). What’s happening right now is that the automated test workflow is hogging all the workers and our testers are waiting for the build to complete. In the end, I had to manually abort all the test workflows.

Another idea we have, apart from limiting concurrency – is prioritized workflow, so we don’t have to abort queued jobs.

2 Likes

Specific apps: definitely, that should be covered with something like branch: "*" to match all branches.

Workflows: good point, we’ll think about that, but right now I’m not sure if I have a use case for that (if we’d have branch based filters). Do you have a use case you could share?

That’s a great idea, I love it! Would make it easier to have faster deploys in case PR builds are hogging your queue.

In our use case, we have a Bitrise workflow that gets triggered when someone pushes on our develop or release branch, or creates a pull request. This workflow uses your new derived data cache (awesome work, btw!) to minimize build time (so it won’t block reviews too long, or prevent further breaks on develop branch). There’s also the workflow that’s being used manually by our QA team to archive the app and deploy it to Crashlytics Beta for internal distribution. This workflow should get higher priority. However, both workflows might execute on the same branch.

(EDIT: Added more explanation) If for example, prioritization is not possible, let’s say I have 3 concurrency plan – then I would limit the workflow for pull request to 1 or 2, while the archive workflow can use full resource. This enables archive workflow to keep going even though the pull request workflow is stacking up. Ideally, this has the same goal as prioritization.

1 Like

Definitely makes sense @dvdchr-tvlk, thank you for the use case & detailed infos! :slight_smile:

I have a user case for this also. For deployment of mobile apps.

Deployment workflows can often involve incrementing build version numbers. I don’t want two deployment workflows running on concurrent jobs as they will both attempt to increment the version number and push back to github. This will mean I get two different builds with the same version number.

I would like to be able to restrict jobs so that if a job is already running on a specific branch or workflow it will wait until it is finished before initiating the next one.

2 Likes

Makes sense @DanielCreagh and it’s definitely planned, but we don’t have an ETA yet.

Description of the feature request

Currently we have a plan for 3 simultaneous builds under 1 organisation. Is there a way to limit 1 build for project so that 1 project has only one simulationus build. We need this case as some projects would ask for 3 builds when they are triggered from 3 different branches. And at the same time another project would ask for a build but it will have to queue as all the build slots are busy. Currently its not possible to optimize the build trigger for the projects.

Use case / for what or how I would use it

In the case I have 3 free slots when one project triggers the build 3 times it will keep 2 of its builds in the queue and if another project would ask for a build it can take 1 slot without having to abort the previous project builds and 1 slot will still be free.

Thanks

Hi @DeveloperExozet,

Thank you for the feature request! I’m merging this thread to an already existing (and planned) one, which is about the same feature:

2 Likes

Hey there, sorry for the bump.

I have a scenario in which one workflow (primary) starts a number of concurrent workflows at some point (through the Bitrise Start Build step).

If I have 3 concurrencies available, if 3 primary workflows are executed, there will be a deadlock situation: the concurrent workflows they start are all put on hold, and the primary workflows waiting for them fail due to timeout.

I haven’t found a setting to limit the number of workflows running at the same time, so I’m bumping this thread to find out if I’m missing something. Thanks in advance!

2 Likes

Hi @dicksson.oliveira!

Unfortunately there is no way around this at the moment, but the bump is very much welcomed! :slight_smile:

1 Like