iOS build times increasing after 16-12-2016 due to python3 not resolving openssl dependency

[sdduursma]

Stack: Xcode 7.3.1, installed on OS X 10.11 (El Capitan)

Between 16 Dec 2016 and 19 Dec 2016 our Bitrise build times increased quite drastically. After some investigating I made the following findings:

Our builds depend on openssl and python3 among other things. In the workflow we’re installing openssl and python like this:

brew install openssl
brew install python3

After 16 Dec, brew install openssl was outputting the following:

Warning: openssl is a keg-only and another version is linked to opt.
Use `brew install --force` if you want to install this version

I ran brew info openssl before brew install openssl which outputted the following:

openssl: stable 1.0.2k (bottled) [keg-only]
SSL/TLS cryptography library
https://openssl.org/
/usr/local/Cellar/openssl/1.0.2h (1,691 files, 17.2MB)
  Built from source on 2016-05-04 at 06:43:30 with: --universal
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/openssl.rb
==> Dependencies
Build: makedepend
==> Options
--universal
	Build a universal binary
--without-test
	Skip build-time tests (not recommended)
==> Caveats
A CA file has been bootstrapped using certificates from the SystemRoots
keychain. To add additional certificates (e.g. the certificates added in
the System keychain), place .pem files in
  /usr/local/etc/openssl/certs

and run
  /usr/local/opt/openssl/bin/c_rehash

This formula is keg-only, which means it was not symlinked into /usr/local.

Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries

To me this looked like openssl was now pre-installed on the Bitrise VM, and installing openssl using brew install openssl was failing because of this. Note that the pre-installed version in the Cellar is 1.0.2h.

openssl is a dependency of python3. After 16 Dec, the duration of the installation of python3 increased from about 1 min to 13 min. Looking at the output from brew install python3, brew was not anymore finding openssl as a python3 dependency:
before 16 Dec:
==> Installing dependencies for python3: readline, sqlite, gdbm
after 16 Dec:
==> Installing dependencies for python3: readline, sqlite, gdbm, openssl
Thus, brew installed openssl again to resolve the python3 dependency, and this installation was taking about 13 min.

Currently we’ve resolved this issue by installing openssl like this before installing python3:
brew install --force openssl
which as far as I see should override the pre-installed version of openssl, with a version of openssl that does satisfy python3’s dependency.

I have the following questions:

1. Is there anything that changed in the Bitrise ecosystem that could have caused installing openssl with brew install openssl to fail? Like a version of openssl being pre-installed on the Bitrise VM?
2. Why does the pre-installed version of openssl not satisfy the python3 dependency? Could it have anything to do with the version in the Cellar being 1.0.2h (as opposed to 1.0.2k, which is the version after installing with brew) ?

[viktorbenei]

@sdduursma

Is there anything that changed in the Bitrise ecosystem that could have caused installing openssl with brew install openssl to fail? Like a version of openssl being pre-installed on the Bitrise VM?

Xcode 7.3.1 stack is never updated. That’s an LTS stack since Xcode 7.3.1 came out, there was absolutely no update on it since - for more information about stack and update types see:

How are the bitrise.io stacks updated and with what frequency? What's the stack deprecation / removal policy?

The stacks on bitrise.io are updated every weekend, but what’s updated depends on the stack. There are stacks where only certain caches are updated (no preinstalled tools etc.), and there are stacks which are completely re-built every week. You can find the update type of the stacks below. Xcode stacks Xcode stacks, except the “Edge” one, are built once, and are not updated at all, except with Bitrise CLI related updates, dependency manager cache updates and with critical fixes. These are Stabl…

@sdduursma

Why does the pre-installed version of openssl not satisfy the python3 dependency? Could it have anything to do with the version in the Cellar being 1.0.2h (as opposed to 1.0.2k, which is the version after installing with brew) ?

You should check brew about this, I’d say most likely the python3 brew formula did change, and now depends on a different version of openssl. It definitely is not related to any update we did on the stack, as other than the Bitrise CLI updates there was absolutely no other update on the stack since Xcode 7.3.1 / May 2016 (http://blog.bitrise.io/2016/05/05/xcode-7-3-1-stack-update.html).