Broken Python 2.7 hashlib in new Xcode 10.3.x Mojave stack

Bitrise Build Issue Report template

Description of the issue

Since the stack we use was upgraded a few days ago, our builds fail while running a python program we install in one of our steps. The program tries to import hashlib, but this fails.

You can reproduce this in a shell inside the stack just by running python2.7 and typing import hashlib. It will fail with a lengthy error message about missing hashing algorithms.

As you can see from the github changelog, the last change to this stack upgraded the openssl version. Results on the web seem to show a mismatch in ssl libraries is the cause of the issue.

Environment:

Where did the issue happen?

Xcode 11.3.x, on macOS 10.14.6 (Mojave)

Which build Step causes the issue and which version of the step?

In a script that runs a python program, in our case gcloud from the google cloud sdk.

Reproducibility

Inside a shell running on the stack:

  1. python
  2. import hashlib

Build log

python output on bitrise SSH
prd-std-r-xcode-10-3-191217092949-WfrHZcbukJ748mH6bjpZ7Q:google-cloud-sdk vagrant$ python
Python 2.7.16 (default, Jun 19 2019, 07:40:37)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512

Hi there @philippe_fcg!

Could you please send us a build log, so we could take a closer look? :slight_smile:

Here’s the output of the shell script step that is failing. What we’re doing here:

  1. Download and extract the google cloud sdk, which contains the CLI programs we need to upload file to Google Cloud Storage
  2. Try to log-in using the gcloud utility. (this fails due to the library hashlib being broken)
build log
+------------------------------------------------------------------------------+
| (8) Deploy (redirects) to gcloud                                             |
+------------------------------------------------------------------------------+
| id: script                                                                   |
| version: 1.1.5                                                               |
| collection: https://github.com/bitrise-io/bitrise-steplib.git                |
| toolkit: bash                                                                |
| time: 2019-12-18T15:45:02Z                                                   |
+------------------------------------------------------------------------------+
|                                                                              |
++ git rev-parse --short HEAD
+ ./scripts/upload-country-redirects-to-gcloud.sh
+ curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-180.0.1-darwin-x86_64.tar.gz
+ tar x
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 22 14.3M   22 3231k    0     0  3663k      0  0:00:04 --:--:--  0:00:04 3660k
 44 14.3M   44 6463k    0     0  2835k      0  0:00:05  0:00:02  0:00:03 2836k
 73 14.3M   73 10.5M    0     0  2864k      0  0:00:05  0:00:03  0:00:02 2864k
 80 14.3M   80 11.5M    0     0  3023k      0  0:00:04  0:00:03  0:00:01 3022k
100 14.3M  100 14.3M    0     0  3493k      0  0:00:04  0:00:04 --:--:-- 3494k
+ ./google-cloud-sdk/bin/gcloud auth activate-service-account  --key-file=XXX
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512
ERROR: gcloud failed to load: 'module' object has no attribute 'sha256'
    gcloud_main = _import_gcloud_main()
    import googlecloudsdk.gcloud_main
    from googlecloudsdk.calliope import base
    from googlecloudsdk.calliope import arg_parsers
    from googlecloudsdk.core import log
    from googlecloudsdk.core import properties
    from googlecloudsdk.core import config
    from googlecloudsdk.core.util import files as file_utils
    class Checksum(object):
    def __init__(self, algorithm=hashlib.sha256):

This usually indicates corruption in your gcloud installation or problems with your Python interpreter.

Please verify that the following is the path to a working Python 2.7 executable:
    /usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

If it is not, please set the CLOUDSDK_PYTHON environment variable to point to a working Python 2.7 executable.

If you are still experiencing problems, please reinstall the Cloud SDK using the instructions here:
    https://cloud.google.com/sdk/
|                                                                              |
+---+---------------------------------------------------------------+----------+
| e[31;1mxe[0m | e[31;1mDeploy (redirects) to gcloud (exit code: 1)e[0m                   | 11.41 sec|
+---+---------------------------------------------------------------+----------+
| Issue tracker: https://github.com/bitrise-io/steps-script/issues             |
| Source: https://github.com/bitrise-io/steps-script                           |
+---+---------------------------------------------------------------+----------+

I do think my steps to reproduce the issue in my original post will better help you take a closer look.

It’s the 10.3 stack not the 11.3 right?

I tested the reproduction you gave (thanks for that!!) and indeed python -c 'import hashlib' gives the error you reported.

We plan to brew uninstall python@2 from brew in the coming weeks and we keep an eye on the brew python 2 situation. The root cause of all of these issues is the removal of OpenSSL 1.0 from brew which happened a few weeks ago, combined with the ongoing process in brew to completely remove python v2 from the whole brew library ( https://github.com/Homebrew/homebrew-core/issues/47274 , Formulas still using Python 2 · Issue #47050 · Homebrew/homebrew-core · GitHub , and a few more at Issues · Homebrew/homebrew-core · GitHub ).

The current workaround/solution that we tested extensively and seems to work reliably on all stacks where python@2 from brew is present:

  1. Either use /usr/bin/python instead of just python
  2. Or alternatively you can brew uninstall python@2 via a Script step. After that Script step in the subsequent steps python (without the full path to system default /usr/bin/python ) should resolve to the system default python.

Tested on the Xcode 10.3 stack:

  • python -c 'import hashlib' results in the reported error. :red_circle:
  • /usr/bin/python -c 'import hashlib' works without any issues. :green_apple:
1 Like

Thank you Viktor for all the details! I hadn’t realized so much activity was going on around the python 2 deprecation, but I’m glad to see it. The google cloud tools can be configured to use python3, which is probably the best solution.

1 Like

Indeed, and in the recent gcloud updates I remember seeing messages about that they are migrating the whole gcloud to python3, most components already supporting/using python3 with a few remaining ones (like bq) where the transition is still work-in-progress.

Checking https://cloud.google.com/python/docs/python2-sunset/ and gcloud topic startup it seems these are the only remaining gsutil tools which don’t support python3:

Other Python tools shipped in the Cloud SDK do not yet support Python 3,
    including:

      ▪ dev_appserver
      ▪ endpointscfg

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.