Bug 2064343

Summary: hmac and hashlib raising issues on FIPS env: _hashlib.UnsupportedDigestmodError: Unsupported digestmod functools.partial(<function __hash_new at 0x7f6c30c10700>, 'sha256')
Product: Red Hat Enterprise Linux 9 Reporter: Andre <afariasa>
Component: python3.9Assignee: Python Maintainers <python-maint>
Status: CLOSED NOTABUG QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: CentOS StreamCC: ben.blattberg, bstinson, cheimes, dbelyavs, jwboyer, pviktori, torsava
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-04 12:22:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andre 2022-03-15 15:49:28 UTC
Description of problem:

~~~
digest = functools.partial(hashlib.new, 'sha256')
hmac.new("key".encode("utf8"), b'message', digest)
~~~

This code works fine on Centos9 without FIPS enabled, but when enabling FIPS we see the following error:

~~~
Traceback (most recent call last):
  File "/home/cloud-user/test_hmac.py", line 8, in <module>
    hmac.new("key".encode("utf8"), b'message', digest)
  File "/usr/lib64/python3.9/hmac.py", line 189, in new
    return HMAC(key, msg, digestmod)
  File "/usr/lib64/python3.9/hmac.py", line 60, in __init__
    self._init_hmac(key, msg, digestmod)
  File "/usr/lib64/python3.9/hmac.py", line 69, in _init_hmac
    self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
_hashlib.UnsupportedDigestmodError: Unsupported digestmod functools.partial(<function __hash_new at 0x7f6c30c10700>, 'sha256')
~~~

Version-Release number of selected component (if applicable):
Python 3.9.10

How reproducible:
Always

Steps to Reproduce:
1. Run the script on Centos9 with fips enabled

Actual results:

~~~
Traceback (most recent call last):
  File "/home/cloud-user/test_hmac.py", line 8, in <module>
    hmac.new("key".encode("utf8"), b'message', digest)
  File "/usr/lib64/python3.9/hmac.py", line 189, in new
    return HMAC(key, msg, digestmod)
  File "/usr/lib64/python3.9/hmac.py", line 60, in __init__
    self._init_hmac(key, msg, digestmod)
  File "/usr/lib64/python3.9/hmac.py", line 69, in _init_hmac
    self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
_hashlib.UnsupportedDigestmodError: Unsupported digestmod functools.partial(<function __hash_new at 0x7f6c30c10700>, 'sha256')
~~~

Expected results:

Same as without FIPS enabled

Additional info:
Script used to test: http://pastebin.test.redhat.com/1037225

Comment 1 Christian Heimes 2022-03-15 15:58:37 UTC
Does

   hmac.new(b'key', b'message', 'sha256')

work for you?

Comment 2 Christian Heimes 2022-03-15 16:05:06 UTC
You should either pass the name of a hash function or the hash constructor directly to hmac.new(). In both cases Python will use the OpenSSL EVP interface for MAC.

>>> import hmac, hashlib, functools

>>> hmac.new(b'key', b'message', 'sha256')
<hmac.HMAC object at 0x7fe9dbf24d60>
>>> hmac.new(b'key', b'message', hashlib.sha256)
<hmac.HMAC object at 0x7fe9dbe93950>

If you pass something else to HMAC, then it triggers a different code path that falls back to a pure Python implementation. This pure Python implementation is not FIPS compliant and therefore fails in FIPS mode:

>>> hmac.new(b'key', b'message', functools.partial(hashlib.new, 'sha256'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.9/hmac.py", line 189, in new
    return HMAC(key, msg, digestmod)
  File "/usr/lib64/python3.9/hmac.py", line 60, in __init__
    self._init_hmac(key, msg, digestmod)
  File "/usr/lib64/python3.9/hmac.py", line 69, in _init_hmac
    self._hmac = _hashopenssl.hmac_new(key, msg, digestmod=digestmod)
_hashlib.UnsupportedDigestmodError: Unsupported digestmod functools.partial(<function __hash_new at 0x7fe9dbea5c10>, 'sha256')

Comment 3 Dmitry Belyavskiy 2022-03-15 16:10:02 UTC
Could you please show the corresponding C code you are invoking?

Comment 4 Christian Heimes 2022-03-15 16:23:03 UTC
It's not an OpenSSL problem. It is a problem how the code attempts to construct an HMAC object with Python. The preferred and correct way is to either pass the name of a hash function or the hash object. The additional indirection with functools.partial() triggers a code path that is blocked in FIPS mode.

https://gitlab.com/redhat/centos-stream/rpms/python3.9/-/blob/c076ac88b9366e289bda88e3a9797c9384856963/00329-fips.patch#L264

Comment 5 Christian Heimes 2022-03-15 16:31:00 UTC
Original code is https://github.com/openstack/swift/blob/master/swift/common/middleware/tempurl.py#L753

I'm moving the ticket to Python team for visibility.

Comment 6 Petr Viktorin (pviktori) 2022-03-16 09:46:36 UTC
The issue is that in FIPS mode we need to ensure that only OpenSSL is used for the hashes. OpenSSL needs a string algorithm name, so if you pass in another object we need to "unwrap" it and get the algorithm name for it. (If that's not possible, upstream Pyhon will fall back to a slower Python implementation which calls that object, but that code path is disabled in FIPS mode.)
The "unwrapping" is only done for hash constructors. I guess we can allow partial(hashlib.new, ...) as well, but we're getting into whack-a-mole/diminishing-returns.

Could you use the `hash_algorithm` name directly, instead of the `digest` partial?


> Could you please show the corresponding C code you are invoking?

Start with the Python code which is roughly this: https://github.com/encukou/cpython/blob/fips_rhel9/Lib/hmac.py#L58
And the C: 
- _hmac_new: https://github.com/encukou/cpython/blob/fips_rhel9/Modules/_hashopenssl.c#L1493
- py_digest_by_digestmod: https://github.com/encukou/cpython/blob/fips_rhel9/Modules/_hashopenssl.c#L322

Comment 7 Petr Viktorin (pviktori) 2022-03-23 13:31:02 UTC
Could you use the `hash_algorithm` name directly, instead of the `digest` partial?

Comment 8 Tomas Orsava 2022-05-04 12:22:35 UTC
This has been without a response for a month and a half, please reopen if you want to discuss the issue further.

Comment 9 Red Hat Bugzilla 2023-09-15 01:52:45 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days

Comment 10 ben.blattberg 2025-01-06 19:01:28 UTC
I'm running into a similar problem and just wanted to note it here for others/myself when I forget:

I'm running Red Hat-patched Python; with pallets/Flask and pallets/itsdangerous, both updated to use a lazy loader rather than `hashlib.sha1` directly:

- https://github.com/pallets/flask/commit/db461112c70d5f2bf93c7a6ac27eeb665c232dd0
- https://github.com/pallets/itsdangerous/commit/7f4dcf83a07bb3d53f4e0e65ef1b43327b4cca90

With those two alterations (Python erroring, pallets using `_lazy_sha1`), I'm noticing failures in a FIPS environment. My current thought is to avoid the lazy loader and patch the objects to call the hash algorithm as their default, which I suppose is the whole point of having a lazy loading func in the first place.

But if there's another way to think about this issue, I'd be happy to hear it.