Description of problem: This looks to be related to the switch to OpenSSL 3 for Python 2.7 MD4 is no longer a supported digest algorithm by OpenSSL but calling hashlib.new('md4') does not raise ValueError, which is what is expected for instance by python-passlib's test suite. Since no exception is raised, programs continue and then get incorrect digest values. Version-Release number of selected component (if applicable): python2.7-2.7.18-24.fc37 Here is the behaviour with python2.7-2.7.18-24.fc37: Python 2.7.18 (default, Jul 22 2022, 00:00:00) [GCC 12.1.1 20220628 (Red Hat 12.1.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import hashlib >>> m = hashlib.new('md4') >>> m.digest() '\x10i\xd2\xd1\xde\x7f\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00' Here is the behaviour with python2.7-2.7.18-22.fc36.x86_64: Python 2.7.18 (default, Jun 9 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import hashlib >>> m = hashlib.new('md4') >>> m.digest() '1\xd6\xcf\xe0\xd1j\xe91\xb7<Y\xd7\xe0\xc0\x89\xc0' Here is the behaviour with python36-3.6.8-38.module_el8.5.0+895+a459eca8.x86_64: Python 3.6.8 (default, Mar 25 2022, 11:15:52) [GCC 8.5.0 20210514 (Red Hat 8.5.0-10)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import hashlib >>> m = hashlib.new('md4') >>> m.digest() b'1\xd6\xcf\xe0\xd1j\xe91\xb7<Y\xd7\xe0\xc0\x89\xc0' Behaviour with python3-3.11.0~b4-2.fc37.x86_64: Python 3.11.0b4 (main, Jul 22 2022, 00:00:00) [GCC 12.1.1 20220628 (Red Hat 12.1.1-3)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import hashlib >>> m = hashlib.new('md4') Traceback (most recent call last): File "/usr/lib64/python3.11/hashlib.py", line 160, in __hash_new return _hashlib.new(name, data, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: [digital envelope routines] unsupported During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python3.11/hashlib.py", line 166, in __hash_new return __get_builtin_constructor(name)(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/hashlib.py", line 123, in __get_builtin_constructor raise ValueError('unsupported hash type ' + name) ValueError: unsupported hash type md4 So it appears that the current rawhide build not only fails to raise ValueError, it also computes incorrect digest values.
Interestingly, pypys: Python 2.7.18 (f1d28f87e6d8, Jul 22 2022, 16:05:41) [PyPy 7.3.9 with GCC 12.1.1 20220628 (Red Hat 12.1.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>> import hashlib >>>> m = hashlib.new('md4') >>>> m.digest() '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' Python 3.9.12 (dd85a1aa7d80, Jul 22 2022, 16:05:26) [PyPy 7.3.9 with GCC 12.1.1 20220628 (Red Hat 12.1.1-3)] on linux Type "help", "copyright", "credits" or "license" for more information. >>>> import hashlib >>>> m = hashlib.new('md4') >>>> m.digest() b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Python 3.10 fails as expected on Fedora 36 with "unsupported hash type md4": $ python3 Python 3.10.5 (main, Jun 9 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)] on linux >>> import hashlib >>> m=hashlib.new('md4') Traceback (most recent call last): File "/usr/lib64/python3.10/hashlib.py", line 160, in __hash_new return _hashlib.new(name, data, **kwargs) ValueError: [digital envelope routines] unsupported During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python3.10/hashlib.py", line 166, in __hash_new return __get_builtin_constructor(name)(data) File "/usr/lib64/python3.10/hashlib.py", line 123, in __get_builtin_constructor raise ValueError('unsupported hash type ' + name) ValueError: unsupported hash type md4 Versions: python3-3.10.5-2.fc36.x86_64 with openssl-3.0.5-1.fc36.x86_64.
(In reply to Miro Hrončok from comment #1) > Interestingly, pypys: > > Python 2.7.18 (f1d28f87e6d8, Jul 22 2022, 16:05:41) > [PyPy 7.3.9 with GCC 12.1.1 20220628 (Red Hat 12.1.1-3)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>>> import hashlib > >>>> m = hashlib.new('md4') > >>>> m.digest() > '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' > > > Python 3.9.12 (dd85a1aa7d80, Jul 22 2022, 16:05:26) > [PyPy 7.3.9 with GCC 12.1.1 20220628 (Red Hat 12.1.1-3)] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>>> import hashlib > >>>> m = hashlib.new('md4') > >>>> m.digest() > b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' I've asked about this at https://mail.python.org/archives/list/pypy-dev@python.org/thread/6ZP546NX4BTLGIZNHZHZD3Y6ERYUOQHW/
I'm playing with this on Fedora 36 with the latest OpenSSL 3 from the stable repo and Python 2 built from rawhide sources. The difference between md2 - which behaves correctly and raises a ValueError - and md4 at this line: https://github.com/fedora-python/cpython/blob/fedora-2.7/Modules/_hashopenssl.c#L560 The EVP_get_digestbyname function returns NULL for md2 but not for md4. This is handled a little bit later in: https://github.com/fedora-python/cpython/blob/821450bb6b5231430cecca2247a9872de75f0f47/Modules/_hashopenssl.c#L502-L505 where digest is NULL for md2 and initial_ctx is also NULL, see the third argument at https://github.com/fedora-python/cpython/blob/821450bb6b5231430cecca2247a9872de75f0f47/Modules/_hashopenssl.c#L562-L563 So, this seems to be a problem in OpenSSL. Also, when you compare how it behaves in CLI, there is a difference: [root@8368f2ebbc21 /]# echo "booo" | openssl dgst -md2 dgst: Unknown option or message digest: md2 dgst: Use -help for summary. 40ECDD03087F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (md2 : 0), Properties (<null>) [root@8368f2ebbc21 /]# echo "booo" | openssl dgst -md4 Error setting digest 40DCB131C77F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD4 : 85), Properties () 40DCB131C77F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:252: I was able to go down to this function: https://github.com/openssl/openssl/blob/fc5888ccb60f33b366972299f30b976c4dc12162/crypto/objects/o_names.c#L152-L187 which return NULL for md2 and evp_md for md4. That's my maximum for today.
Changing to the OpenSSL component so we could get the opinion of the team.
If you use OpenSSL 3.0, you should use EVP_MD_fetch and check the result. Using old interfaces will cause problems.
Interesting, thanks for the quick response. I see that there is also a long read with the migration guide: https://www.openssl.org/docs/manmaster/man7/migration_guide.html Christian, we might be able to fix this problem, but you have much better knowledge. Could you please look at whether there are more deprecated functions in use?
I see two big changes to the upstream code leading to the fix in the python3 branches: https://github.com/python/cpython/commit/995b5d38e7cc24cac3de8dfd516115f86b0bcf80 and then for utilizing EVP_MD_fetch: https://github.com/python/cpython/commit/443b308fee088e21bbf472c376c5c9e3648f916c Maybe it could be possible to just change the call from EVP_get_digestbyname to EVP_MD_fetch.
I've prepared a minimal fix for this issue: https://src.fedoraproject.org/rpms/python2.7/pull-request/37 Verification: python2.7-2.7.18-24.fc37.x86_64 # python2 Python 2.7.18 (default, Jul 22 2022, 00:00:00) [GCC 12.1.1 20220628 (Red Hat 12.1.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import hashlib >>> hashlib.new("md4") <md4 HASH object @ 0x7f80cef14c90> >>> python2.7-2.7.18-25.fc37.x86_64 # python2 Python 2.7.18 (default, Aug 8 2022, 00:00:00) [GCC 12.1.1 20220628 (Red Hat 12.1.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import hashlib >>> hashlib.new("md4") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.7/hashlib.py", line 130, in __hash_new return __get_builtin_constructor(name)(string) File "/usr/lib64/python2.7/hashlib.py", line 97, in __get_builtin_constructor raise ValueError('unsupported hash type ' + name) ValueError: unsupported hash type md4
This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle. Changing version to 37.
I tried the scratch build from the pull request and my python-passlib package ran through its test suite successfully (this had been how I discovered the issue in the first place).
FEDORA-2022-aeff25a305 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-aeff25a305
FEDORA-2022-aeff25a305 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.
Reopening for rawhide.
FEDORA-2022-e84a23f52b has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2022-e84a23f52b
FEDORA-2022-e84a23f52b has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.