https://github.com/pyca/cryptography/blob/36.0.1/src/cryptography/hazmat/backends/openssl/rsa.py#L210-L211 compares the return value of EVP_PKEY_CTX_set_signature_md() for equality to 0.

The return value section of https://www.openssl.org/docs/man3.0/man3/EVP_PKEY_CTX_set_signature_md.html says this:

> All other functions described on this page return a positive value for success and 0 or a negative value for failure. In particular a return value of -2 indicates the operation is not supported by the public key algorithm.

python-cryptography should check for ret <= 0.


This explains the different result, in combination with a different section from the same manpage:

> Two RSA padding modes behave differently if EVP_PKEY_CTX_set_signature_md() is used. If this function is called for PKCS#1 padding the plaintext buffer is an actual digest value and is encapsulated in a DigestInfo structure according to PKCS#1 when signing and this structure is expected (and stripped off) when verifying. If this control is not used with RSA and PKCS#1 padding then the supplied data is used directly and not encapsulated. In the case of X9.31 padding for RSA the algorithm identifier byte is added or checked and removed if this control is called. If it is not called then the first byte of the plaintext buffer is expected to be the algorithm identifier byte.

The code then behaves as if EVP_PKEY_CTX_set_signature_md() was never called and signs the plaintext (which is the SHA1 hash of the message) with PKCS1v1.5 padding. The openssl-pkeyutl(1) manpage at https://www.openssl.org/docs/man3.0/man1/openssl-pkeyutl.html explains that:

> In PKCS#1 padding if the message digest is not set then the supplied data is signed or verified directly instead of using a DigestInfo structure. If a digest is set then the a DigestInfo structure is used and its the length must correspond to the digest type.

We can also see this in your case:

# echo -n "A message I want to sign" | openssl dgst -sha1
SHA1(stdin)= b4bc01a07dd7c18e7a51702b490e340a90e98156
# echo "958e6fd3db8c824c806f4fe7dd45f91f9e1c08d0992959cb5ef0b65bb57636173c1fd3284f4a5da530820436c94d4b9bc9f4932614242a01ceb1c16fee0b9e1acc7c36ec5bf0300d293f43046454cb3500a3d40c687da6e4cbb00985d14e752b67a8a9b6e51ab9104edb26b5a4b38afc76e096d04b6999d13b92eb5abfbc3c9e984ee83e65f4e1343925ff749eaec043871cfec64c2f88cac7e70417279964116ee7beb776987604a8fe2dd34ea1343b1248c48261e1dff9e4f2bf2d83101f6641eb1b9bd6211f834702e7d32174d918e032cbb589922f97bbdc796623a2d33d20b3f7ecbbb86b7fa26695cda4f73327db467572f90f7011051db03064e8b6c0" | xxd -ps -r > signature
# openssl pkeyutl -verifyrecover -in signature -inkey theirrsa.key -pkeyopt rsa_padding_mode:pkcs1 | xxd
Enter pass phrase for theirrsa.key:
00000000: b4bc 01a0 7dd7 c18e 7a51 702b 490e 340a  ....}...zQp+I.4.
00000010: 90e9 8156                                ...V


The correct signature wraps the hash in a DigestInfo structure:

# echo "205a2e2780f00c4cad1cdb4277ee391163e56aa97c21b4949217c4120c69282a233b7f949f2d084099f851eafc828d6738bca7e2e4836d04a0ec1cf6048723de3f6143fe0ec20d2b9b8d8eee8b58a5d89119b610dc8d99d08c3b8a5e66288640dbc53ac6be47e6a68c2d84c676d70bdc48ece9e7725873b0e277520838154dfecff4f22186deb815080f1200892ca262917b90f7cedff3ec006465a0b629c666850430cd31f72d9eed1936f226bdfbc040ae3095030678db6723f62b0a7eb17e27671c6630ef092e98b66a69c7fc20b26a3abbb6a3db1b6224a766a25818b1b25e7e3196d45d9d9c3f836a82884ea0a5dd1705fdc95a917a41ddd987721c9d5c" | xxd -ps -r > signature-correct
# openssl pkeyutl -verifyrecover -in signature-correct -inkey theirrsa.key -pkeyopt rsa_padding_mode:pkcs1 | xxd
Enter pass phrase for theirrsa.key:
00000000: 3021 3009 0605 2b0e 0302 1a05 0004 14b4  0!0...+.........
00000010: bc01 a07d d7c1 8e7a 5170 2b49 0e34 0a90  ...}...zQp+I.4..
00000020: e981 56                                  ..V


openssl can not refuse to sign payloads where it does not know the used hash algorithm, because from an API point of view this looks exactly the same as signing raw data, which users may still want to do.

I recommend to fix the error check in src/cryptography/hazmat/backends/openssl/rsa.py. Let me know if you agree so that I can move the ticket to python-cryptography.

Good catch! Thanks for you analysis.

It's a python-cryptography bug. I have opened upstream bug https://github.com/pyca/cryptography/issues/6927 .

I'm requestion exception+ so I'm able to attach the BZ to the RHEL 9.0.0 erratum. No additional work is required. The reported issue is fixed by rebase BZ https://bugzilla.redhat.com/show_bug.cgi?id=2059630, which has already exception+ and a successful build in pre-verification.

[root@server ~]# rpm -q python3-cryptography openssl 
python3-cryptography-36.0.1-1.el9_0.x86_64
openssl-3.0.1-15.el9_0.1.x86_64
[root@server ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.0 Beta (Plow)

[root@server ~]# python3
Python 3.9.10 (main, Feb  9 2022, 00:00:00) 
[GCC 11.2.1 20220127 (Red Hat 11.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cryptography.hazmat.backends.openssl import backend
>>> backend._is_fips_enabled()
False
>>> from cryptography.hazmat.primitives.asymmetric import rsa
>>> from cryptography.hazmat.primitives import hashes
>>> from cryptography.hazmat.primitives.asymmetric import padding
>>> private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
>>> private_key.sign(b"message", padding.PKCS1v15(), hashes.SHA1())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 501, in sign
    return _rsa_sig_sign(self._backend, padding, algorithm, self, data)
  File "/usr/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 244, in _rsa_sig_sign
    pkey_ctx = _rsa_sig_setup(
  File "/usr/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 213, in _rsa_sig_setup
    raise UnsupportedAlgorithm(
cryptography.exceptions.UnsupportedAlgorithm: sha1 is not supported by this backend for RSA signing.
>>> 

Based on above observations marking bugzilla verified

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: python-cryptography), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2580