+++ This bug was initially created as a clone of Bug #2063838 +++

The signature verification part of the krb5 PKINIT plugin was initially implemented using PKCS7_verify[1], and error handling was written according to this function[2].

This code was modified[3] to use CMS_verify[4] when building against >= OpenSSL 1.0, but kept PKCS7_verify error codes handling.

As a consequence, the KDC_ERR_DIGEST_IN_SIGNED_DATA_NOT_ACCEPTED error, which should explain the digest algorithm is not support is no raised and an unrelated error is displayed:

"Failed to verify CMS message: content type not enveloped data"

Actual CMS_verify error codes should be handled.

[1] https://www.openssl.org/docs/man3.0/man3/PKCS7_verify.html
[2] https://github.com/krb5/krb5/blob/0ef0646069c1d1376aa632a4791ea7e429f5ae9b/src/plugins/preauth/pkinit/pkinit_crypto_openssl.c#L1290-L1297
[3] https://github.com/krb5/krb5/commit/1b57e7650da41dbef6a50c6e925583d6c8ed9e52#diff-4aab62556d7562101df00d240f5444ed99a48a8e00d06c57a9bf5cb29b8a1ea8R1395
[4] https://www.openssl.org/docs/man3.0/man3/CMS_verify.html

--- Additional comment from Julien Rische on 2022-07-29 15:51:59 UTC ---

There are 2 points to fix:

  * Error codes have to be modified to match the new function.
  * The last error from OpenSSL stack should be picked rather that the first one which is always CMS_R_CONTENT_TYPE_NOT_ENVELOPED_DATA[1], therefore it does not allow to differentiate an error caused by an unaccepted digest algorithm and an invalid signature.

The error code we could expect based on the code of OpenSSL CMS signature verification are:

  * CMS_R_NO_MATCHING_DIGEST[2]
  * CMS_R_NO_MATCHING_SIGNATURE (not used in practice)
  * CMS_R_UNKNOWN_DIGEST_ALGORITHM[3]

But I've run some tests and the error codes that seems to be actually raised are:

  * EVP_R_INVALID_DIGEST: the OID is recognized the algorithm is not allowed by the policy in signature verification
  * RSA_R_DIGEST_NOT_ALLOWED: OpenSSL fails to infer a signature algorithm based on the digest one and the key type (the more generic code PROV_R_DIGEST_NOT_ALLOWED might be considered too)
  * CMS_R_VERIFICATION_FAILURE: signature verification failed

We should use ERR_peek_last_error()[4] instead of ERR_peek_error() in order to get these error codes.


[1] https://github.com/openssl/openssl/blob/openssl-3.0.1/crypto/cms/cms_err.c#L45-L46
[2] https://github.com/openssl/openssl/blob/openssl-3.0.1/crypto/cms/cms_lib.c#L458
[3] https://github.com/openssl/openssl/blob/openssl-3.0.1/crypto/cms/cms_sd.c#L869
[4] https://www.openssl.org/docs/man3.0/man3/ERR_peek_last_error.html

--- Additional comment from Julien Rische on 2022-08-02 07:40:20 UTC ---

Pull request:
https://github.com/krb5/krb5/pull/1264

--- Additional comment from Julien Rische on 2022-08-19 08:47:10 UTC ---

EVP_R_INVALID_DIGEST is actually raised by code in a downstream OpenSSL patch[1]. I will handle it in a krb5 downstream patch too.

[1] https://src.fedoraproject.org/rpms/openssl/blob/rawhide/f/0049-Allow-disabling-of-SHA1-signatures.patch#_217