+++ This bug was initially created as a clone of Bug #2060798 +++ Description of problem: we hit this while trying to install OSP on centos9/rhel9. After having installed freeipa we use https://docs.ansible.com/ansible/latest/collections/community/general/ipa_role_module.html to create a simple role: - name: add nova host manager role ipa_role: name: Nova Host Manager description: Nova Host Manager ipa_user: "{{ ipa_principal }}" ipa_pass: "{{ ipa_password }}" privilege: - Nova Host Management This fails with the following error in /var/log/httpd/error_log: [Fri Mar 04 07:58:03.548673 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] mod_wsgi (pid=13804): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'. [Fri Mar 04 07:58:03.549291 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] Traceback (most recent call last): [Fri Mar 04 07:58:03.549333 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] File "/usr/lib/python3.9/site-packages/ipaserver/wsgi.py", line 71, in application [Fri Mar 04 07:58:03.549338 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] return api.Backend.wsgi_dispatch(environ, start_response) [Fri Mar 04 07:58:03.549343 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 301, in __call__ [Fri Mar 04 07:58:03.549345 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] return self.route(environ, start_response) [Fri Mar 04 07:58:03.549349 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 313, in route [Fri Mar 04 07:58:03.549351 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] return app(environ, start_response) [Fri Mar 04 07:58:03.549356 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 1066, in __call__ [Fri Mar 04 07:58:03.549369 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] result = attempt_kinit(user_principal, password, [Fri Mar 04 07:58:03.549372 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 996, in attempt_kinit [Fri Mar 04 07:58:03.549374 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] self.kinit(user_principal, password, [Fri Mar 04 07:58:03.549377 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 1094, in kinit [Fri Mar 04 07:58:03.549379 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] kinit_armor( [Fri Mar 04 07:58:03.549382 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] File "/usr/lib/python3.9/site-packages/ipalib/install/kinit.py", line 129, in kinit_armor [Fri Mar 04 07:58:03.549384 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] run(args, env=env, raiseonerr=True, capture_error=True) [Fri Mar 04 07:58:03.549387 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] File "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 598, in run [Fri Mar 04 07:58:03.549389 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] raise CalledProcessError( [Fri Mar 04 07:58:03.549420 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] ipapython.ipautil.CalledProcessError: CalledProcessError(Command ['/usr/bin/kinit', '-n', '-c', '/run/ipa/ccaches/armor_13804', '-X', 'X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt', '-X', 'X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'] returned non-zero exit status 1: 'kinit: Cannot read password while getting initial credentials\\n') trying to run the kinit command manually: [root@freeipa-0 ~]# export KRB5_TRACE=/dev/stdout [root@freeipa-0 ~]# kinit -n -c /tmp/armor_13804 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem [2406] 1646386032.615628: Getting initial credentials for WELLKNOWN/ANONYMOUS [2406] 1646386032.615630: Sending unauthenticated request [2406] 1646386032.615631: Sending request (178 bytes) to BGP.FTW [2406] 1646386032.615632: Initiating TCP connection to stream 172.30.4.2:88 [2406] 1646386032.615633: Sending TCP request to stream 172.30.4.2:88 [2406] 1646386032.615634: Received answer (526 bytes) from stream 172.30.4.2:88 [2406] 1646386032.615635: Terminating TCP connection to stream 172.30.4.2:88 [2406] 1646386032.615636: Response was from primary KDC [2406] 1646386032.615637: Received error from KDC: -1765328359/Additional pre-authentication required [2406] 1646386032.615640: Preauthenticating using KDC method data [2406] 1646386032.615641: Processing preauth types: PA-PK-AS-REQ (16), PA-FX-FAST (136), PA-ETYPE-INFO2 (19), PA-PKINIT-KX (147), PA-SPAKE (151), PA-ENC-TIMESTAMP (2), PA_AS_FRESHNESS (150), PA-FX-COOKIE (133) [2406] 1646386032.615642: Selected etype info: etype aes256-cts, salt "BGP.FTWWELLKNOWNANONYMOUS", params "" [2406] 1646386032.615643: Received cookie: MIT1\x00\x00\x00\x01\xea\xb8\x0b\xb4L}\x18\xc3\xbaTc8?\xe4\xb7p\xb2\x8e\xbf\x85\xc0\xa9.\xfc\\xbe\xaa\xeb\xaf&\x0f\xcbv_\T\x06!\xdch\xd4`\xb6\xa8\\xb4E13\xdb\x10\x82A>\x 05\x05\xae\xe5\xder\xd0\xc7A2M\xdf\\x10I\xe1\x99$\x1ct.\xb3Lu\x0fcRd:\x8f^\x1e\x0b\xdf:\xe3\x81p\xd56C.\xdd&t\xd1\xf6PE$Y\xc3G$\xa0\x06\xa2\x9cxU\x10\xe2\x1fI\xf5\x7f\xae\xa28\xe7\xb8cS\x09\x9b\xc3 [2406] 1646386032.615644: Preauth module pkinit (147) (info) returned: 0/Success [2406] 1646386032.615645: PKINIT client received freshness token from KDC [2406] 1646386032.615646: Preauth module pkinit (150) (info) returned: 0/Success [2406] 1646386032.615647: PKINIT loading CA certs and CRLs from FILE [2406] 1646386032.615648: PKINIT loading CA certs and CRLs from FILE [2406] 1646386032.615649: PKINIT loading CA certs and CRLs from FILE [2406] 1646386032.615650: PKINIT client computed kdc-req-body checksum 9/3EA2950DCF38A9868B8B8C5E040F69865752C33A [2406] 1646386032.615652: PKINIT client making DH request [2406] 1646386032.615653: Preauth module pkinit (16) (real) returned: 0/Success [2406] 1646386032.615654: Produced preauth for next request: PA-FX-COOKIE (133), PA-PK-AS-REQ (16) [2406] 1646386032.615655: Sending request (1648 bytes) to BGP.FTW [2406] 1646386032.615656: Initiating TCP connection to stream 172.30.4.2:88 [2406] 1646386032.615657: Sending TCP request to stream 172.30.4.2:88 [2406] 1646386032.615658: Received answer (2558 bytes) from stream 172.30.4.2:88 [2406] 1646386032.615659: Terminating TCP connection to stream 172.30.4.2:88 [2406] 1646386032.615660: Response was from primary KDC [2406] 1646386032.615661: Processing preauth types: PA-PK-AS-REP (17), PA-PKINIT-KX (147) [2406] 1646386032.615662: Preauth module pkinit (147) (info) returned: 0/Success [2406] 1646386032.615663: PKINIT OpenSSL error: Failed to verify CMS message [2406] 1646386032.615664: PKINIT OpenSSL error: error:1700006B:CMS routines::content type not enveloped data [2406] 1646386032.615665: PKINIT OpenSSL error: error:03000098:digital envelope routines::invalid digest [2406] 1646386032.615666: PKINIT client could not verify DH reply [2406] 1646386032.615667: Preauth module pkinit (17) (real) returned: -1765328320/Failed to verify CMS message: content type not enveloped data [2406] 1646386032.615668: Produced preauth for next request: (empty) [2406] 1646386032.615669: Getting AS key, salt "BGP.FTWWELLKNOWNANONYMOUS", params "" Password for WELLKNOWN/ANONYMOUS: kinit: Password read interrupted while getting initial credentials AFAICS there were recent changes in openssl that seem related: * Fri Feb 25 2022 Clemens Lang <cllang> - 1:3.0.1-14 - Prevent use of SHA1 with ECDSA - Resolves: rhbz#2031742 * Fri Feb 25 2022 Dmitry Belyavskiy <dbelyavs> - 1:3.0.1-13 - OpenSSL will generate keys with prime192v1 curve if it is provided using explicit parameters - Resolves: rhbz#1977867 * Thu Feb 24 2022 Peter Robinson <pbrobinson> - 1:3.0.1-12 - Support KBKDF (NIST SP800-108) with an R value of 8bits - Resolves: rhbz#2027261 * Wed Feb 23 2022 Clemens Lang <cllang> - 1:3.0.1-11 - Allow SHA1 usage in MGF1 for RSASSA-PSS signatures - Resolves: rhbz#2031742 * Wed Feb 23 2022 Dmitry Belyavskiy <dbelyavs> - 1:3.0.1-10 - rebuilt * Tue Feb 22 2022 Clemens Lang <cllang> - 1:3.0.1-9 - Allow SHA1 usage in HMAC in TLS - Resolves: rhbz#2031742 * Tue Feb 22 2022 Dmitry Belyavskiy <dbelyavs> - 1:3.0.1-8 - OpenSSL will generate keys with prime192v1 curve if it is provided using explicit parameters - Resolves: rhbz#1977867 - pkcs12 export broken in FIPS mode - Resolves: rhbz#2049265 * Tue Feb 22 2022 Clemens Lang <cllang> - 1:3.0.1-8 - Disable SHA1 signature creation and verification by default - Set rh-allow-sha1-signatures = yes to re-enable - Resolves: rhbz#2031742 If I downgrade openssl to 3.0.1-5 things work as expected. (I've also seen https://pagure.io/freeipa/issue/9119 / https://github.com/freeipa/freeipa/pull/6197 but I haven't tested these patches) Version-Release number of selected component (if applicable): ipa-server-common-4.9.8-6.el9.noarch openssl-3.0.1-14.el9 --- Additional comment from Alexander Bokovoy on 2022-03-05 09:29:23 UTC --- This sounds like https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/KSD3NNVQY7JFDL7DDJCVIYDLR4AXP4R2/ I'm moving this to krb5 to let Julien to investigate. Perhaps we need to change a way we export certificates for PKINIT on IPA side but let's get through krb5 first. Please note that krb5 needs some changes to work with openssl 3.0.1 in RHEL 9 too. --- Additional comment from Alexander Bokovoy on 2022-03-05 09:45:25 UTC --- RFC4556: 3.1.1. Required Algorithms All PKINIT implementations MUST support the following algorithms: o AS reply key enctypes: aes128-cts-hmac-sha1-96 and aes256-cts- hmac-sha1-96 [RFC3962]. o Signature algorithm: sha-1WithRSAEncryption [RFC3370]. o AS reply key delivery method: the Diffie-Hellman key delivery method, as described in Section 3.2.3.1. In addition, implementations of this specification MUST be capable of processing the Extended Key Usage (EKU) extension and the id-pkinit- san (as defined in Section 3.2.2) otherName of the Subject Alternative Name (SAN) extension in X.509 certificates [RFC3280]. RFC8636 allows algorithm agility for PKINIT KDFs: 6. KDF Agility Section 3.2.3.1 of [RFC4556] is updated to define additional key derivation functions (KDFs) to derive a Kerberos protocol key based on the secret value generated by the Diffie-Hellman key exchange. Section 3.2.1 of [RFC4556] is updated to add a new field to the AuthPack structure to indicate which new KDFs are supported by the client. Section 3.2.3 of [RFC4556] is updated to add a new field to the DHRepInfo structure to indicate which KDF is selected by the KDC. The KDF algorithm described in this document (based on [SP80056A]) can be implemented using any cryptographic hash function. A new KDF for PKINIT usage is identified by an object identifier. The following KDF object identifiers are defined: id-pkinit OBJECT IDENTIFIER ::= { iso(1) identified-organization(3) dod(6) internet(1) security(5) kerberosv5(2) pkinit (3) } -- Defined in RFC 4556 and quoted here for the reader. id-pkinit-kdf OBJECT IDENTIFIER ::= { id-pkinit kdf(6) } -- PKINIT KDFs id-pkinit-kdf-ah-sha1 OBJECT IDENTIFIER ::= { id-pkinit-kdf sha1(1) } -- SP800-56A ASN.1 structured hash-based KDF using SHA-1 id-pkinit-kdf-ah-sha256 OBJECT IDENTIFIER ::= { id-pkinit-kdf sha256(2) } -- SP800-56A ASN.1 structured hash-based KDF using SHA-256 id-pkinit-kdf-ah-sha512 OBJECT IDENTIFIER ::= { id-pkinit-kdf sha512(3) } -- SP800-56A ASN.1 structured hash-based KDF using SHA-512 id-pkinit-kdf-ah-sha384 OBJECT IDENTIFIER ::= { id-pkinit-kdf sha384(4) } -- SP800-56A ASN.1 structured hash-based KDF using SHA-384 Where id-pkinit is defined in [RFC4556]. All key derivation functions specified above use the one-step key derivation method described in Section 5.8.2.1 of [SP80056A], choosing the ASN.1 format for FixedInfo, and Section 4.1 of [SP80056C], choosing option 1 for the auxiliary function H. id-pkinit-kdf-ah-sha1 uses SHA-1 [RFC6234] as the hash function. id-pkinit-kdf-ah-sha256, id-pkinit-kdf-ah- sha356, and id-pkinit-kdf-ah-sha512 use SHA-256 [RFC6234], SHA-384 [RFC6234], and SHA-512 [RFC6234], respectively. MIT Kerberos support SHA-1, SHA-256, SHA-512 but does not use SHA-384: (in src/plugins/preauth/pkinit/pkinit_crypto_openssl.c): /* Return the OpenSSL descriptor for the given RFC 5652 OID specified in RFC * 8636. RFC 8636 defines a SHA384 variant, but we don't use it. */ static const EVP_MD * algid_to_md(const krb5_data *alg_id) { if (data_eq(*alg_id, sha1_id)) return EVP_sha1(); if (data_eq(*alg_id, sha256_id)) return EVP_sha256(); if (data_eq(*alg_id, sha512_id)) return EVP_sha512(); return NULL; } The failure message comes from pkini_as_rep_parse() function on the client side: /* * Parse PA-PK-AS-REP message. Optionally evaluates the message's * certificate chain. * Optionally returns various components. */ .... switch(kdc_reply->choice) { case choice_pa_pk_as_rep_dhInfo: pkiDebug("as_rep: DH key transport algorithm\n"); #ifdef DEBUG_ASN1 print_buffer_bin(kdc_reply->u.dh_Info.dhSignedData.data, kdc_reply->u.dh_Info.dhSignedData.length, "/tmp/client_kdc_signeddata"); #endif if ((retval = cms_signeddata_verify(context, plgctx->cryptoctx, reqctx->cryptoctx, reqctx->idctx, CMS_SIGN_SERVER, reqctx->opts->require_crl_checking, (unsigned char *) kdc_reply->u.dh_Info.dhSignedData.data, kdc_reply->u.dh_Info.dhSignedData.length, (unsigned char **)&dh_data.data, &dh_data.length, NULL, NULL, NULL)) != 0) { pkiDebug("failed to verify pkcs7 signed data\n"); TRACE_PKINIT_CLIENT_REP_DH_FAIL(context); <---- this is the '[2406] 1646386032.615666: PKINIT client could not verify DH reply' message goto cleanup; } TRACE_PKINIT_CLIENT_REP_DH(context); break; .... Luca, Ade, could you please use on the client side update-crypto-policies --set DEFAULT:SHA-1 and retry kinit attempt on the client side? --- Additional comment from Luca Miccini on 2022-03-07 07:50:21 UTC --- I think there is a typo and it should be "update-crypto-policies --set DEFAULT:SHA1" I am trying it now, will report back --- Additional comment from Luca Miccini on 2022-03-07 08:39:26 UTC --- "update-crypto-policies --set DEFAULT:SHA1" + reboot seems to be OK as workaround, thanks! --- Additional comment from Alexander Bokovoy on 2022-03-07 11:19:01 UTC --- We found that cms_signeddata_create() method, used by both PKINIT client and PKINIT KDC modules, hard-codes SHA-1 as a digest algorithm and uses sha1WithRSAEncryption without any alternatives. Since KDC internally has access to SHA-1 (we have to, for reasons unrelated), it was able to generate the response but on the client side PKINIT client operates in the environment where SHA-1 is disabled, it cannot verify it. This is a generic CMS SignedData message, so it can be made flexible to use actually supported algorithm. The 'other' side which consumes this CMS SignedData message is passing it to OpenSSL's CMS_verify() function which should handle any of supported algorithms. --- Additional comment from Julien Rische on 2022-03-09 12:18:04 UTC --- After a closer look at RFC8636[1] and comparison with the current MIT Kerberos implementation, we realized it was partially implemented: * KDC agility (section 6): Implemented[2] * PK Authenticator checksum agility (section 3): Supposed to rely on KDF agility, but currently only using SHA-1[3][4][5]. * CMS digest algorithm agility (section 4): Not implemented[6]. In case of KDC_ERR_DIGEST_IN_SIGNED_DATA_NOT_ACCEPTED error, the KDC should provide a list of supported algorithms to the client. * X.509 certificate signer algorithm agility (section 5): Not implemented. The KDC_ERR_DIGEST_IN_CERT_NOT_ACCEPTED error is defined[7], but never used. The section 7 "Interoperability" describes the behavior between an old client and a KDC implementing algorithm agility and the opposite. In these cases, they fallback to SHA1-based functions, making it necessary in this kind of situations. For now, I will provide a patch using the following OpenSSL function to allow use of SHA-1 signatures in a library context: int ossl_ctx_legacy_digest_signatures_allowed_set(OSSL_LIB_CTX *libctx, int allow, int loadconfig) The actions to take in order to achieve full support of algorithm agility for PKINIT will be discussed upstream. [1] https://datatracker.ietf.org/doc/html/rfc8636 [2] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_crypto_openssl.c#L2353 [3] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_srv.c#L549 [4] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_clnt.c#L122 [5] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_clnt.c#L718 [6] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_crypto_openssl.c#L1683 [7] https://github.com/krb5/krb5/blob/master/src/include/k5-int.h#L406 --- Additional comment from Simo Sorce on 2022-03-09 20:35:43 UTC --- Note that for normal use (non FIPS), the users that need pkinit interop with an older server should be able to set crypto-policies to LEGACY and have it working. That is the mechanism we made available for those cases where SHA-1 signatures are still needed in a deployment. --- Additional comment from Alexander Bokovoy on 2022-03-10 10:55:50 UTC --- Looking at Heimdal implementation, it is already defaulting to SHA-256 for signature and digest algorithm since 2009: https://github.com/heimdal/heimdal/commit/c4c71cc41a2763a23867c7c6a041d1f4f1ebcbf7 https://github.com/heimdal/heimdal/blob/master/lib/hx509/crypto.c#L1513-L1523 the use of the defaults is triggered in sig_process() when a passed-in digest_alg in the context is NULL and the certificate where it would otherwise be picked up is NULL too: https://github.com/heimdal/heimdal/blob/master/lib/hx509/cms.c#L1283-L1291 and https://github.com/heimdal/heimdal/blob/master/lib/hx509/crypto.c#L2644-L2649 This means we can move to default in MIT Kerberos to use SHA-256 in the created CMS signeddata for both client and KDC parts of the PKINIT preauth method. We still need to be able to verify SHA-1 signatures sent by the clients in both normal and FIPS mode as this is explicitly allowed by SP 800-131Ar2[1] table 8. [1] https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-131Ar2.pdf --- Additional comment from Julien Rische on 2022-03-11 15:16:45 UTC --- Upstream PR for using SHA-256 instead of SHA-1 for PKINIT CMS digest/signature: https://github.com/krb5/krb5/pull/1243
FEDORA-2022-e3ba1fca4d has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-e3ba1fca4d
FEDORA-2022-92087bf4a6 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-92087bf4a6
FEDORA-2022-e3ba1fca4d has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-e3ba1fca4d` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-e3ba1fca4d See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-92087bf4a6 has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-92087bf4a6` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-92087bf4a6 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-5ec9b50ced has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-5ec9b50ced
FEDORA-2022-5ec9b50ced has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-5ec9b50ced` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-5ec9b50ced See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-5ec9b50ced has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-92087bf4a6 has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2022-e3ba1fca4d has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.