Bug 2067121

+++ This bug was initially created as a clone of Bug #2060798 +++

Description of problem:

we hit this while trying to install OSP on centos9/rhel9.

After having installed freeipa we use https://docs.ansible.com/ansible/latest/collections/community/general/ipa_role_module.html to create a simple role:

    - name: add nova host manager role
      ipa_role:
        name: Nova Host Manager
        description: Nova Host Manager
        ipa_user: "{{ ipa_principal }}"
        ipa_pass: "{{ ipa_password }}"
        privilege:
          - Nova Host Management

This fails with the following error in /var/log/httpd/error_log:

[Fri Mar 04 07:58:03.548673 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] mod_wsgi (pid=13804): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[Fri Mar 04 07:58:03.549291 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] Traceback (most recent call last):
[Fri Mar 04 07:58:03.549333 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]   File "/usr/lib/python3.9/site-packages/ipaserver/wsgi.py", line 71, in application
[Fri Mar 04 07:58:03.549338 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]     return api.Backend.wsgi_dispatch(environ, start_response)
[Fri Mar 04 07:58:03.549343 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]   File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 301, in __call__
[Fri Mar 04 07:58:03.549345 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]     return self.route(environ, start_response)
[Fri Mar 04 07:58:03.549349 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]   File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 313, in route
[Fri Mar 04 07:58:03.549351 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]     return app(environ, start_response)
[Fri Mar 04 07:58:03.549356 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]   File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 1066, in __call__
[Fri Mar 04 07:58:03.549369 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]     result = attempt_kinit(user_principal, password,
[Fri Mar 04 07:58:03.549372 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]   File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 996, in attempt_kinit
[Fri Mar 04 07:58:03.549374 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]     self.kinit(user_principal, password,
[Fri Mar 04 07:58:03.549377 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]   File "/usr/lib/python3.9/site-packages/ipaserver/rpcserver.py", line 1094, in kinit
[Fri Mar 04 07:58:03.549379 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]     kinit_armor(
[Fri Mar 04 07:58:03.549382 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]   File "/usr/lib/python3.9/site-packages/ipalib/install/kinit.py", line 129, in kinit_armor
[Fri Mar 04 07:58:03.549384 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]     run(args, env=env, raiseonerr=True, capture_error=True)
[Fri Mar 04 07:58:03.549387 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]   File "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 598, in run
[Fri Mar 04 07:58:03.549389 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356]     raise CalledProcessError(
[Fri Mar 04 07:58:03.549420 2022] [wsgi:error] [pid 13804:tid 14167] [remote 172.30.4.2:54356] ipapython.ipautil.CalledProcessError: CalledProcessError(Command ['/usr/bin/kinit', '-n', '-c', '/run/ipa/ccaches/armor_13804', '-X', 'X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt', '-X', 'X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem'] returned non-zero exit status 1: 'kinit: Cannot read password while getting initial credentials\\n')

trying to run the kinit command manually:

[root@freeipa-0 ~]# export KRB5_TRACE=/dev/stdout         
[root@freeipa-0 ~]# kinit -n -c /tmp/armor_13804 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem                                                 
[2406] 1646386032.615628: Getting initial credentials for WELLKNOWN/ANONYMOUS                                                                                                                              
[2406] 1646386032.615630: Sending unauthenticated request                                                                                                                                                          
[2406] 1646386032.615631: Sending request (178 bytes) to BGP.FTW                                                                                                                                                   
[2406] 1646386032.615632: Initiating TCP connection to stream 172.30.4.2:88                                                                                                                                        
[2406] 1646386032.615633: Sending TCP request to stream 172.30.4.2:88
[2406] 1646386032.615634: Received answer (526 bytes) from stream 172.30.4.2:88                                                                                                                                    
[2406] 1646386032.615635: Terminating TCP connection to stream 172.30.4.2:88                                                                                                                                        
[2406] 1646386032.615636: Response was from primary KDC                                                                                                                                                            
[2406] 1646386032.615637: Received error from KDC: -1765328359/Additional pre-authentication required                                                                                                              
[2406] 1646386032.615640: Preauthenticating using KDC method data                                                                                                                                                   
[2406] 1646386032.615641: Processing preauth types: PA-PK-AS-REQ (16), PA-FX-FAST (136), PA-ETYPE-INFO2 (19), PA-PKINIT-KX (147), PA-SPAKE (151), PA-ENC-TIMESTAMP (2), PA_AS_FRESHNESS (150), PA-FX-COOKIE (133)   
[2406] 1646386032.615642: Selected etype info: etype aes256-cts, salt "BGP.FTWWELLKNOWNANONYMOUS", params ""                                                                                                       
[2406] 1646386032.615643: Received cookie: MIT1\x00\x00\x00\x01\xea\xb8\x0b\xb4L}\x18\xc3\xbaTc8?\xe4\xb7p\xb2\x8e\xbf\x85\xc0\xa9.\xfc\\xbe\xaa\xeb\xaf&\x0f\xcbv_\T\x06!\xdch\xd4`\xb6\xa8\\xb4E13\xdb\x10\x82A>\x
05\x05\xae\xe5\xder\xd0\xc7A2M\xdf\\x10I\xe1\x99$\x1ct.\xb3Lu\x0fcRd:\x8f^\x1e\x0b\xdf:\xe3\x81p\xd56C.\xdd&t\xd1\xf6PE$Y\xc3G$\xa0\x06\xa2\x9cxU\x10\xe2\x1fI\xf5\x7f\xae\xa28\xe7\xb8cS\x09\x9b\xc3              
[2406] 1646386032.615644: Preauth module pkinit (147) (info) returned: 0/Success                                                                                                                                   
[2406] 1646386032.615645: PKINIT client received freshness token from KDC                                                                                                                                          
[2406] 1646386032.615646: Preauth module pkinit (150) (info) returned: 0/Success                                                                                                                                   
[2406] 1646386032.615647: PKINIT loading CA certs and CRLs from FILE                                                                                                                                                
[2406] 1646386032.615648: PKINIT loading CA certs and CRLs from FILE                                                                                                                                               
[2406] 1646386032.615649: PKINIT loading CA certs and CRLs from FILE                                                                                                                                               
[2406] 1646386032.615650: PKINIT client computed kdc-req-body checksum 9/3EA2950DCF38A9868B8B8C5E040F69865752C33A                                                                                                  
[2406] 1646386032.615652: PKINIT client making DH request                                                                                                                                                          
[2406] 1646386032.615653: Preauth module pkinit (16) (real) returned: 0/Success      
[2406] 1646386032.615654: Produced preauth for next request: PA-FX-COOKIE (133), PA-PK-AS-REQ (16)
[2406] 1646386032.615655: Sending request (1648 bytes) to BGP.FTW                                                                                                                                                  
[2406] 1646386032.615656: Initiating TCP connection to stream 172.30.4.2:88                                                                                                                                        
[2406] 1646386032.615657: Sending TCP request to stream 172.30.4.2:88                                                                                                                                              
[2406] 1646386032.615658: Received answer (2558 bytes) from stream 172.30.4.2:88                                                                                                                                   
[2406] 1646386032.615659: Terminating TCP connection to stream 172.30.4.2:88                                                                                                                                       
[2406] 1646386032.615660: Response was from primary KDC                                                                                                                                                            
[2406] 1646386032.615661: Processing preauth types: PA-PK-AS-REP (17), PA-PKINIT-KX (147)                                                                                                                          
[2406] 1646386032.615662: Preauth module pkinit (147) (info) returned: 0/Success                                                                                                                                   
[2406] 1646386032.615663: PKINIT OpenSSL error: Failed to verify CMS message                                                                                                                                       
[2406] 1646386032.615664: PKINIT OpenSSL error: error:1700006B:CMS routines::content type not enveloped data                                                                                                       
[2406] 1646386032.615665: PKINIT OpenSSL error: error:03000098:digital envelope routines::invalid digest                                                                                                           
[2406] 1646386032.615666: PKINIT client could not verify DH reply                                                                                                                                                  
[2406] 1646386032.615667: Preauth module pkinit (17) (real) returned: -1765328320/Failed to verify CMS message: content type not enveloped data                                                                    
[2406] 1646386032.615668: Produced preauth for next request: (empty)                                                                                                                                               
[2406] 1646386032.615669: Getting AS key, salt "BGP.FTWWELLKNOWNANONYMOUS", params ""                                                                                                                              
Password for WELLKNOWN/ANONYMOUS:                                                                                                                                                                          
kinit: Password read interrupted while getting initial credentials                                                 

AFAICS there were recent changes in openssl that seem related:

* Fri Feb 25 2022 Clemens Lang <cllang> - 1:3.0.1-14
- Prevent use of SHA1 with ECDSA
- Resolves: rhbz#2031742

* Fri Feb 25 2022 Dmitry Belyavskiy <dbelyavs> - 1:3.0.1-13
- OpenSSL will generate keys with prime192v1 curve if it is provided using explicit parameters
- Resolves: rhbz#1977867

* Thu Feb 24 2022 Peter Robinson <pbrobinson> - 1:3.0.1-12
- Support KBKDF (NIST SP800-108) with an R value of 8bits
- Resolves: rhbz#2027261

* Wed Feb 23 2022 Clemens Lang <cllang> - 1:3.0.1-11
- Allow SHA1 usage in MGF1 for RSASSA-PSS signatures
- Resolves: rhbz#2031742

* Wed Feb 23 2022 Dmitry Belyavskiy <dbelyavs> - 1:3.0.1-10
- rebuilt

* Tue Feb 22 2022 Clemens Lang <cllang> - 1:3.0.1-9
- Allow SHA1 usage in HMAC in TLS
- Resolves: rhbz#2031742

* Tue Feb 22 2022 Dmitry Belyavskiy <dbelyavs> - 1:3.0.1-8
- OpenSSL will generate keys with prime192v1 curve if it is provided using explicit parameters
- Resolves: rhbz#1977867
- pkcs12 export broken in FIPS mode
- Resolves: rhbz#2049265

* Tue Feb 22 2022 Clemens Lang <cllang> - 1:3.0.1-8
- Disable SHA1 signature creation and verification by default
- Set rh-allow-sha1-signatures = yes to re-enable
- Resolves: rhbz#2031742


If I downgrade openssl to 3.0.1-5 things work as expected.

(I've also seen https://pagure.io/freeipa/issue/9119 / https://github.com/freeipa/freeipa/pull/6197 but I haven't tested these patches)

Version-Release number of selected component (if applicable):

ipa-server-common-4.9.8-6.el9.noarch
openssl-3.0.1-14.el9

--- Additional comment from Alexander Bokovoy on 2022-03-05 09:29:23 UTC ---

This sounds like https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/KSD3NNVQY7JFDL7DDJCVIYDLR4AXP4R2/

I'm moving this to krb5 to let Julien to investigate. Perhaps we need to change a way we export certificates for PKINIT on IPA side but let's get through krb5 first.

Please note that krb5 needs some changes to work with openssl 3.0.1 in RHEL 9 too.

--- Additional comment from Alexander Bokovoy on 2022-03-05 09:45:25 UTC ---

RFC4556:

3.1.1.  Required Algorithms

   All PKINIT implementations MUST support the following algorithms:

   o  AS reply key enctypes: aes128-cts-hmac-sha1-96 and aes256-cts-
      hmac-sha1-96 [RFC3962].

   o  Signature algorithm: sha-1WithRSAEncryption [RFC3370].

   o  AS reply key delivery method: the Diffie-Hellman key delivery
      method, as described in Section 3.2.3.1.

   In addition, implementations of this specification MUST be capable of
   processing the Extended Key Usage (EKU) extension and the id-pkinit-
   san (as defined in Section 3.2.2) otherName of the Subject
   Alternative Name (SAN) extension in X.509 certificates [RFC3280].

RFC8636 allows algorithm agility for PKINIT KDFs:

6.  KDF Agility

   Section 3.2.3.1 of [RFC4556] is updated to define additional key
   derivation functions (KDFs) to derive a Kerberos protocol key based
   on the secret value generated by the Diffie-Hellman key exchange.
   Section 3.2.1 of [RFC4556] is updated to add a new field to the
   AuthPack structure to indicate which new KDFs are supported by the
   client.  Section 3.2.3 of [RFC4556] is updated to add a new field to
   the DHRepInfo structure to indicate which KDF is selected by the KDC.

   The KDF algorithm described in this document (based on [SP80056A])
   can be implemented using any cryptographic hash function.

   A new KDF for PKINIT usage is identified by an object identifier.
   The following KDF object identifiers are defined:

   id-pkinit OBJECT IDENTIFIER ::=
            { iso(1) identified-organization(3) dod(6) internet(1)
              security(5) kerberosv5(2) pkinit (3) }
       -- Defined in RFC 4556 and quoted here for the reader.

   id-pkinit-kdf OBJECT IDENTIFIER      ::= { id-pkinit kdf(6) }
       -- PKINIT KDFs

   id-pkinit-kdf-ah-sha1 OBJECT IDENTIFIER
       ::= { id-pkinit-kdf sha1(1) }
       -- SP800-56A ASN.1 structured hash-based KDF using SHA-1

   id-pkinit-kdf-ah-sha256 OBJECT IDENTIFIER
       ::= { id-pkinit-kdf sha256(2) }
       -- SP800-56A ASN.1 structured hash-based KDF using SHA-256

   id-pkinit-kdf-ah-sha512 OBJECT IDENTIFIER
       ::= { id-pkinit-kdf sha512(3) }
       -- SP800-56A ASN.1 structured hash-based KDF using SHA-512

   id-pkinit-kdf-ah-sha384 OBJECT IDENTIFIER
       ::= { id-pkinit-kdf sha384(4) }
       -- SP800-56A ASN.1 structured hash-based KDF using SHA-384

   Where id-pkinit is defined in [RFC4556].  All key derivation
   functions specified above use the one-step key derivation method
   described in Section 5.8.2.1 of [SP80056A], choosing the ASN.1 format
   for FixedInfo, and Section 4.1 of [SP80056C], choosing option 1 for
   the auxiliary function H.  id-pkinit-kdf-ah-sha1 uses SHA-1 [RFC6234]
   as the hash function.  id-pkinit-kdf-ah-sha256, id-pkinit-kdf-ah-
   sha356, and id-pkinit-kdf-ah-sha512 use SHA-256 [RFC6234], SHA-384
   [RFC6234], and SHA-512 [RFC6234], respectively.


MIT Kerberos support SHA-1, SHA-256, SHA-512 but does not use SHA-384:

(in src/plugins/preauth/pkinit/pkinit_crypto_openssl.c):

/* Return the OpenSSL descriptor for the given RFC 5652 OID specified in RFC
 * 8636.  RFC 8636 defines a SHA384 variant, but we don't use it. */
static const EVP_MD *
algid_to_md(const krb5_data *alg_id)
{
    if (data_eq(*alg_id, sha1_id))
        return EVP_sha1();
    if (data_eq(*alg_id, sha256_id))
        return EVP_sha256();
    if (data_eq(*alg_id, sha512_id))
        return EVP_sha512();
    return NULL;
}

The failure message comes from pkini_as_rep_parse() function on the client side:

/*
 * Parse PA-PK-AS-REP message. Optionally evaluates the message's
 * certificate chain.
 * Optionally returns various components.
 */

....

    switch(kdc_reply->choice) {
    case choice_pa_pk_as_rep_dhInfo:
        pkiDebug("as_rep: DH key transport algorithm\n");
#ifdef DEBUG_ASN1
        print_buffer_bin(kdc_reply->u.dh_Info.dhSignedData.data,
                         kdc_reply->u.dh_Info.dhSignedData.length, "/tmp/client_kdc_signeddata");
#endif
        if ((retval = cms_signeddata_verify(context, plgctx->cryptoctx,
                                            reqctx->cryptoctx, reqctx->idctx, CMS_SIGN_SERVER,
                                            reqctx->opts->require_crl_checking,
                                            (unsigned char *)
                                            kdc_reply->u.dh_Info.dhSignedData.data,
                                            kdc_reply->u.dh_Info.dhSignedData.length,
                                            (unsigned char **)&dh_data.data,
                                            &dh_data.length,
                                            NULL, NULL, NULL)) != 0) {
            pkiDebug("failed to verify pkcs7 signed data\n");
            TRACE_PKINIT_CLIENT_REP_DH_FAIL(context);  <---- this is the '[2406] 1646386032.615666: PKINIT client could not verify DH reply' message
            goto cleanup;
        }
        TRACE_PKINIT_CLIENT_REP_DH(context);
        break;
....


Luca, Ade, could you please use on the client side

 update-crypto-policies --set DEFAULT:SHA-1

and retry kinit attempt on the client side?

--- Additional comment from Luca Miccini on 2022-03-07 07:50:21 UTC ---

I think there is a typo and it should be "update-crypto-policies --set DEFAULT:SHA1" 
I am trying it now, will report back

--- Additional comment from Luca Miccini on 2022-03-07 08:39:26 UTC ---

"update-crypto-policies --set DEFAULT:SHA1" + reboot seems to be OK as workaround, thanks!

--- Additional comment from Alexander Bokovoy on 2022-03-07 11:19:01 UTC ---

We found that cms_signeddata_create() method, used by both PKINIT client and PKINIT KDC modules, hard-codes SHA-1 as a digest algorithm and uses sha1WithRSAEncryption without any alternatives.

Since KDC internally has access to SHA-1 (we have to, for reasons unrelated), it was able to generate the response but on the client side PKINIT client operates in the environment where SHA-1 is disabled, it cannot verify it.

This is a generic CMS SignedData message, so it can be made flexible to use actually supported algorithm. The 'other' side which consumes this CMS SignedData message is passing it to OpenSSL's CMS_verify() function which should handle any of supported algorithms.

--- Additional comment from Julien Rische on 2022-03-09 12:18:04 UTC ---

After a closer look at RFC8636[1] and comparison with the current MIT Kerberos implementation, we realized it was partially implemented:

* KDC agility (section 6): Implemented[2]

* PK Authenticator checksum agility (section 3):
  Supposed to rely on KDF agility, but currently only using SHA-1[3][4][5].

* CMS digest algorithm agility (section 4):
  Not implemented[6]. In case of KDC_ERR_DIGEST_IN_SIGNED_DATA_NOT_ACCEPTED error, the KDC should provide a list of supported algorithms to the client.

* X.509 certificate signer algorithm agility (section 5):
  Not implemented. The KDC_ERR_DIGEST_IN_CERT_NOT_ACCEPTED error is defined[7], but never used.

The section 7 "Interoperability" describes the behavior between an old client and a KDC implementing algorithm agility and the opposite. In these cases, they fallback to SHA1-based functions, making it necessary in this kind of situations.

For now, I will provide a patch using the following OpenSSL function to allow use of SHA-1 signatures in a library context:

  int ossl_ctx_legacy_digest_signatures_allowed_set(OSSL_LIB_CTX *libctx, int allow, int loadconfig)

The actions to take in order to achieve full support of algorithm agility for PKINIT will be discussed upstream.

[1] https://datatracker.ietf.org/doc/html/rfc8636
[2] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_crypto_openssl.c#L2353
[3] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_srv.c#L549
[4] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_clnt.c#L122
[5] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_clnt.c#L718
[6] https://github.com/krb5/krb5/blob/krb5-1.19.2-final/src/plugins/preauth/pkinit/pkinit_crypto_openssl.c#L1683
[7] https://github.com/krb5/krb5/blob/master/src/include/k5-int.h#L406

--- Additional comment from Simo Sorce on 2022-03-09 20:35:43 UTC ---

Note that for normal use (non FIPS), the users that need pkinit interop with an older server should be able to set crypto-policies to LEGACY and have it working.
That is the mechanism we made available for those cases where SHA-1 signatures are still needed in a deployment.

--- Additional comment from Alexander Bokovoy on 2022-03-10 10:55:50 UTC ---

Looking at Heimdal implementation, it is already defaulting to SHA-256 for signature and digest algorithm since 2009:
https://github.com/heimdal/heimdal/commit/c4c71cc41a2763a23867c7c6a041d1f4f1ebcbf7

https://github.com/heimdal/heimdal/blob/master/lib/hx509/crypto.c#L1513-L1523

the use of the defaults is triggered in sig_process() when a passed-in digest_alg in the context is NULL and the certificate where it would otherwise be picked up is NULL too:

https://github.com/heimdal/heimdal/blob/master/lib/hx509/cms.c#L1283-L1291

and

https://github.com/heimdal/heimdal/blob/master/lib/hx509/crypto.c#L2644-L2649

This means we can move to default in MIT Kerberos to use SHA-256 in the created CMS signeddata for both client and KDC parts of the PKINIT preauth method.

We still need to be able to verify SHA-1 signatures sent by the clients in both normal and FIPS mode as this is explicitly allowed by SP 800-131Ar2[1] table 8.

[1] https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-131Ar2.pdf

--- Additional comment from Julien Rische on 2022-03-11 15:16:45 UTC ---

Upstream PR for using SHA-256 instead of SHA-1 for PKINIT CMS digest/signature:
https://github.com/krb5/krb5/pull/1243