Created attachment 1693711 [details]
SSL Labs scan report

My workaround does not fix GnuTLS.

# gnutls-cli api.ipify.org
p11-kit: overriding trust for anchor in blacklist: addtrust-external-root.p11-kit
Processed 154 CA certificate(s).
Resolving 'api.ipify.org'...
Connecting to '184.73.165.106:443'...
- Certificate type: X.509
- Got a certificate list of 4 certificates.
- Certificate[0] info:
 - subject `OU=Domain Control Validated,OU=PositiveSSL Wildcard,CN=*.ipify.org', issuer `C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Domain Validation Secure Server CA', RSA key 2048 bits, signed using RSA-SHA256, activated `2018-01-24 00:00:00 UTC', expires `2021-01-23 23:59:59 UTC', SHA-1 fingerprint `a8ec3c8e035158e5a9c0b5fe8cd1b4eced4c09a9'
        Public Key ID:
                8e05c08fb342748ee63ac348448821bc628b8150
        Public key's random art:
                +--[ RSA 2048]----+
                |*oE..            |
                |=o. o.           |
                |+..+ o.          |
                |+++ + ..         |
                |==.  o  S        |
                |.oo .  +         |
                |+. .  . .        |
                |=.               |
                | o               |
                +-----------------+

- Certificate[1] info:
 - subject `C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Domain Validation Secure Server CA', issuer `C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Certification Authority', RSA key 2048 bits, signed using RSA-SHA384, activated `2014-02-12 00:00:00 UTC', expires `2029-02-11 23:59:59 UTC', SHA-1 fingerprint `339cdd57cfd5b141169b615ff31428782d1da639'
- Certificate[2] info:
 - subject `C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Certification Authority', issuer `C=SE,O=AddTrust AB,OU=AddTrust External TTP Network,CN=AddTrust External CA Root', RSA key 4096 bits, signed using RSA-SHA384, activated `2000-05-30 10:48:38 UTC', expires `2020-05-30 10:48:38 UTC', SHA-1 fingerprint `f5ad0bcc1ad56cd150725b1c866c30ad92ef21b0'
- Certificate[3] info:
 - subject `C=SE,O=AddTrust AB,OU=AddTrust External TTP Network,CN=AddTrust External CA Root', issuer `C=SE,O=AddTrust AB,OU=AddTrust External TTP Network,CN=AddTrust External CA Root', RSA key 2048 bits, signed using RSA-SHA1, activated `2000-05-30 10:48:38 UTC', expires `2020-05-30 10:48:38 UTC', SHA-1 fingerprint `02faf3e291435468607857694df5e45b68851868'
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate. 
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.

Created attachment 1693778 [details]
/etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit

(In reply to Christian Heimes from comment #3)
> My workaround does not fix GnuTLS.

Daiki has a fix for GnuTLS in https://gitlab.com/gnutls/gnutls/-/merge_requests/1271. Would have been really nice to have known about this in advance so the Sunday evening heroics would not have been necessary. :/ I guess a separate bug will be needed if we want to fix GnuTLS in RHEL 7.

(In reply to Michael Catanzaro from comment #5)
> Daiki has a fix for GnuTLS in
> https://gitlab.com/gnutls/gnutls/-/merge_requests/1271. Would have been
> really nice to have known about this in advance so the Sunday evening
> heroics would not have been necessary. :/ I guess a separate bug will be
> needed if we want to fix GnuTLS in RHEL 7.

Daiki has submitted the fix to Fedora 32, https://bodhi.fedoraproject.org/updates/FEDORA-2020-6ec1d85ab1 . I have tested the new build in a podman container and are happy to confirm that his changeset addresses the issue for AddTrust.

We knew about the expiring cert well in advance (#1721955), but nobody was aware how the issue was going to impact OpenSSL 1.0.2 and GnuTLS. :(

Thanks for the heroics!

I'll pile onto this here and point out that it appears python 2.7 on RHEL 7 DOES NOT have issues with the new certificate, but python36 does:

Python 2.7.5 (default, Sep 26 2019, 13:23:47) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://example.com/')
>>>

Whereas:

Python 3.6.8 (default, Sep 26 2019, 11:57:09) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://example.com/')
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 394, in connect
    ssl_context=context,
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 773, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 1033, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 645, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 438, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 651, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 502, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='exmaple.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))

I know a separate report is needed, but there is good info here, and perhaps others will find this useful.

There are no updates packages yet. Is any customer seeing any kind of specific issues? Or are customer merely worried that the presence of the expire AddTrust External Root cert is going to impact systems?

RHEL 8:
* OpenSSL and NSS based applications are not impacted if the service has an alternative trust chain.
* GnuTLS based applications are impacted and will require a new version of GnuTLS to work.

RHEL 7:
* OpenSSL based applications are affected. As a workaround the customer can blacklist the certificate and regenerate the cert bundles until Bob has created a new ca-certificate package.
* GnuTLS based applications are impacted and will require a new version of GnuTLS to work.

According to Ryan Sleevi Java is not affected.

To blacklist and rebuild the cert bundles the customer can run the commands as root:

---
# trust dump --filter "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;type=cert" > /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
# update-ca-trust extract
---

The "trust" command is part of p11-kit-trust. It's safe to create the file addtrust-external-root.p11-kit on one machine and copy it to other machines.


All systems:
I recommend to remove all cross-signed certificates rom all server configurations. For example there are are **two** variants of the "COMODO RSA Certification Authority". One is a self-signed root CA and one is an intermediate CA that is signed by "AddTrust External CA Root". The cross-signed certificate is:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            27:66:ee:56:eb:49:f3:8e:ab:d7:70:a2:fc:84:de:22
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
        Validity
            Not Before: May 30 10:48:38 2000 GMT
            Not After : May 30 10:48:38 2020 GMT
        Subject: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
        ...
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A
            X509v3 Subject Key Identifier: 
                BB:AF:7E:02:3D:FA:A6:F1:3C:84:8E:AD:EE:38:98:EC:D9:32:32:D4

(In reply to Erinn Looney-Triggs from comment #8)
> I'll pile onto this here and point out that it appears python 2.7 on RHEL 7
> DOES NOT have issues with the new certificate, but python36 does:

I'm getting opposite results on RHEL 7.9 nightly:

$ python2
Python 2.7.5 (default, Mar 20 2020, 17:08:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> urllib2.urlopen
<function urlopen at 0x7f44a12c00c8>
>>> urllib2.urlopen("https://api.ipify.org/")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>

$ python3
Python 3.6.8 (default, Apr  3 2020, 12:24:13) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request                           
>>> urllib.request.urlopen("https://api.ipify.org/")
<http.client.HTTPResponse object at 0x7fc3fcf772b0>


Please open a new bug and CC me. I'm the upstream owner of Python's ssl module and curious about your findings.

I'm forwarding this to the customer to see if this gets them going in the meantime.

I also get this on centos7 with python2

As quick fix, I ended up editing /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and commenting out this entry:

# AddTrust External Root
#-----BEGIN CERTIFICATE-----
#MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
#MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs
#IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290
#MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux
#FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h
#bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v
#dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt
#H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9
#uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX
#mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX
#a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN
#E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0
#WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD
#VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0
#Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU
#cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx
#IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN
#AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH
#YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5
#6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC
#Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX
#c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a
#mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ=
#-----END CERTIFICATE-----


Depending on the version / update status of centos7, this file had only this AddTrust CA, or a few others too. Disabling only this one is enough to fix the immediate problem.
Package updates soon will hopefully correct it properly.

Created attachment 1693957 [details]
trustflag.py demo script

OpenSSL 1.1.0 has enabled X509_V_FLAG_TRUSTED_FIRST flag by default, https://www.openssl.org/docs/man1.1.0/man3/X509_VERIFY_PARAM_set_flags.html . The flag is available in OpenSSL 1.0.2 but not enabled by default. However Python has enabled the flag in its ssl module for more than five years and hadn't got any complain. The upstream patch https://github.com/python/cpython/commit/fdb19715879babc580f63bc129f5b0ff46482d1c never landed in RHEL 7.

I can confirm that setting the flags resolves the problem for OpenSSL on RHEL 7. The flag is not set in python2. Setting the flags gets rid of the cert validation error.

# python2 trustflag.py 
verify_flags 0x0L
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>
verify_flags 0x8000L
success
# python3 trustflag.py 
verify_flags 0x8000
success
verify_flags 0x8000
success

Setting the flight by default in OpenSSL is a one-line change and would make our copy of OpenSSL 1.0.2 more behave like OpenSSL 1.1.1 on RHEL 8.

When I am using "openssl s_client -connect domain_name" it shows expired certificate in chain i.e AddTrust External. I have commented this cert in ca-bundle.crt and my curl command is running for https://url after commenting the cert in ca-bundle.crt but as I said s_client shows this expired certficate in chain, do we need a fix for that, if yes? then what possible could be the fix. I am using AWS Linux instance.

We're seeing this disrupt various automated systems on our RHEL 6 and 7 machines here as we're using GoDaddy certificates on our public servers. I've seen RHEL 7 and 8 mentioned here, but will RHEL 6 get some love? It's out of most support phases, but not quite EOL yet.

I'm afraid the workaround doesn't work for various reasons:

a) "/usr/bin/trust" isn't packaged for RHEL 7

b) Generating the file on Fedora 31 and transferring it over gives:

> p11-kit: addtrust-external-root.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute
> p11-kit: addtrust-external-root.p11-kit: nss-server-distrust-after: invalid or unsupported attribute
> p11-kit: addtrust-external-root.p11-kit: nss-email-distrust-after: invalid or unsupported attribute

c) Removing those still gives:

> update-ca-trust: Warning: The dynamic CA configuration feature is in the disabled state
> p11-kit: overriding trust for anchor in blacklist: addtrust-external-root.p11-kit
> p11-kit: duplicate 'AddTrust External Root' certificate found in: ca-bundle.legacy.crt

And I'm still getting failures from openssl:

> # openssl s_client -connect XXX:443
> ...
>     Verify return code: 10 (certificate has expired)

Created attachment 1694101 [details]
/etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit

Sorry, I didn't check older RHEL 7 versions for the presence of the 'trust' command.

Here is an updated file that works with older versions of update-ca-trust. You can copy the file instead of generating the blacklist file in each machine. You still have to run "update-ca-trust", though.

The client-side workaround with blacklist + update-ca-trust does not work for me on RHEL 6 or RHEL 7.3 with OpenSSL 1.0.1. It only works on RHEL 7.4 and newer with OpenSSL 1.0.2. It seems like OpenSSL 1.0.1 cannot establish a correct trust chain. It fails with "Verify return code: 19 (self signed certificate in certificate chain)".

The blacklist approach is a workaround any way. The correct fix for the problem is Martin Poole's approach. Customers have to update the certificate chain on their servers and remove all alternative trust chain and intermediate CAs that are connected to "AddTrust External Root". This will address the problem for all clients.

Ah, all right. I missed that detail. I can confirm that removing that intermediate certificate on the server fixed the issue on all RHEL 6 and RHEL 7 clients right away.

(In reply to Christian Heimes from comment #18)
> The client-side workaround with blacklist + update-ca-trust does not work
> for me on RHEL 6 or RHEL 7.3 with OpenSSL 1.0.1. It only works on RHEL 7.4
> and newer with OpenSSL 1.0.2. It seems like OpenSSL 1.0.1 cannot establish a
> correct trust chain. It fails with "Verify return code: 19 (self signed
> certificate in certificate chain)".

Historical interest, but I think that was fixed in Fedora bug #1166614 with this patch: https://src.fedoraproject.org/rpms/openssl/blob/f21/f/openssl-1.0.1k-alt-chains.patch

Can you please reply over my comment?

I have a number RHEL 7.8 hosts that I have been "fixing" by commenting or deleting the following out of /etc/pki/tls/certs/ca-bundle.crt
# AddTrust External CA Root
# USERTrust RSA Certification Authority

When I tried using the blacklist commands it results in a different error, but does not remedy the issue. 

# trust dump --filter "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;type=cert" > /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
# trust dump --filter "pkcs11:id=%53%79%bf%5a%aa%2b%4a%cf%54%80%e1%d8%9b%c0%9d%f2%b2%03%66%cb;type=cert" > /etc/pki/ca-trust/source/blacklist/USERTrust-RSA-inter.p11-kit
# update-ca-trust

After running these commands and testing the cert with: "openssl s_client -connect ldap.example.com", I get "Verify return code: 2 (unable to get issuer certificate)", rather than the expiration error. 

Not sure why the blacklist doesn't work but commenting or delete the CA entries from that file does.

Why are you blocking the "USERTrust RSA Certification Authority" CA cert? There are two different " USERTrust RSA Certification Authority" certificates. One is a root CA certificate and the other is a cross-signed intermediate CA certificate.

You blocked just blocked the good root CA. That cert is a valid and not-expired root CA. 

$ trust dump --filter "pkcs11:id=%53%79%bf%5a%aa%2b%4a%cf%54%80%e1%d8%9b%c0%9d%f2%b2%03%66%cb;type=cert" | openssl x509 -text 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            01:fd:6d:30:fc:a3:ca:51:a8:1b:bc:64:0e:35:03:2d
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
        Validity
            Not Before: Feb  1 00:00:00 2010 GMT
            Not After : Jan 18 23:59:59 2038 GMT
        Subject: C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
        ..
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE


The problematic intermediate, cross-signed CA certificate looks like this. Please note that it has the same subject key id and subject, but a different issuer and an authority key id.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            13:ea:28:70:5b:f4:ec:ed:0c:36:63:09:80:61:43:36
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
        Validity
            Not Before: May 30 10:48:38 2000 GMT
            Not After : May 30 10:48:38 2020 GMT
        Subject: C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
        ...
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A

            X509v3 Subject Key Identifier: 
                53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB

(In reply to Brandon Wise from comment #22)
> I have a number RHEL 7.8 hosts that I have been "fixing" by commenting or
> deleting the following out of /etc/pki/tls/certs/ca-bundle.crt
> # AddTrust External CA Root
> # USERTrust RSA Certification Authority
> 
> When I tried using the blacklist commands it results in a different error,
> but does not remedy the issue. 
> 
> # trust dump --filter
> "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;
> type=cert" >
> /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
> # trust dump --filter
> "pkcs11:id=%53%79%bf%5a%aa%2b%4a%cf%54%80%e1%d8%9b%c0%9d%f2%b2%03%66%cb;
> type=cert" > /etc/pki/ca-trust/source/blacklist/USERTrust-RSA-inter.p11-kit
> # update-ca-trust
> 
> After running these commands and testing the cert with: "openssl s_client
> -connect ldap.example.com", I get "Verify return code: 2 (unable to get
> issuer certificate)", rather than the expiration error. 
> 
> Not sure why the blacklist doesn't work but commenting or delete the CA
> entries from that file does.

I wanted to make a quick note to expand on this a touch from our experience. Removing/commenting the certs out of /etc/pki/tls/certs/ca-bundle.crt must be done in conjunction with blacklisting the two certs, not in place of. If the certs are commented or removed and not blacklisted, they'll be added back in the next time update-ca-trust is run. That could make for some headaches down the road if it gets run as part of an install script.

Also thank you for posting this, it really relieved a lot of headache for us!

(In reply to JMU Libraries from comment #24)
> I wanted to make a quick note to expand on this a touch from our experience.
> Removing/commenting the certs out of /etc/pki/tls/certs/ca-bundle.crt must
> be done in conjunction with blacklisting the two certs, not in place of. If
> the certs are commented or removed and not blacklisted, they'll be added
> back in the next time update-ca-trust is run. That could make for some
> headaches down the road if it gets run as part of an install script.

You should not edit the ca-bundle.crt manually. The file is auto-generated by update-ca-trust command. Also only block "AddTrust External CA Root". Do not block the "USERTrust RSA Certification Authority" root CA certificate with serial number 01:fd:6d:30:fc:a3:ca:51:a8:1b:bc:64:0e:35:03:2d. The certificate is required and must not be blacklisted.

Step 1)
Block the "AddTrust External CA Root" certificate. You can either copy my attchment addtrust-external-root.p11-kit to /etc/pki/ca-trust/source/blacklist/ or create the p11-kit file with the command

   trust dump --filter "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;type=cert" > /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit

Step 2)
Regenerate cert bundles with the command 

   update-ca-trust extract

The command updates /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem, /etc/pki/tls/certs/ca-bundle.crt, and several other bundles.

(In reply to Christian Heimes from comment #25)
>    trust dump --filter
> "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;
> type=cert" >
> /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
>
>    update-ca-trust extract

The above steps did not give the expected result for me.

Created attachment 1694350 [details]
sectigo download

The Red Hat Knowledge Base (Article) 5117881 says:

The replacement intermediate certificate can be found on the page How to Download & Install Sectigo Intermediate Certificates - RSA and is the download labelled USERTrust RSA Root xSigned using AAA CA [ Cross Signed ]

However, by typing it ctrl-f there are multiple choices. This can confuse users.

(In reply to Agostino Sarubbo from comment #26)
> The above steps did not give the expected result for me.

I'm unable to assist you without detailed information about your system, package versions, commands, and error messages.

(In reply to Christian Heimes from comment #28)
> (In reply to Agostino Sarubbo from comment #26)
> > The above steps did not give the expected result for me.
> 
> I'm unable to assist you without detailed information about your system,
> package versions, commands, and error messages.

I'm running ansible from a Gentoo machine and the target is a Centos7 machine.
When trying to get an object from S3 with aws_s3 I get SSL verification failed

# rpm -qa | grep boto
python-boto3-1.4.6-5.el7.noarch

To reproduce the issue you should have an S3 account.

The problem is reproducible on each Centos7 machine I have.

Obviously everything worked before 30May

> I'm running ansible from a Gentoo machine and the target is a Centos7
> machine.
> When trying to get an object from S3 with aws_s3 I get SSL verification
> failed
> 
> # rpm -qa | grep boto
> python-boto3-1.4.6-5.el7.noarch
> 
> To reproduce the issue you should have an S3 account.
> 
> The problem is reproducible on each Centos7 machine I have.
> 
> Obviously everything worked before 30May

Could you please open a CentOS bug at https://bugs.centos.org/ ? Include additional information like distro version and openssl version.

CentOS doesn't fix bugs in upstream packages but will inherit the fix once Red Hat release the RHEL packages. Once RHEL packages are available and pushed to git.centos.rg by RH RCM, they will be picked up and rebuilt for CentOS. Until then you'll have to use one of the workarounds in this bz.

Statement

Our current recommendation which should ensure the quickest resolution is to delete any expired intermediate certificates in their application files.

There is no need to delete or blacklist any certificates in the the system provided bundle.

The basics of this are covered in the following support article.

    Sectigo Root and Intermediate Certificate Expiry - May 2020
    https://access.redhat.com/articles/5117881

Any specific problems should be raised as support tickets.

what if we can not change APP side easily. blacklisting addtrust results in following error 

Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA256
    Session-ID: 2964B5956EDAD44CFB441869A5A15EEC8CBA6A1517F0948BFDB14AC78EAFBBFD
    Session-ID-ctx: 
    Master-Key: C107E531A889CBE17C0619CF8DD3678B67C4FC8455D0B6219CD9AEDA0B1E827C6AA827B9C30209A623755C9652D52D12
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1591122262
    Timeout   : 300 (sec)
    Verify return code: 2 (unable to get issuer certificate)

(In reply to Martin Poole from comment #32)
> Statement
> 
> Our current recommendation which should ensure the quickest resolution is to
> delete any expired intermediate certificates in their application files.
> 

Simply commenting out SSLCertificateChainFile in the Apache config where it used the old intermediate certificate does not work for me.
All legacy clients such as curl on RHEL7 still fail.  I had to update SSLCertificateChainFile on the server with the new intermediate/chain certificate that expires in 2038 to get those legacy clients to work with the server.

My apologies, it would seem my update was not clear enough.


I meant that the expired certificates within the file specified by SSLCertificateChainFile should be removed.

For example, if the chain illustrated in the head of this bug https://bugzilla.redhat.com/show_bug.cgi?id=1842174#c0 was provided by httpd with mod_ssl the SSLCertificateChainFile would have contained two certificates

    Serial Number:
            2b:2e:6e:ea:d9:75:36:6c:14:8a:6e:db:a3:7c:8c:07
    Subject: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
        Validity
            Not Before: Feb 12 00:00:00 2014 GMT
            Not After : Feb 11 23:59:59 2029 GMT
    Issuer: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority

and

    Serial Number:
            27:66:ee:56:eb:49:f3:8e:ab:d7:70:a2:fc:84:de:22
    Subject: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
        Validity
            Not Before: May 30 10:48:38 2000 GMT
            Not After : May 30 10:48:38 2020 GMT
    Issuer: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root


The fix being to simply delete the second certificate.

I see.  Yes.  Taking the old chain cert file and removing the expired certs from it did indeed create a working chain file for me.

(In reply to Christian Heimes from comment #30)
> Could you please open a CentOS bug at https://bugs.centos.org/ ? Include
> additional information like distro version and openssl version.

Done, the response was:
####
CentOS is a rebuild of the sources used to create RHEL. We do not modify anything except to remove branding and logos.  If/when RH fixes the issue and incorporates it into RHEL and releases a patched version, then CentOS will pick it up and rebuild it. Until then  you'll have to use one of the workarounds in the bug reported that you have referenced
####

It looks like a rubber wall, but this is an old story....at least for me.

The work is ready to go.

1,3: We have 30 customer cases waiting, this is a high profile bug that adversely affects many customers.
We want to show that we are responsive and release an 7.8.z package, but we need 7.9 acks to be able to commit the same code in dist-git

2: The risk is low, we have automated tests and well defined QA procedures to test ca-cert bundle changes.

See comment 41.

The same situation will hit later this year when the DST Root CA X3, used by Let's Encrypt, will expire:
https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
Let's Encrypt opted for compatibility with old Android over compatibility with old OpenSSL, so RHEL 7 will be impacted by this.

So it's still imperative to enable X509_V_FLAG_TRUSTED_FIRST by default in openssl, as this can not be done through configuration.

Alternatively, RedHat could push an updated ca-certificates package with expired CA's removed, to avoid the issue altogether.

Administrators can do that manually as suggested here: https://access.redhat.com/articles/5117881