Bug 1842174 - AddTrust External Root CA certificate expiration causes cert validation issue
Summary: AddTrust External Root CA certificate expiration causes cert validation issue
Keywords:
Status: POST
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ca-certificates
Version: 7.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Bob Relyea
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On: 1845612
Blocks: 1845707
TreeView+ depends on / blocked
 
Reported: 2020-05-30 19:08 UTC by Christian Heimes
Modified: 2020-07-03 16:44 UTC (History)
33 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1845707 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
SSL Labs scan report (139.63 KB, application/pdf)
2020-05-30 19:16 UTC, Christian Heimes
no flags Details
/etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit (2.47 KB, text/plain)
2020-05-31 08:35 UTC, Christian Heimes
no flags Details
trustflag.py demo script (690 bytes, text/plain)
2020-05-31 21:30 UTC, Christian Heimes
no flags Details
/etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit (2.59 KB, text/plain)
2020-06-01 13:57 UTC, Christian Heimes
no flags Details
sectigo download (97.73 KB, image/png)
2020-06-02 08:29 UTC, Agostino Sarubbo
no flags Details


Links
System ID Priority Status Summary Last Updated
Gitlab gnutls/gnutls/-/issues/1008 None None None 2020-05-31 16:35:28 UTC
Red Hat Bugzilla 1721955 None None None 2020-05-30 19:10:38 UTC
Red Hat Bugzilla 1840767 None None None 2020-05-31 21:47:05 UTC
Red Hat Knowledge Base (Article) 5117881 None None None 2020-06-02 08:03:03 UTC

Internal Links: 1842314 1850306

Description Christian Heimes 2020-05-30 19:08:53 UTC
Description of problem:
The "AddTrust External Root" CA certificate has expired today. There is an alternative chain to another root CA. However OpenSSL 1.0.2 fails to verify the chain if the expired root CA cert is in the trust store.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
        Validity
            Not Before: May 30 10:48:38 2000 GMT
            Not After : May 30 10:48:38 2020 GMT
        Subject: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root


Version-Release number of selected component (if applicable):
ca-certificates-2019.2.32-76.el7_7.noarch
openssl-1.0.2k-19.el7.x86_64


How reproducible:
always

Steps to Reproduce:
1. openssl s_client -connect api.ipify.org:443

Actual results:
# openssl s_client -connect api.ipify.org:443
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify error:num=10:certificate has expired
notAfter=May 30 10:48:38 2020 GMT
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.ipify.org
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFSTCCBDGgAwIBAgIRAJIP0bf+S4iutu1asMNsVmgwDQYJKoZIhvcNAQELBQAw
gZAxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTYwNAYD
VQQDEy1DT01PRE8gUlNBIERvbWFpbiBWYWxpZGF0aW9uIFNlY3VyZSBTZXJ2ZXIg
Q0EwHhcNMTgwMTI0MDAwMDAwWhcNMjEwMTIzMjM1OTU5WjBYMSEwHwYDVQQLExhE
b21haW4gQ29udHJvbCBWYWxpZGF0ZWQxHTAbBgNVBAsTFFBvc2l0aXZlU1NMIFdp
bGRjYXJkMRQwEgYDVQQDDAsqLmlwaWZ5Lm9yZzCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBAMCdCYpCtwGoVFah51Gy/ZrgPmk8s2JTBMZlnKGnLKP7r1SC
feRHgmcGYiAMtXwfFWnA3ZGQ6L9Gm00T1G0K/FpS2POxtDNfRWLiOFS80hW0UnXr
XCvnyHVK0+pXs/3CuOqj8iSnMPdZsly/dhIt0zaWM8zRB8789RIr+zRmlFuXzJQT
Yvul/SKVPZ8gW6HwblTAWL+xZ5KRof8yiokR2WZtl21NQ+Ox9/JnpNMPlCTgHeKR
XhAR1zEg2Hn1FGshS5ypAa0O+8QgsheKzYWdq4bcEJRcYMXQg0S6eBB8GLsj1tqO
KVYmk1S9lcbS4vjzBGh5rhER6IuMTqstChP8WncCAwEAAaOCAdMwggHPMB8GA1Ud
IwQYMBaAFJCvajqUWgvYkOoSVnPfQ7Q6KNrnMB0GA1UdDgQWBBQcEDPSTQdru56u
AXGhYiQn+yPvczAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAdBgNVHSUE
FjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwTwYDVR0gBEgwRjA6BgsrBgEEAbIxAQIC
BzArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8uY29tL0NQUzAI
BgZngQwBAgEwVAYDVR0fBE0wSzBJoEegRYZDaHR0cDovL2NybC5jb21vZG9jYS5j
b20vQ09NT0RPUlNBRG9tYWluVmFsaWRhdGlvblNlY3VyZVNlcnZlckNBLmNybDCB
hQYIKwYBBQUHAQEEeTB3ME8GCCsGAQUFBzAChkNodHRwOi8vY3J0LmNvbW9kb2Nh
LmNvbS9DT01PRE9SU0FEb21haW5WYWxpZGF0aW9uU2VjdXJlU2VydmVyQ0EuY3J0
MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5jb21vZG9jYS5jb20wIQYDVR0RBBow
GIILKi5pcGlmeS5vcmeCCWlwaWZ5Lm9yZzANBgkqhkiG9w0BAQsFAAOCAQEAjPOZ
Mqwt7PZErXI5LXmGM2VfScatgpfZncJYRBgbKjtrI2HBt7446pjHXqzpImxC8rGj
s3AS15vxIz11ERiHqCskEQbg/0AYKJ1TNr5KrL/K8RY9vtGL1WDQCxMqSZcocvYr
YIVam8YkTitO9+xD0K0Icyvgans60Z4nkWJG+ZqRZgNi6TDbfBHWeSiTOc2q4MxI
lFXfP8/nyE2Jz/SO2JPatL05Q3VVJBOHtdpm070tZpIWFYXo9fV7xzHHx4WLHdHm
/ajStwWJoAGPWFUdmG+xoBqzuhADJRGA9L8DjLleBb6mIM39wKvQqgBSTvZlSiUv
4HUHfqpgGH4HX+1Sag==
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.ipify.org
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5903 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: E46B7A3821D8EE745867787C2AE6E319EFCE2886B973C508EFECA8C1B005870D
    Session-ID-ctx: 
    Master-Key: FFE6F29827EF514A72B117FE0B326496F33B9E9B7F2513A16AFC713711F0F14FC6155DB7E3BA97A094A6977456050B94
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1590865338
    Timeout   : 300 (sec)
    Verify return code: 10 (certificate has expired)
---


Expected results:
    Verify return code: 0 (ok)


Additional info:
Ryan Sleevi (https://twitter.com/sleevi_/status/1266647545675210753) and Hynek Schlawack (https://twitter.com/hynek/status/1266713203372933121) made me aware of the issue. Ryan's thread on Twitter contains more information on the issue.

Workaround:
Blacklisting the certificate solves the issue for me on RHEL 7.9:

# trust dump --filter "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;type=cert" > /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
# update-ca-trust extract
# trust list | grep -C2 "AddTrust External"
p11-kit: overriding trust for anchor in blacklist: addtrust-external-root.p11-kit
pkcs11:id=%ad%bd%98%7a%34%b4%26%f7%fa%c4%26%54%ef%03%bd%e0%24%cb%54%1a;type=cert
    type: certificate
    label: AddTrust External Root
    trust: blacklisted
    category: authority

# openssl s_client -connect api.ipify.org:443 | grep "Verify return code"
    Verify return code: 0 (ok)

Comment 2 Christian Heimes 2020-05-30 19:16:30 UTC
Created attachment 1693711 [details]
SSL Labs scan report

Comment 3 Christian Heimes 2020-05-30 19:17:18 UTC
My workaround does not fix GnuTLS.

# gnutls-cli api.ipify.org
p11-kit: overriding trust for anchor in blacklist: addtrust-external-root.p11-kit
Processed 154 CA certificate(s).
Resolving 'api.ipify.org'...
Connecting to '184.73.165.106:443'...
- Certificate type: X.509
- Got a certificate list of 4 certificates.
- Certificate[0] info:
 - subject `OU=Domain Control Validated,OU=PositiveSSL Wildcard,CN=*.ipify.org', issuer `C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Domain Validation Secure Server CA', RSA key 2048 bits, signed using RSA-SHA256, activated `2018-01-24 00:00:00 UTC', expires `2021-01-23 23:59:59 UTC', SHA-1 fingerprint `a8ec3c8e035158e5a9c0b5fe8cd1b4eced4c09a9'
        Public Key ID:
                8e05c08fb342748ee63ac348448821bc628b8150
        Public key's random art:
                +--[ RSA 2048]----+
                |*oE..            |
                |=o. o.           |
                |+..+ o.          |
                |+++ + ..         |
                |==.  o  S        |
                |.oo .  +         |
                |+. .  . .        |
                |=.               |
                | o               |
                +-----------------+

- Certificate[1] info:
 - subject `C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Domain Validation Secure Server CA', issuer `C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Certification Authority', RSA key 2048 bits, signed using RSA-SHA384, activated `2014-02-12 00:00:00 UTC', expires `2029-02-11 23:59:59 UTC', SHA-1 fingerprint `339cdd57cfd5b141169b615ff31428782d1da639'
- Certificate[2] info:
 - subject `C=GB,ST=Greater Manchester,L=Salford,O=COMODO CA Limited,CN=COMODO RSA Certification Authority', issuer `C=SE,O=AddTrust AB,OU=AddTrust External TTP Network,CN=AddTrust External CA Root', RSA key 4096 bits, signed using RSA-SHA384, activated `2000-05-30 10:48:38 UTC', expires `2020-05-30 10:48:38 UTC', SHA-1 fingerprint `f5ad0bcc1ad56cd150725b1c866c30ad92ef21b0'
- Certificate[3] info:
 - subject `C=SE,O=AddTrust AB,OU=AddTrust External TTP Network,CN=AddTrust External CA Root', issuer `C=SE,O=AddTrust AB,OU=AddTrust External TTP Network,CN=AddTrust External CA Root', RSA key 2048 bits, signed using RSA-SHA1, activated `2000-05-30 10:48:38 UTC', expires `2020-05-30 10:48:38 UTC', SHA-1 fingerprint `02faf3e291435468607857694df5e45b68851868'
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate. 
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.

Comment 4 Christian Heimes 2020-05-31 08:35:00 UTC
Created attachment 1693778 [details]
/etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit

Comment 5 Michael Catanzaro 2020-05-31 16:08:06 UTC
(In reply to Christian Heimes from comment #3)
> My workaround does not fix GnuTLS.

Daiki has a fix for GnuTLS in https://gitlab.com/gnutls/gnutls/-/merge_requests/1271. Would have been really nice to have known about this in advance so the Sunday evening heroics would not have been necessary. :/ I guess a separate bug will be needed if we want to fix GnuTLS in RHEL 7.

Comment 6 Christian Heimes 2020-05-31 16:32:58 UTC
(In reply to Michael Catanzaro from comment #5)
> Daiki has a fix for GnuTLS in
> https://gitlab.com/gnutls/gnutls/-/merge_requests/1271. Would have been
> really nice to have known about this in advance so the Sunday evening
> heroics would not have been necessary. :/ I guess a separate bug will be
> needed if we want to fix GnuTLS in RHEL 7.

Daiki has submitted the fix to Fedora 32, https://bodhi.fedoraproject.org/updates/FEDORA-2020-6ec1d85ab1 . I have tested the new build in a podman container and are happy to confirm that his changeset addresses the issue for AddTrust.

We knew about the expiring cert well in advance (#1721955), but nobody was aware how the issue was going to impact OpenSSL 1.0.2 and GnuTLS. :(

Thanks for the heroics!

Comment 8 Erinn Looney-Triggs 2020-05-31 18:48:48 UTC
I'll pile onto this here and point out that it appears python 2.7 on RHEL 7 DOES NOT have issues with the new certificate, but python36 does:

Python 2.7.5 (default, Sep 26 2019, 13:23:47) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://example.com/')
>>>

Whereas:

Python 3.6.8 (default, Sep 26 2019, 11:57:09) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://example.com/')
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 394, in connect
    ssl_context=context,
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 773, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 1033, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 645, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 438, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 651, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 502, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='exmaple.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))

I know a separate report is needed, but there is good info here, and perhaps others will find this useful.

Comment 9 Christian Heimes 2020-05-31 18:58:45 UTC
There are no updates packages yet. Is any customer seeing any kind of specific issues? Or are customer merely worried that the presence of the expire AddTrust External Root cert is going to impact systems?

RHEL 8:
* OpenSSL and NSS based applications are not impacted if the service has an alternative trust chain.
* GnuTLS based applications are impacted and will require a new version of GnuTLS to work.

RHEL 7:
* OpenSSL based applications are affected. As a workaround the customer can blacklist the certificate and regenerate the cert bundles until Bob has created a new ca-certificate package.
* GnuTLS based applications are impacted and will require a new version of GnuTLS to work.

According to Ryan Sleevi Java is not affected.

To blacklist and rebuild the cert bundles the customer can run the commands as root:

---
# trust dump --filter "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;type=cert" > /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
# update-ca-trust extract
---

The "trust" command is part of p11-kit-trust. It's safe to create the file addtrust-external-root.p11-kit on one machine and copy it to other machines.


All systems:
I recommend to remove all cross-signed certificates rom all server configurations. For example there are are **two** variants of the "COMODO RSA Certification Authority". One is a self-signed root CA and one is an intermediate CA that is signed by "AddTrust External CA Root". The cross-signed certificate is:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            27:66:ee:56:eb:49:f3:8e:ab:d7:70:a2:fc:84:de:22
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
        Validity
            Not Before: May 30 10:48:38 2000 GMT
            Not After : May 30 10:48:38 2020 GMT
        Subject: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
        ...
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A
            X509v3 Subject Key Identifier: 
                BB:AF:7E:02:3D:FA:A6:F1:3C:84:8E:AD:EE:38:98:EC:D9:32:32:D4

Comment 10 Christian Heimes 2020-05-31 19:15:12 UTC
(In reply to Erinn Looney-Triggs from comment #8)
> I'll pile onto this here and point out that it appears python 2.7 on RHEL 7
> DOES NOT have issues with the new certificate, but python36 does:

I'm getting opposite results on RHEL 7.9 nightly:

$ python2
Python 2.7.5 (default, Mar 20 2020, 17:08:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> urllib2.urlopen
<function urlopen at 0x7f44a12c00c8>
>>> urllib2.urlopen("https://api.ipify.org/")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>

$ python3
Python 3.6.8 (default, Apr  3 2020, 12:24:13) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request                           
>>> urllib.request.urlopen("https://api.ipify.org/")
<http.client.HTTPResponse object at 0x7fc3fcf772b0>


Please open a new bug and CC me. I'm the upstream owner of Python's ssl module and curious about your findings.

Comment 11 Joe Wright 2020-05-31 19:46:18 UTC
I'm forwarding this to the customer to see if this gets them going in the meantime.

Comment 12 Paul Wouters 2020-05-31 20:55:16 UTC
I also get this on centos7 with python2

As quick fix, I ended up editing /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and commenting out this entry:

# AddTrust External Root
#-----BEGIN CERTIFICATE-----
#MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
#MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs
#IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290
#MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux
#FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h
#bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v
#dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt
#H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9
#uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX
#mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX
#a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN
#E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0
#WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD
#VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0
#Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU
#cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx
#IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN
#AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH
#YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5
#6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC
#Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX
#c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a
#mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ=
#-----END CERTIFICATE-----


Depending on the version / update status of centos7, this file had only this AddTrust CA, or a few others too. Disabling only this one is enough to fix the immediate problem.
Package updates soon will hopefully correct it properly.

Comment 13 Christian Heimes 2020-05-31 21:30:38 UTC
Created attachment 1693957 [details]
trustflag.py demo script

OpenSSL 1.1.0 has enabled X509_V_FLAG_TRUSTED_FIRST flag by default, https://www.openssl.org/docs/man1.1.0/man3/X509_VERIFY_PARAM_set_flags.html . The flag is available in OpenSSL 1.0.2 but not enabled by default. However Python has enabled the flag in its ssl module for more than five years and hadn't got any complain. The upstream patch https://github.com/python/cpython/commit/fdb19715879babc580f63bc129f5b0ff46482d1c never landed in RHEL 7.

I can confirm that setting the flags resolves the problem for OpenSSL on RHEL 7. The flag is not set in python2. Setting the flags gets rid of the cert validation error.

# python2 trustflag.py 
verify_flags 0x0L
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>
verify_flags 0x8000L
success
# python3 trustflag.py 
verify_flags 0x8000
success
verify_flags 0x8000
success

Setting the flight by default in OpenSSL is a one-line change and would make our copy of OpenSSL 1.0.2 more behave like OpenSSL 1.1.1 on RHEL 8.

Comment 14 12lovelysharma20@gmail.com 2020-06-01 07:53:20 UTC
When I am using "openssl s_client -connect domain_name" it shows expired certificate in chain i.e AddTrust External. I have commented this cert in ca-bundle.crt and my curl command is running for https://url after commenting the cert in ca-bundle.crt but as I said s_client shows this expired certficate in chain, do we need a fix for that, if yes? then what possible could be the fix. I am using AWS Linux instance.

Comment 15 Pierre Ossman 2020-06-01 12:29:21 UTC
We're seeing this disrupt various automated systems on our RHEL 6 and 7 machines here as we're using GoDaddy certificates on our public servers. I've seen RHEL 7 and 8 mentioned here, but will RHEL 6 get some love? It's out of most support phases, but not quite EOL yet.

I'm afraid the workaround doesn't work for various reasons:

a) "/usr/bin/trust" isn't packaged for RHEL 7

b) Generating the file on Fedora 31 and transferring it over gives:

> p11-kit: addtrust-external-root.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute
> p11-kit: addtrust-external-root.p11-kit: nss-server-distrust-after: invalid or unsupported attribute
> p11-kit: addtrust-external-root.p11-kit: nss-email-distrust-after: invalid or unsupported attribute

c) Removing those still gives:

> update-ca-trust: Warning: The dynamic CA configuration feature is in the disabled state
> p11-kit: overriding trust for anchor in blacklist: addtrust-external-root.p11-kit
> p11-kit: duplicate 'AddTrust External Root' certificate found in: ca-bundle.legacy.crt

And I'm still getting failures from openssl:

> # openssl s_client -connect XXX:443
> ...
>     Verify return code: 10 (certificate has expired)

Comment 18 Christian Heimes 2020-06-01 13:57:31 UTC
Created attachment 1694101 [details]
/etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit

Sorry, I didn't check older RHEL 7 versions for the presence of the 'trust' command.

Here is an updated file that works with older versions of update-ca-trust. You can copy the file instead of generating the blacklist file in each machine. You still have to run "update-ca-trust", though.

The client-side workaround with blacklist + update-ca-trust does not work for me on RHEL 6 or RHEL 7.3 with OpenSSL 1.0.1. It only works on RHEL 7.4 and newer with OpenSSL 1.0.2. It seems like OpenSSL 1.0.1 cannot establish a correct trust chain. It fails with "Verify return code: 19 (self signed certificate in certificate chain)".

The blacklist approach is a workaround any way. The correct fix for the problem is Martin Poole's approach. Customers have to update the certificate chain on their servers and remove all alternative trust chain and intermediate CAs that are connected to "AddTrust External Root". This will address the problem for all clients.

Comment 19 Pierre Ossman 2020-06-01 14:25:41 UTC
Ah, all right. I missed that detail. I can confirm that removing that intermediate certificate on the server fixed the issue on all RHEL 6 and RHEL 7 clients right away.

Comment 20 Michael Catanzaro 2020-06-01 14:30:44 UTC
(In reply to Christian Heimes from comment #18)
> The client-side workaround with blacklist + update-ca-trust does not work
> for me on RHEL 6 or RHEL 7.3 with OpenSSL 1.0.1. It only works on RHEL 7.4
> and newer with OpenSSL 1.0.2. It seems like OpenSSL 1.0.1 cannot establish a
> correct trust chain. It fails with "Verify return code: 19 (self signed
> certificate in certificate chain)".

Historical interest, but I think that was fixed in Fedora bug #1166614 with this patch: https://src.fedoraproject.org/rpms/openssl/blob/f21/f/openssl-1.0.1k-alt-chains.patch

Comment 21 12lovelysharma20@gmail.com 2020-06-01 14:32:10 UTC
Can you please reply over my comment?

Comment 22 Brandon Wise 2020-06-01 15:57:06 UTC
I have a number RHEL 7.8 hosts that I have been "fixing" by commenting or deleting the following out of /etc/pki/tls/certs/ca-bundle.crt
# AddTrust External CA Root
# USERTrust RSA Certification Authority

When I tried using the blacklist commands it results in a different error, but does not remedy the issue. 

# trust dump --filter "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;type=cert" > /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
# trust dump --filter "pkcs11:id=%53%79%bf%5a%aa%2b%4a%cf%54%80%e1%d8%9b%c0%9d%f2%b2%03%66%cb;type=cert" > /etc/pki/ca-trust/source/blacklist/USERTrust-RSA-inter.p11-kit
# update-ca-trust

After running these commands and testing the cert with: "openssl s_client -connect ldap.example.com", I get "Verify return code: 2 (unable to get issuer certificate)", rather than the expiration error. 

Not sure why the blacklist doesn't work but commenting or delete the CA entries from that file does.

Comment 23 Christian Heimes 2020-06-01 16:25:21 UTC
Why are you blocking the "USERTrust RSA Certification Authority" CA cert? There are two different " USERTrust RSA Certification Authority" certificates. One is a root CA certificate and the other is a cross-signed intermediate CA certificate.

You blocked just blocked the good root CA. That cert is a valid and not-expired root CA. 

$ trust dump --filter "pkcs11:id=%53%79%bf%5a%aa%2b%4a%cf%54%80%e1%d8%9b%c0%9d%f2%b2%03%66%cb;type=cert" | openssl x509 -text 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            01:fd:6d:30:fc:a3:ca:51:a8:1b:bc:64:0e:35:03:2d
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
        Validity
            Not Before: Feb  1 00:00:00 2010 GMT
            Not After : Jan 18 23:59:59 2038 GMT
        Subject: C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
        ..
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE


The problematic intermediate, cross-signed CA certificate looks like this. Please note that it has the same subject key id and subject, but a different issuer and an authority key id.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            13:ea:28:70:5b:f4:ec:ed:0c:36:63:09:80:61:43:36
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
        Validity
            Not Before: May 30 10:48:38 2000 GMT
            Not After : May 30 10:48:38 2020 GMT
        Subject: C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
        ...
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:AD:BD:98:7A:34:B4:26:F7:FA:C4:26:54:EF:03:BD:E0:24:CB:54:1A

            X509v3 Subject Key Identifier: 
                53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB

Comment 24 JMU Libraries 2020-06-01 18:10:49 UTC
(In reply to Brandon Wise from comment #22)
> I have a number RHEL 7.8 hosts that I have been "fixing" by commenting or
> deleting the following out of /etc/pki/tls/certs/ca-bundle.crt
> # AddTrust External CA Root
> # USERTrust RSA Certification Authority
> 
> When I tried using the blacklist commands it results in a different error,
> but does not remedy the issue. 
> 
> # trust dump --filter
> "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;
> type=cert" >
> /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
> # trust dump --filter
> "pkcs11:id=%53%79%bf%5a%aa%2b%4a%cf%54%80%e1%d8%9b%c0%9d%f2%b2%03%66%cb;
> type=cert" > /etc/pki/ca-trust/source/blacklist/USERTrust-RSA-inter.p11-kit
> # update-ca-trust
> 
> After running these commands and testing the cert with: "openssl s_client
> -connect ldap.example.com", I get "Verify return code: 2 (unable to get
> issuer certificate)", rather than the expiration error. 
> 
> Not sure why the blacklist doesn't work but commenting or delete the CA
> entries from that file does.

I wanted to make a quick note to expand on this a touch from our experience. Removing/commenting the certs out of /etc/pki/tls/certs/ca-bundle.crt must be done in conjunction with blacklisting the two certs, not in place of. If the certs are commented or removed and not blacklisted, they'll be added back in the next time update-ca-trust is run. That could make for some headaches down the road if it gets run as part of an install script.

Also thank you for posting this, it really relieved a lot of headache for us!

Comment 25 Christian Heimes 2020-06-01 18:47:58 UTC
(In reply to JMU Libraries from comment #24)
> I wanted to make a quick note to expand on this a touch from our experience.
> Removing/commenting the certs out of /etc/pki/tls/certs/ca-bundle.crt must
> be done in conjunction with blacklisting the two certs, not in place of. If
> the certs are commented or removed and not blacklisted, they'll be added
> back in the next time update-ca-trust is run. That could make for some
> headaches down the road if it gets run as part of an install script.

You should not edit the ca-bundle.crt manually. The file is auto-generated by update-ca-trust command. Also only block "AddTrust External CA Root". Do not block the "USERTrust RSA Certification Authority" root CA certificate with serial number 01:fd:6d:30:fc:a3:ca:51:a8:1b:bc:64:0e:35:03:2d. The certificate is required and must not be blacklisted.

Step 1)
Block the "AddTrust External CA Root" certificate. You can either copy my attchment addtrust-external-root.p11-kit to /etc/pki/ca-trust/source/blacklist/ or create the p11-kit file with the command

   trust dump --filter "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;type=cert" > /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit

Step 2)
Regenerate cert bundles with the command 

   update-ca-trust extract

The command updates /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem, /etc/pki/tls/certs/ca-bundle.crt, and several other bundles.

Comment 26 Agostino Sarubbo 2020-06-02 07:54:00 UTC
(In reply to Christian Heimes from comment #25)
>    trust dump --filter
> "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;
> type=cert" >
> /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
>
>    update-ca-trust extract

The above steps did not give the expected result for me.

Comment 27 Agostino Sarubbo 2020-06-02 08:29:14 UTC
Created attachment 1694350 [details]
sectigo download

The Red Hat Knowledge Base (Article) 5117881 says:

The replacement intermediate certificate can be found on the page How to Download & Install Sectigo Intermediate Certificates - RSA and is the download labelled USERTrust RSA Root xSigned using AAA CA [ Cross Signed ]

However, by typing it ctrl-f there are multiple choices. This can confuse users.

Comment 28 Christian Heimes 2020-06-02 10:06:51 UTC
(In reply to Agostino Sarubbo from comment #26)
> The above steps did not give the expected result for me.

I'm unable to assist you without detailed information about your system, package versions, commands, and error messages.

Comment 29 Agostino Sarubbo 2020-06-02 10:11:00 UTC
(In reply to Christian Heimes from comment #28)
> (In reply to Agostino Sarubbo from comment #26)
> > The above steps did not give the expected result for me.
> 
> I'm unable to assist you without detailed information about your system,
> package versions, commands, and error messages.

I'm running ansible from a Gentoo machine and the target is a Centos7 machine.
When trying to get an object from S3 with aws_s3 I get SSL verification failed

# rpm -qa | grep boto
python-boto3-1.4.6-5.el7.noarch

To reproduce the issue you should have an S3 account.

The problem is reproducible on each Centos7 machine I have.

Obviously everything worked before 30May

Comment 30 Christian Heimes 2020-06-02 11:52:17 UTC
> I'm running ansible from a Gentoo machine and the target is a Centos7
> machine.
> When trying to get an object from S3 with aws_s3 I get SSL verification
> failed
> 
> # rpm -qa | grep boto
> python-boto3-1.4.6-5.el7.noarch
> 
> To reproduce the issue you should have an S3 account.
> 
> The problem is reproducible on each Centos7 machine I have.
> 
> Obviously everything worked before 30May

Could you please open a CentOS bug at https://bugs.centos.org/ ? Include additional information like distro version and openssl version.

Comment 31 Trevor Hemsley 2020-06-02 15:23:41 UTC
CentOS doesn't fix bugs in upstream packages but will inherit the fix once Red Hat release the RHEL packages. Once RHEL packages are available and pushed to git.centos.rg by RH RCM, they will be picked up and rebuilt for CentOS. Until then you'll have to use one of the workarounds in this bz.

Comment 32 Martin Poole 2020-06-02 16:13:44 UTC
Statement

Our current recommendation which should ensure the quickest resolution is to delete any expired intermediate certificates in their application files.

There is no need to delete or blacklist any certificates in the the system provided bundle.

The basics of this are covered in the following support article.

    Sectigo Root and Intermediate Certificate Expiry - May 2020
    https://access.redhat.com/articles/5117881

Any specific problems should be raised as support tickets.

Comment 33 Veaceslav Mindru 2020-06-02 18:25:00 UTC
what if we can not change APP side easily. blacklisting addtrust results in following error 

Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA256
    Session-ID: 2964B5956EDAD44CFB441869A5A15EEC8CBA6A1517F0948BFDB14AC78EAFBBFD
    Session-ID-ctx: 
    Master-Key: C107E531A889CBE17C0619CF8DD3678B67C4FC8455D0B6219CD9AEDA0B1E827C6AA827B9C30209A623755C9652D52D12
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1591122262
    Timeout   : 300 (sec)
    Verify return code: 2 (unable to get issuer certificate)

Comment 34 Paul Raines 2020-06-03 15:14:31 UTC
(In reply to Martin Poole from comment #32)
> Statement
> 
> Our current recommendation which should ensure the quickest resolution is to
> delete any expired intermediate certificates in their application files.
> 

Simply commenting out SSLCertificateChainFile in the Apache config where it used the old intermediate certificate does not work for me.
All legacy clients such as curl on RHEL7 still fail.  I had to update SSLCertificateChainFile on the server with the new intermediate/chain certificate that expires in 2038 to get those legacy clients to work with the server.

Comment 35 Martin Poole 2020-06-03 15:24:26 UTC
My apologies, it would seem my update was not clear enough.


I meant that the expired certificates within the file specified by SSLCertificateChainFile should be removed.

For example, if the chain illustrated in the head of this bug https://bugzilla.redhat.com/show_bug.cgi?id=1842174#c0 was provided by httpd with mod_ssl the SSLCertificateChainFile would have contained two certificates

    Serial Number:
            2b:2e:6e:ea:d9:75:36:6c:14:8a:6e:db:a3:7c:8c:07
    Subject: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
        Validity
            Not Before: Feb 12 00:00:00 2014 GMT
            Not After : Feb 11 23:59:59 2029 GMT
    Issuer: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority

and

    Serial Number:
            27:66:ee:56:eb:49:f3:8e:ab:d7:70:a2:fc:84:de:22
    Subject: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
        Validity
            Not Before: May 30 10:48:38 2000 GMT
            Not After : May 30 10:48:38 2020 GMT
    Issuer: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root


The fix being to simply delete the second certificate.

Comment 36 Paul Raines 2020-06-03 15:38:25 UTC
I see.  Yes.  Taking the old chain cert file and removing the expired certs from it did indeed create a working chain file for me.

Comment 37 Agostino Sarubbo 2020-06-04 06:51:59 UTC
(In reply to Christian Heimes from comment #30)
> Could you please open a CentOS bug at https://bugs.centos.org/ ? Include
> additional information like distro version and openssl version.

Done, the response was:
####
CentOS is a rebuild of the sources used to create RHEL. We do not modify anything except to remove branding and logos.  If/when RH fixes the issue and incorporates it into RHEL and releases a patched version, then CentOS will pick it up and rebuild it. Until then  you'll have to use one of the workarounds in the bug reported that you have referenced
####

It looks like a rubber wall, but this is an old story....at least for me.

Comment 41 Simo Sorce 2020-06-09 18:45:41 UTC
The work is ready to go.

1,3: We have 30 customer cases waiting, this is a high profile bug that adversely affects many customers.
We want to show that we are responsive and release an 7.8.z package, but we need 7.9 acks to be able to commit the same code in dist-git

2: The risk is low, we have automated tests and well defined QA procedures to test ca-cert bundle changes.

Comment 42 Bob Relyea 2020-06-09 18:48:59 UTC
See comment 41.


Note You need to log in before you can comment on or make changes to this bug.