Bug 1842314

Summary: AddTrust certificate expiration causes Python to fail on connections
Product: Red Hat Enterprise Linux 7 Reporter: Erinn Looney-Triggs <erinn.looneytriggs>
Component: python3Assignee: Python Maintainers <python-maint>
Status: CLOSED INSUFFICIENT_DATA QA Contact: RHEL CS Apps Subsystem QE <rhel-cs-apps-subsystem-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.8CC: cheimes, cstratak, pviktori, redhat
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-15 12:04:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
trustflag.py demo script none

Description Erinn Looney-Triggs 2020-05-31 19:09:26 UTC
Description of problem:
It appears that python36 https validation is broken with the expiration of the addtrust root, whereas python2 is not broken. Substituted example.com but just assume it is a site signed by sectigo. I am using requests here, but the issue appears to lay in or below the 'ssl' module.


Python 2.7.5 (default, Sep 26 2019, 13:23:47) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://example.com/')
>>>

Whereas:

Python 3.6.8 (default, Sep 26 2019, 11:57:09) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://example.com/')
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 394, in connect
    ssl_context=context,
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 773, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 1033, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 645, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 438, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 651, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 502, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='exmaple.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))

Version-Release number of selected component (if applicable):
python3-3.6.8-13.el7.x86_64
python-2.7.5-88.el7.x86_64


How reproducible:

import socket, ssl

HOST, PORT = 'some.sectigo.signed.site', 443

def handle(conn):
    conn.write(b'GET / HTTP/1.1\n')
    print(conn.recv().decode())

def main():
    sock = socket.socket(socket.AF_INET)
    context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
    context.options |= ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1  # optional
    conn = context.wrap_socket(sock, server_hostname=HOST)
    try:
        conn.connect((HOST, PORT))
        handle(conn)
    finally:
        conn.close()

if __name__ == '__main__':
    main()


Steps to Reproduce:
1. Run the above code using python3 and python2


Actual results:

It fails under python3 with certificate validation errors

Expected results:

It will work under both or fail under both

Additional info:

Comment 3 Christian Heimes 2020-06-03 12:25:05 UTC
Could you please provide package versions of openssl, python-requests, and RHEL 7 version?

Comment 4 Christian Heimes 2020-06-03 13:01:58 UTC
Created attachment 1694827 [details]
trustflag.py demo script

I cannot reproduce the problem on RHEL 7.8 with packages

python-2.7.5-88.el7.x86_64
openssl-1.0.2k-19.el7.x86_64
python3-3.6.8-13.el7.x86_64
ca-certificates-2019.2.32-76.el7_7.noarch

With standard ssl module and builtin urllib / urllib2 Python 2 is failing and Python 3 is passing just fine. The test script also shows that Python 3 has X509_V_FLAG_TRUSTED_FIRST correctly set and Python 2 is missing patch https://github.com/python/cpython/commit/b1ebba5bd569ede9b6f9573d6618fb3a6abddae5 from upstream.

The attached test script prints:

# python3 trustflag.py
3.6.8 (default, Sep 26 2019, 11:57:09) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
OpenSSL 1.0.2k-fips  26 Jan 2017

Try with default verify flags
verify_flags 0x8000
success

Try again with X509_V_FLAG_TRUSTED_FIRST
verify_flags 0x8000
success

# python2 trustflag.py
2.7.5 (default, Sep 26 2019, 13:23:47) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
OpenSSL 1.0.2k-fips  26 Jan 2017

Try with default verify flags
verify_flags 0x0L
FAILED
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>

Try again with X509_V_FLAG_TRUSTED_FIRST
verify_flags 0x8000L
success

Comment 5 Christian Heimes 2020-06-03 13:12:58 UTC
requests on Python 2 is failing for me and there is no requests for Python 3 on RHEL 7. How did you install requests? If you used a virtual environment, then you may have pulled on other dependencies from PyPI like PyOpenSSL and certifi.

# yum install python3-requests
No package python3-requests available.

# rpm -qa python-requests
python-requests-2.6.0-8.el7_7.noarch

# python2
Python 2.7.5 (default, Sep 26 2019, 13:23:47) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("https://addtrust-chain.demo.sslmate.com")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 68, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

Comment 6 Petr Viktorin (pviktori) 2020-06-18 14:34:35 UTC
How did you install requests?

Comment 7 Petr Viktorin (pviktori) 2020-07-15 12:04:19 UTC
Closing, feel free to reopen if you can answer.