Just to try outline the underlying behaviour here.

import requests
> This imports requests module, which inside has:
from requests.packages.urllib3.contrib import pyopenssl
> This triggers the stub code in packages/__init__.py to run for requests.packages.urllib3. This stub code fails import and so then imports urllib3 from the system:
__import__(real_name)
> That enters sys.modules as 'urllib3', but inside the system urllib3 it then imports connectionpool:
from .connectionpool import (
> This enters sys.modules as 'urllib3.connectionpool' and then inside there we import connection:
from .connection import (
> This enters sys.modules as 'urllib3.connection'. Lots other bits happen now which aren't relevant. Then at end of it we copy sys.modules['urllib3'] to sys.modules['requests.packages.urllib3']. Now the first stage done, requests.packages.urllib3.contrib needs importing and so it runs the stub code again. This time import succeeds because the parent is already loaded:
__import__(name)
> Then we get to requests.packages.urllib3.contrib.pyopenssl and import that, and inside that we have:
from .. import connection
> However, this then resolves to 'requests.packages.urllib3.connection'. We don't have this loaded so stub code runs again. We DO have 'urllib3.connection' loaded though... So we actually load it AGAIN, as a new module. Now we do the injection:
pyopenssl.inject_into_urllib3()
> Problem here is that injection happens into 'requests.packages.urllib3.connection' entry in sys.modules. When actually, when you look at the 'import requests' results above... everything inside the requests module is using sys.modules entries for 'urllib3.X'. So the patch completely misses.

Solution is to ensure that after a system module is imported, we scour EVERY submodule imported by it, and ensure we create a corresponding 'requests.packages.urllib3.X' entry in sys.modules that is a direct copy. This ensures there is only one urllib3.connection module loaded instead of two, and ensures the patch hits the same one the requests module is actually using.

SNI now works. Hopefully clears up any unknowns!

Hello and thank for this report as well as the reproducers. As I see from the discussions on github, upstream decided not to apply the patch as they will target python 2.7, so it seems we'll have to divert a bit from upstream in order for the component to work correctly on RHEL 6.8.

Will wait a bit for a final verdict from upstream, as it would be ideal if they merge your patch and then we carry the same patch downstream.

Hi Charalampos Stratakis,

Apologies, that wasn't anything to do with upstream requests module. I'll remove it from the ticket. That was a downstream product (SaltStack) that is relying on python-requests, that is affected by this issue on RedHat 6 (though the issue itself was about Amazon Linux it was the same issue on RedHat). The patch I mentioned there was a workaround for the issue for the time being, so of course they would be unwilling to include it in source, as the root issue resides within the python-requests packaging on RedHat 6. Their statement regarding moving to Python 2.7 only applies to Amazon Linux as of course RedHat 6 does not ship Python 2.7 (that I know) so would not work.

If the python-requests module is fixed, SaltStack works on RedHat 6 without any modification, and it would only need requests_lib to be configured and ndg_httpsclient installed.

This is an issue that needs to be rectified within the python-requests module, specifically the stub code.

Regards,

Jason

Just to add also. This impacts all python applications on RedHat 6 that attempt to use the requests module with a server using SNI, as the reproduction script demonstrates. So it is not isolated to SaltStack.

It seems the patch doesn't work though. Trying the reproducer with a patched requests provides the same traceback.

Did you install python2-ndg_httpsclient? It is required by urllib3 (and therefore also requests) to support SNI. I probably didn't make that clear in the steps, sorry. So even with the patch if you don't have it installed it won't work. Only with it installed AND the patch will the desired result be achieved.

When I get a straining next few days though I will test thoroughly again on a new machine as I can't rule out I messed up somewhere :-) Unless you manage to get the desired result before I test again!

s/straining/moment in/ ...

You were right, by installing python-ndg_httpsclient and also installing python-requests with your patch I was able to verify that your patch indeed works.

The bug and your patch will be evaluated for inclusion at the next release.

Thank you very much for your contribution here.

Also it seems that this is the issue upstream [0]

[0] https://github.com/kennethreitz/requests/issues/3287

Verified:

Old Failed:
python-requests-2.6.0-3.el6.noarch

Output:

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: Test
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [  BEGIN   ] :: Running reproducer :: actually running 'python reproducer.py 2>&1 | tee output.log'
Traceback (most recent call last):
  File "reproducer.py", line 5, in <module>
    r = requests.head('https://downloads.atlassian.com')
  File "/usr/lib/python2.6/site-packages/requests/api.py", line 94, in head
    return request('head', url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 464, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:492: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
:: [   FAIL   ] :: Running reproducer (Expected 0, got 1)
:: [   FAIL   ] :: File 'output.log' should contain 'Status:  302' 
:: [   FAIL   ] :: File 'output.log' should not contain 'Traceback (most recent call last):' 
:: [   FAIL   ] :: File 'output.log' should not contain 'error:14077410:SSL' 
'76a35ff0-e028-43b3-8afa-587fcce79bed'
Test result: FAIL
   metric: 4
   Log: /var/tmp/beakerlib-49057101/journal.txt
    Info: Searching AVC errors produced since 1481640131.54 (Tue Dec 13 09:42:11 2016)
     Searching logs...
     Info: No AVC messages found.
 Writing to /mnt/testarea/tmp.YzAn9V
:
   AvcLog: /mnt/testarea/tmp.YzAn9V

:::::

NEW pass:
Version:    python-requests-2.6.0-4.el6.noarch

Output:
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: Test
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [  BEGIN   ] :: Running reproducer :: actually running 'python reproducer.py 2>&1 | tee output.log'
Status:  302
:: [   PASS   ] :: Running reproducer (Expected 0, got 0)
:: [   PASS   ] :: File 'output.log' should contain 'Status:  302' 
:: [   PASS   ] :: File 'output.log' should not contain 'Traceback (most recent call last):' 
:: [   PASS   ] :: File 'output.log' should not contain 'error:14077410:SSL' 
'c9d2a0a3-bf12-4ba0-878e-44cea3221895'
Test result: PASS
   metric: 0
   Log: /var/tmp/beakerlib-49057103/journal.txt
    Info: Searching AVC errors produced since 1481640221.66 (Tue Dec 13 09:43:41 2016)
     Searching logs...
     Info: No AVC messages found.
 Writing to /mnt/testarea/tmp.IYp4y8
:
   AvcLog: /mnt/testarea/tmp.IYp4y8

:::::::::::::::::::

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0713.html