Description of problem: When running our 22 LC on RHEL7.2 (updated) that connects to our beaker server over SSL, following error appears when importing a distro: 2016-10-13 11:39:41,314 bkr.common.helpers ERROR Login Fail Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/bkr/common/helpers.py", line 81, in run self.function(*self.args, **self.kwargs) File "/usr/lib/python2.7/site-packages/bkr/common/hub.py", line 98, in _login if force or self._hub.auth.renew_session(): File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/python2.7/site-packages/bkr/common/xmlrpc.py", line 563, in request result = transport_class.request(self, *args, **kwargs) File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib/python2.7/site-packages/bkr/common/xmlrpc.py", line 437, in _single_request self.send_content(h, request_body) File "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in send_content connection.endheaders(request_body) File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 826, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 1236, in connect server_hostname=sni_hostname) File "/usr/lib64/python2.7/ssl.py", line 350, in wrap_socket _context=self) TypeError: __init__() got an unexpected keyword argument 'server_hostname' Upgrading to new python-gevent helped, but I'm not sure if it does not break anything else, so I'm logging this bug. FYI, we do not have new version in RHEL-extras (7.2). Version-Release number of selected component (if applicable): happens on our frozen 22 release How reproducible: always Steps to Reproduce: 1. Install RHEL7.2, fully updated, install beaker 22 LC connecting to server via SSL (https://...) 2. beaker-import <any_valid_url> 3. Actual results: Import error (got an unexpected keyword argument 'server_hostname') Expected results: No error Additional info: It seems that updating python-gevent helps. Using the standard install procedure this package is imported from RHEL-extras. I've tried using following workaround, which allowed me to import the distro: # pip install --user --upgrade gevent # export PYTHONPATH=/root/.local/lib/python2.7/site-packages/ # beaker-proxy -f # beaker-import <distro>
Which version of python-gevent was broken? Which version is working?
python-gevent-1.0-2.el7.x86_64 (RHEL-extras) is broken the one pip installed was IIRC 1.1.2.
Okay thanks. I have a feeling that we are passing this kwarg down conditionally (based on the Python version) expecting that the 2.7 stdlib supports it, but the gevent version is slightly older and so its monkey-patched ssl lib is not aware of that new arg. But I need to investigate more.
I would agree with your analysis. I'd blame the "embedded" ssl library too, at least from what I've been able to try and google :).
It would be nice if our dogfood job would set up SSL certificates (under a fake self-signed CA or something) so that the tests used https:// for everything. I assume it would reproduce this bug in that case... Anyway, it seems that EPEL7 still has gevent 1.0.1. I'm not sure if that's intentional by Orion because 1.1 has some slight incompatibilities: http://www.gevent.org/whatsnew_1_1.html#compatibility or if just because nobody ever bothered updating it... So I could try building a new gevent for EPEL7 and letting that soak in our dogfood tests for a while hmmm.
Ahh nope I just remembered that gevent was retired from EPEL7 because it's in RHEL7 Extras now. With version 1.0 only. So this probably needs to be filed as a RHEL bug, assuming I can come up with a simple reproducer against gevent.
I've found RHEL Bug 1416165 which shows a different problem, but would have the same outcome. The likelihood that it gets fixed is very dim, since RHEL 7.4 is already shipped. However in a quick chat with Dan there are a few possible solutions we can take: (I primarily quote Dan here) * file a better RHEL7 bug demonstrating exactly why it is broken right now and needs fixing, perhaps with a simple patch that does not mean rebasing gevent entirely (they will much more likely take that in z stream) * ship our own version of gevent that just overwrites the rhel one (in general, bad) * ship a side-by-side installable compat package containing a newer gevent (less bad but way harder) * maybe figure out what it would cost us to replace gevent with Pythons 'async' which is part of the standard library in Python3 So the way forward would be finding out which solution we would like to go, pick one and go with it.
After further analysis, it seems that this issue should be applicable only for Python >= 2.7.9. Accordingly, to the documentation, they changed the constructor of ssl.SSLSocket old definition: __init__(self, sock, keyfile=None, certfile=None, server_side=False, cert_reqs=0, ssl_version=2, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True, ciphers=None) new definition: __init__(self, sock, keyfile=None, certfile=None, server_side=False, cert_reqs=0, ssl_version=2, ca_certs=None, do_handshake_on_connect=True, suppress_ragged_eofs=True, ciphers=None, server_hostname=None, _context=None) However monkey patch for SSL provided by python-gevent doesn't have this change until 1.0.2 release. I'm testing this issue on RHEL 7.6. By default, RHEL provides me Python 2.7.5 - This version will be used also in RHEL 7.7 (currently in development).