Bug 1244291
Summary: | python-glanceclient-0.17.0-2.el7.noarch.rpm packaged with buggy glanceclient/common/https.py | ||
---|---|---|---|
Product: | [Community] RDO | Reporter: | mariojmdavid |
Component: | python-glanceclient | Assignee: | hguemar |
Status: | CLOSED EOL | QA Contact: | nlevinki <nlevinki> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | Kilo | CC: | ayoung, eglynn, fpercoco, karlthered, mattieu.puel, piekarski.grzegorz, srevivo, yee379 |
Target Milestone: | --- | ||
Target Release: | Kilo | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | python-glanceclient-0.17.2-1.fc23 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2016-05-19 15:32:20 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
mariojmdavid
2015-07-17 16:46:15 UTC
> notice right away the missing import poolmanager in the beggining No, it's not missing, looks like some code was not updated accordingly. Urllib3's PoolManager has no attribute connection_kw but connection_pool_kw, so it was a legitimate error. Look at your diff, and you'll see that attribute was never used directly previously. I already submitted a fix for liberty and kilo branches. https://review.openstack.org/#/c/204521/ https://review.openstack.org/#/c/204523/ If you could apply that patch and comment these reviews, I would be grateful :) hi I will apply the patch and will give a review/comment Mario hi the patch is not enough to solve the problem After applying the patch 'substitute connection_kw by connection_pool_kw' 2015-07-22 15:21:17.234 6051 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/glanceclient/common/https.py", line 162, in _create_glance_httpsconnectionpool 2015-07-22 15:21:17.234 6051 TRACE nova.api.openstack pool = HTTPSConnectionPool(parsed.host, port, **kw) 2015-07-22 15:21:17.234 6051 TRACE nova.api.openstack AttributeError: 'ParseResult' object has no attribute 'host' I got more wrong named attrs, so I did the following, modified _create_glance_httpsconnectionpool # modified mariojmdavid self.host = parsed.netloc.split(':')[0] self.port = parsed.netloc.split(':')[1] or 443 self.scheme = parsed.scheme pool = HTTPSConnectionPool(self.host, self.port, **kw) with self.poolmanager.pools.lock: self.poolmanager.pools[(self.scheme, self.host, self.port)] = pool After I got this exception 2015-07-22 16:35:30.125 6053 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 237, in call 2015-07-22 16:35:30.125 6053 TRACE nova.api.openstack host=host, port=port, reason=six.text_type(e)) 2015-07-22 16:35:30.125 6053 TRACE nova.api.openstack GlanceConnectionFailed: Connection to glance host nimbus.ncg.ingrid.pt:9292 failed: Error finding address for glance+https://nimbus.ncg.ingrid.pt:9292/v1/images/CentOS-7-x86_64-GenericCloud: ('Connection aborted.', error('getaddrinfo() argument 2 must be integer or string',)) 2015-07-22 16:35:30.125 6053 TRACE nova.api.openstack The connection is done with the scheme = 'glance+https' and it should be with https I don't have knowledge about how this scheme would work on an actual connection I saw in another bug report that this scheme was added in commit 695dbc910d2068aff89a68d6f5aa088cc835766e Still I modify https.py and http.py and taken out glance+ from the scheme def get_connection(self, url, proxies=None): try: return super(HTTPSAdapter, self).get_connection(url, proxies) except KeyError: # NOTE(sigamvirus24): This works around modifying a module global # which fixes bug #1396550 # The scheme is most likely glance+https but check anyway # modified by mariojmdavid just https if not url.startswith('https://'): class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool): scheme = 'https' #mariojmdavid In http.py class HTTPClient(object): ... if not compression: self.session.mount("https://", https.HTTPSAdapter()) 2015-07-22 17:44:42.378 16711 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 641, in get_adapter 2015-07-22 17:44:42.378 16711 TRACE nova.api.openstack raise InvalidSchema("No connection adapters were found for '%s'" % url) 2015-07-22 17:44:42.378 16711 TRACE nova.api.openstack InvalidSchema: No connection adapters were found for 'glance+https://nimbus.ncg.ingrid.pt:9292/v1/images/CentOS-7-x86_64-GenericCloud' finally removed from http.py the line self.endpoint = 'glance+' + self.endpoint and got an 2015-07-22 18:25:47.698 20691 TRACE nova.image.glance File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 218, in _request 2015-07-22 18:25:47.698 20691 TRACE nova.image.glance raise exc.CommunicationError(message=message) 2015-07-22 18:25:47.698 20691 TRACE nova.image.glance CommunicationError: Error finding address for https://nimbus.ncg.ingrid.pt:9292/v1/images/CentOS-7-x86_64-GenericCloud: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed 2015-07-22 18:25:47.698 20691 TRACE nova.image.glance 2015-07-22 18:25:47.699 20691 ERROR nova.api.openstack [req-9dbaf88f-22ac-45b0-ba2a-8aafc219907c 2ed32b53df014cb6a9b444e4f38257a0 21f61c3b2ce04ff3aae52bc5428d0d22 - - -] Caught error: Connection to glance host nimbus.ncg.ingrid.pt:9292 failed: Error finding address for https://nimbus.ncg.ingrid.pt:9292/v1/images/CentOS-7-x86_64-GenericCloud: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -------------------------------------- Reaching here, I would have to ask first about ssl_compression or no_ssl_compression I don't know where it is set, if its some conf file, and ssl_compression related to which service I have glance over swift, so the images are in swift the images are qcow2, so I understand that there is no need for ssl_compression in Although I have at the moment in glance-api.conf swift_store_ssl_compression = True is this the what you are referring class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool): """ HTTPSConnectionPool will be instantiated when a new connection is requested to the HTTPSAdapter.This implementation overwrites the _new_conn method and returns an instances of glanceclient's VerifiedHTTPSConnection which handles no compression. ssl_compression is hard-coded to False because this will be used just when the user sets --no-ssl-compression. """ ---------------------------------------- now a final thing about my setup I have a virtual IP, 2 haproxys, 3 controller nodes expose the dashboard and all OS apis under that single VIP that is nimbus.ncg.ingrid.pt all goes to https, and I have a x.509 certificate for nimbus.ncg.ingrid.pt at the haproxies all https to nimbus transforms into http for the underlying control nodes tell me if you need more info best Mario error/mistake on my part I have reverted 'https' again to 'glance+https' AND had a mistake in https.py self.host = parsed.netloc.split(':')[0] should be in fact self.host = parsed.netloc.rsplit(':', 1)[0] After this change, I now get to the next exception which is the latest reported in my previous comment: ERROR nova.api.openstack [req-46d37997-6cc6-4449-b2bb-438972d0c624 2ed32b53df014cb6a9b444e4f38257a0 21f61c3b2ce04ff3aae52bc5428d0d22 - - -] Caught error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')] the ssl cert error has been solved by including our "private" ca in the trusted bundle I can confirm what I wrote in the review comment to the patches, the following patch is needed /root/https.py is the patched one # diff https.py /root/https.py 157c157 < kw = self.poolmanager.connection_kw --- > kw = self.poolmanager.connection_pool_kw 162c162,163 < pool = HTTPSConnectionPool(parsed.host, port, **kw) --- > self.host = parsed.netloc.rsplit(':', 1)[0] > pool = HTTPSConnectionPool(self.host, port, **kw) 165c166 < self.poolmanager.pools[(parsed.scheme, parsed.host, port)] = pool --- > self.poolmanager.pools[(parsed.scheme, self.host, port)] = pool Thanks for your feedback, it's appreciated :) I think this is solved alredy in python-glanceclient 0.17.2. I tested this by simply replace https.py from python-glanceclient 0.17.2. Hello, I have the same issue in a very similar context (RDO kilo on CentOS with reverse https proxy). I tried python-glanceclient-0.17.2-1.fc23 which did not solve the problem. Upgrading to cbs.centos.org python-glanceclient-1.1.0-1.el7 did. Is there a pending official RDO release to fix this ? All the best. Issue being clients are consummed by services which relies on very specific versions. https://github.com/openstack/requirements/blob/stable/kilo/global-requirements.txt#L125 As F23 ships Kilo, we can't upgrade this, workaround would be using rawhide version of python-glanceclient. So we'll try to backport the fix. Note that we're removing services from Fedora and so we'll be able to upgrade clients more aggressively starting F24. Hi again, obviously, using 1.1.0 leads to other issues while addressing the former. I followed your advise to give a try with rawhide 0.17.2 but I still have the problem. Exemple for a nova boot command : nova-controller[19838]: nova.api.openstack Caught error: Expecting value: line 1 column 1 (char 0) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack Traceback (most recent call last): 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__ 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return req.get_response(self.application) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack application, catch_exc_info=False) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return resp(environ, start_response) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__ 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return self._call_app(env, start_response) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return self._app(env, _fake_start_response) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return resp(environ, start_response) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return resp(environ, start_response) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__ 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack response = self.app(environ, start_response) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return resp(environ, start_response) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return self.func(req, *args, **kwargs) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in __call__ 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack content_type, body, accept) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in _process_stack 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in dispatch 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return method(req=request, **action_args) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/images.py", line 125, in index 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack **page_params) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 68, in get_all 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return session.detail(context, **kwargs) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 274, in detail 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack for image in images: 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 249, in list 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack for image in paginate(params, return_request_id): 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 233, in paginate 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack images, resp = self._list(url, "images") 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 63, in _list 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack resp, body = self.client.get(url) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 262, in get 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return self._request('GET', url, **kwargs) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 249, in _request 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack body_iter = resp.json() 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/requests/models.py", line 819, in json 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return json.loads(self.text, **kwargs) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 488, in loads 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return _default_decoder.decode(s) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 370, in decode 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack obj, end = self.raw_decode(s) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack File "/usr/lib64/python2.7/site-packages/simplejson/decoder.py", line 389, in raw_decode 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack return self.scan_once(s, idx=_w(s, idx).end()) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2015-10-21 13:12:55.692 19838 TRACE nova.api.openstack Am I missing something ? Can I safely bypass the kilo requirement for glanceclient<0.18 and which version should I use ? Do you know which specific bugfixes and patchsets for 0.17.2 circumvent this particular issue ? Best regards, hi, i notice that the python glance client from http://rdo.fedorapeople.org/openstack-kilo/rdo-release-kilo.rpm is still version 0.17.0.2.el7; which still has this bug. mariojmdavid's patch fixes the bug in my testing. When will it be merged? This bug is against a Version which has reached End of Life. If it's still present in supported release (http://releases.openstack.org), please update Version and reopen. |