Bug 1313254
Summary: | no retry when using 'nova image-list' on next configured glance api_servers endpoint | |||
---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Martin Schuppert <mschuppe> | |
Component: | openstack-nova | Assignee: | Diana Clarke <dclarke> | |
Status: | CLOSED ERRATA | QA Contact: | Prasanth Anbalagan <panbalag> | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 6.0 (Juno) | CC: | berrange, dasmith, dclarke, eglynn, jruzicka, kchamart, lyarwood, sbauza, sferdjao, sgordon, sreber, srevivo, vromanso | |
Target Milestone: | --- | Keywords: | ZStream | |
Target Release: | 7.0 (Kilo) | |||
Hardware: | All | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | openstack-nova-2015.1.3-11.el7ost | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1330628 (view as bug list) | Environment: | ||
Last Closed: | 2016-04-26 16:54:16 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: |
Description
Martin Schuppert
2016-03-01 09:12:24 UTC
/usr/lib/python2.7/site-packages/nova/image/glance.py 283 def detail(self, context, **kwargs): 284 """Calls out to Glance for a list of detailed image information.""" 285 params = _extract_query_params(kwargs) 286 try: ** actual client call --> 287 images = self._client.call(context, 1, 'list', **params) 288 except Exception: 289 _reraise_translated_exception() 290 291 _images = [] ** here we fail from the trace --> 292 for image in images: 293 if _is_image_available(context, image): 294 _images.append(_translate_from_glance(image)) 295 296 return _images We should already fail in call: 222 def call(self, context, version, method, *args, **kwargs): 223 """Call a glance client method. If we get a connection error, 224 retry the request according to CONF.glance.num_retries. 225 """ exceptions which lead to a retry --> 226 retry_excs = (glanceclient.exc.ServiceUnavailable, 227 glanceclient.exc.InvalidEndpoint, 228 glanceclient.exc.CommunicationError) 229 num_attempts = 1 + CONF.glance.num_retries 230 231 LOG.debug("maxretry: %r", num_attempts) 232 for attempt in xrange(1, num_attempts + 1): 233 LOG.debug("retry: %r", attempt) 234 client = self.client or self._create_onetime_client(context, 235 version) 236 try: 237 return getattr(client.images, method)(*args, **kwargs) 238 except retry_excs as e: 239 host = self.host 240 port = self.port 241 242 if attempt < num_attempts: 243 extra = "retrying" 244 else: 245 extra = 'done trying' 246 247 error_msg = (_("Error contacting glance server " 248 "'%(host)s:%(port)s' for '%(method)s', " 249 "%(extra)s.") % 250 {'host': host, 'port': port, 251 'method': method, 'extra': extra}) 252 LOG.exception(error_msg) 253 if attempt == num_attempts: 254 raise exception.GlanceConnectionFailed( 255 host=host, port=port, reason=six.text_type(e)) 256 time.sleep(1) This is indeed a bug, and it looks like it's been there for years. I'm surprised that I can't find an upstream bug report for it – I'll add one tomorrow. Quick summary in the mean time: In the case of 'nova image-list', glanceclient returns a python generator rather than an actual list of images. https://github.com/openstack/python-glanceclient/blob/d59e341a4cd99a8488d5cf41052d9b218379ac87/glanceclient/v1/images.py#L268 Because a generator is returned, an exception will never be raised here, so the retry mechanism is never executed. https://github.com/openstack/nova/blob/83261f3106a8bdde38d258a74da777add4956290/nova/image/glance.py#L249 Update: A fix for this bug has landed in upstream Nova master (Newton/OSP10). https://review.openstack.org/#/c/293127/ I have also proposed upstream backport patches for Nova Liberty/OSP8 & Mitaka/OSP9. https://review.openstack.org/#/c/295324/ https://review.openstack.org/#/c/295319/ This will likely not be backported to OSP6 and OSP7. Workarounds to consider instead: 1) glance image-list: http://docs.openstack.org/user-guide/common/cli_manage_images.html http://docs.openstack.org/openstack-ops/content/user_facing_images.html http://docs.openstack.org/cli-reference/glance.html 2) glance public_endpoint: http://docs.openstack.org/developer/glance/configuring.html#configuring-glance-public-endpoint Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0694.html |