Bug 1313254
| Summary: | no retry when using 'nova image-list' on next configured glance api_servers endpoint | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Martin Schuppert <mschuppe> | |
| Component: | openstack-nova | Assignee: | Diana Clarke <dclarke> | |
| Status: | CLOSED ERRATA | QA Contact: | Prasanth Anbalagan <panbalag> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 6.0 (Juno) | CC: | berrange, dasmith, dclarke, eglynn, jruzicka, kchamart, lyarwood, sbauza, sferdjao, sgordon, sreber, srevivo, vromanso | |
| Target Milestone: | --- | Keywords: | ZStream | |
| Target Release: | 7.0 (Kilo) | |||
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-nova-2015.1.3-11.el7ost | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1330628 (view as bug list) | Environment: | ||
| Last Closed: | 2016-04-26 16:54:16 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
|
Description
Martin Schuppert
2016-03-01 09:12:24 UTC
/usr/lib/python2.7/site-packages/nova/image/glance.py
283 def detail(self, context, **kwargs):
284 """Calls out to Glance for a list of detailed image information."""
285 params = _extract_query_params(kwargs)
286 try:
** actual client call -->
287 images = self._client.call(context, 1, 'list', **params)
288 except Exception:
289 _reraise_translated_exception()
290
291 _images = []
** here we fail from the trace -->
292 for image in images:
293 if _is_image_available(context, image):
294 _images.append(_translate_from_glance(image))
295
296 return _images
We should already fail in call:
222 def call(self, context, version, method, *args, **kwargs):
223 """Call a glance client method. If we get a connection error,
224 retry the request according to CONF.glance.num_retries.
225 """
exceptions which lead to a retry -->
226 retry_excs = (glanceclient.exc.ServiceUnavailable,
227 glanceclient.exc.InvalidEndpoint,
228 glanceclient.exc.CommunicationError)
229 num_attempts = 1 + CONF.glance.num_retries
230
231 LOG.debug("maxretry: %r", num_attempts)
232 for attempt in xrange(1, num_attempts + 1):
233 LOG.debug("retry: %r", attempt)
234 client = self.client or self._create_onetime_client(context,
235 version)
236 try:
237 return getattr(client.images, method)(*args, **kwargs)
238 except retry_excs as e:
239 host = self.host
240 port = self.port
241
242 if attempt < num_attempts:
243 extra = "retrying"
244 else:
245 extra = 'done trying'
246
247 error_msg = (_("Error contacting glance server "
248 "'%(host)s:%(port)s' for '%(method)s', "
249 "%(extra)s.") %
250 {'host': host, 'port': port,
251 'method': method, 'extra': extra})
252 LOG.exception(error_msg)
253 if attempt == num_attempts:
254 raise exception.GlanceConnectionFailed(
255 host=host, port=port, reason=six.text_type(e))
256 time.sleep(1)
This is indeed a bug, and it looks like it's been there for years. I'm surprised that I can't find an upstream bug report for it – I'll add one tomorrow.
Quick summary in the mean time:
In the case of 'nova image-list', glanceclient returns a python generator rather than an actual list of images.
https://github.com/openstack/python-glanceclient/blob/d59e341a4cd99a8488d5cf41052d9b218379ac87/glanceclient/v1/images.py#L268
Because a generator is returned, an exception will never be raised here, so the retry mechanism is never executed.
https://github.com/openstack/nova/blob/83261f3106a8bdde38d258a74da777add4956290/nova/image/glance.py#L249
Update:
A fix for this bug has landed in upstream Nova master (Newton/OSP10).
https://review.openstack.org/#/c/293127/
I have also proposed upstream backport patches for Nova Liberty/OSP8 & Mitaka/OSP9.
https://review.openstack.org/#/c/295324/
https://review.openstack.org/#/c/295319/
This will likely not be backported to OSP6 and OSP7. Workarounds to consider instead:
1) glance image-list:
http://docs.openstack.org/user-guide/common/cli_manage_images.html
http://docs.openstack.org/openstack-ops/content/user_facing_images.html
http://docs.openstack.org/cli-reference/glance.html
2) glance public_endpoint:
http://docs.openstack.org/developer/glance/configuring.html#configuring-glance-public-endpoint
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0694.html |