1382737 – Nova snapshot fails in ceph direct image setup

Bug 1382737 - Nova snapshot fails in ceph direct image setup

Summary: Nova snapshot fails in ceph direct image setup

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-heat-templates
Sub Component:
Version:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	10.0 (Newton)
Assignee:	Erno Kuvaja
QA Contact:	Attila Fazekas
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1388119 (view as bug list)
Depends On:
Blocks:	1500265
TreeView+	depends on / blocked

Reported:	2016-10-07 14:30 UTC by Attila Fazekas
Modified:	2023-02-22 23:02 UTC (History)
CC List:	10 users (show)
Fixed In Version:	openstack-tripleo-heat-templates-5.0.0-0.5.0rc3.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1500265 (view as bug list)
Environment:
Last Closed:	2016-12-14 16:13:58 UTC
Target Upstream Version:
Embargoed:
Flags:	scohen: needinfo+

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1632285	None	None	None	2016-10-11 10:54:46 UTC
OpenStack gerrit	386462	'None'	MERGED	Enable Glance multiple locations when using Ceph	2020-12-30 07:44:49 UTC
OpenStack gerrit	387299	'None'	MERGED	glance_multiple_locations when NovaEnableRbdBackend=true	2020-12-30 07:44:51 UTC
Red Hat Product Errata	RHEA-2016:2948	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 10 enhancement update	2016-12-14 19:55:27 UTC

Description Attila Fazekas 2016-10-07 14:30:22 UTC

Description of problem:
Even when I am trying to create a snapshot of server as an admin it fails.



Version-Release number of selected component (if applicable):
openstack-nova-compute-14.0.0-0.20160929203854.59653c6.el7ost.noarch

How reproducible:
always

Steps to Reproduce:
basic setup with ceph.

Actual results:
#openstack server image create f969bd74-6c74-43f6-8b0a-7ed444318739
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                                                          |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum         | None                                                                                                                                                                           |
| container_format | None                                                                                                                                                                           |
| created_at       | 2016-10-07T14:12:22Z                                                                                                                                                           |
| disk_format      | None                                                                                                                                                                           |
| file             | /v2/images/02916b0a-0323-4eac-a0d4-c35b70e4efdb/file                                                                                                                           |
| id               | 02916b0a-0323-4eac-a0d4-c35b70e4efdb                                                                                                                                           |
| min_disk         | 0                                                                                                                                                                              |
| min_ram          | 0                                                                                                                                                                              |
| name             | test-snap                                                                                                                                                                      |
| owner            | af6cabae0def4cd0800596303c45612b                                                                                                                                               |
| properties       | base_image_ref='90825c15-8c54-4e01-8670-4a1a32024b1b', image_type='snapshot', instance_uuid='f969bd74-6c74-43f6-8b0a-7ed444318739', user_id='25085416a11e40d890e1617acadac032' |
| protected        | False                                                                                                                                                                          |
| schema           | /v2/schemas/image                                                                                                                                                              |
| size             | None                                                                                                                                                                           |
| status           | queued                                                                                                                                                                         |
| tags             |                                                                                                                                                                                |
| updated_at       | 2016-10-07T14:12:22Z                                                                                                                                                           |
| virtual_size     | None                                                                                                                                                                           |
| visibility       | private                                                                                                                                                                        |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# openstack image show 02916b0a-0323-4eac-a0d4-c35b70e4efdb
Could not find resource 02916b0a-0323-4eac-a0d4-c35b70e4efdb


Expected results:
I get my snapshot.


Additional info:
glance backend is ceph, and
show_image_direct_url = True .

In the compute log:
4318739] Cleaning up image 02916b0a-0323-4eac-a0d4-c35b70e4efdb decorated_function /usr/lib/python2.7/site-packages/nova/compute/manager.py:236
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] Traceback (most recent call last):
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 232, in decorated_function
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     *args, **kwargs)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3035, in snapshot_instance
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     task_states.IMAGE_SNAPSHOT)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3065, in _snapshot_instance
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     update_task_state)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1545, in snapshot
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     ignore_errors=True)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     self.force_reraise()
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     six.reraise(self.type_, self.value, self.tb)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1485, in snapshot
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     purge_props=False)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 130, in update
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     purge_props=purge_props)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 742, in update
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     _reraise_translated_image_exception(image_id)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 1069, in _reraise_translated_image_exception
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     six.reraise(new_exc, None, exc_trace)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 740, in update
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     image = self._update_v2(context, sent_service_image_meta, data)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 754, in _update_v2
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     image = self._add_location(context, image_id, location)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 639, in _add_location
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     location, {})
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 177, in call
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     result = getattr(controller, method)(*args, **kwargs)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 311, in add_location
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     self._send_image_update_request(image_id, add_patch)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 296, in _send_image_update_request
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     self.http_client.patch(url, headers=hdrs, data=json.dumps(patch_body))
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 296, in patch
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     return self._request('PATCH', url, **kwargs)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 279, in _request
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     resp, body_iter = self._handle_response(resp)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739]     raise exc.from_response(resp, resp.content)
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] ImageNotAuthorized: Not authorized for image 02916b0a-0323-4eac-a0d4-c35b70e4efdb.
2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] 

2016-10-07 14:12:34.135 6923 DEBUG oslo_messaging._drivers.amqpdriver [req-6b845f23-efb5-4970-aefc-35e1d22d5a9a 25085416a11e40d890e1617acadac032 af6cabae0def4cd0800596303c45612b - - -] CAST unique_id: 097b362b81
794b93899798d2ff63fc4a NOTIFY exchange 'nova' topic 'versioned_notifications.error' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:432
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server [req-6b845f23-efb5-4970-aefc-35e1d22d5a9a 25085416a11e40d890e1617acadac032 af6cabae0def4cd0800596303c45612b - - -] Exception during message handling
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 242, in decorated_function
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     instance=instance)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 232, in decorated_function
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     *args, **kwargs)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3035, in snapshot_instance
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     task_states.IMAGE_SNAPSHOT)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3065, in _snapshot_instance
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     update_task_state)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1545, in snapshot
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     ignore_errors=True)
/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1545, in snapshot
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     ignore_errors=True)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     self.force_reraise()
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1485, in snapshot
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     purge_props=False)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 130, in update
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     purge_props=purge_props)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 742, in update
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     _reraise_translated_image_exception(image_id)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 1069, in _reraise_translated_image_exception
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     six.reraise(new_exc, None, exc_trace)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 740, in update
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     image = self._update_v2(context, sent_service_image_meta, data)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 754, in _update_v2
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     image = self._add_location(context, image_id, location)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 639, in _add_location
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     location, {})
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 177, in call
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     result = getattr(controller, method)(*args, **kwargs)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 311, in add_location
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     self._send_image_update_request(image_id, add_patch)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 296, in _send_image_update_request
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     self.http_client.patch(url, headers=hdrs, data=json.dumps(patch_body))
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 296, in patch
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     return self._request('PATCH', url, **kwargs)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 279, in _request
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     resp, body_iter = self._handle_response(resp)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server     raise exc.from_response(resp, resp.content)
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server ImageNotAuthorized: Not authorized for image 02916b0a-0323-4eac-a0d4-c35b70e4efdb.
2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server 


 ImageNotAuthorized: Not authorized for image 02916b0a-0323-4eac-a0d4-c35b70e4efdb.\n2016-10-07 14:12:32.112 6923 ERROR nova.virt.libvirt.driver


Here the creds was ok (n-cpu to glance:9292):
 
6923  sendto(25, "GET /v2/schemas/image HTTP/1.1\r\nHost: 172.17.3.10:9292\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-glanceclient\r\nX-Roles: admin\r\nX-Aut
h-Token: 973d5c0bba4c4c5c8fd284a0ae713830\r\nX-Tenant-Id: af6cabae0def4cd0800596303c45612b\r\nX-User-Id: 25085416a11e40d890e1617acadac032\r\nX-Identity-Status: Confirmed\r\nContent-Type: application/octet-stream
\r\n\r\n", 386, 0, NULL, 0) = 386


6923  recvfrom(25, "HTTP/1.1 200 OK\r\nContent-Type: application/json; charset=UTF-8\r\nContent-Length: 4137\r\nX-Openstack-Request-Id: req-154aefab-865f-447a-ace5-7905ebc6244c\r\nDate: Fri, 07 Oct 2016 14:12:31
 GMT\r\n\r\n{\"additionalProperties\": {\"type\": \"string\"}, \"name\": \"image\", \"links\": [{\"href\": \"{self}\", \"rel\": \"self\"}, {\"href\": \"{file}\", \"rel\": \"enclosure\"}, {\"href\": \"{schema}\",
 \"rel\": \"describedby\"}], \"properties\": {\"status\": {\"readOnly\": true, \"enum\": [\"queued\", \"saving\", \"active\", \"killed\", \"deleted\", \"pending_delete\", \"deactivated\"], \"type\": \"string\", 
\"description\": \"Status of the image\"}, \"tags\": {\"items\": {\"type\": \"string\", \"maxLength\": 255}, \"type\": \"array\", \"description\": \"List of strings related to the image\"}, \"kernel_id\": {\"pat
tern\": \"^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$\", \"type\": [\"null\", \"string\"], \"description\": \"ID of image stored in Glance that should be used as the k
ernel when booting an AMI-style image.\", \"is_base\": false}, \"container_format\": {\"enum\": [null, \"ami\", \"ari\", \"aki\", \"bare\", \"ovf\", \"ova\", \"docker\"], \"type\": [\"null\", \"string\"], \"desc
ription\": \"Format of the container\"}, \"min_ram\": {\"type\": \"integer\", \"description\": \"Amount of ram (in MB) required to boot image.\"}, \"ramdisk_id\": {\"pattern\": \"^([0-9a-fA-F]){8}-([0-9a-fA-F]){
4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$\", \"type\": [\"null\", \"string\"], \"description\": \"ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.\",
 \"is_base\": false}, \"locations\": {\"items\": {\"required\": [\"url\", \"metadata\"], \"type\": \"object\", \"properties\": {\"url\": {\"type\": \"string\", \"maxLength\": 255}, \"metadata\": {\"type\": \"obj
ect\"}}}, \"type\": \"array\", \"description\": \"A set of URLs to access the image file kept in external store\"}, \"visibility\": {\"enum\": [\"public\", \"private\"], \"type\": \"string\", \"description\": \"
Scope of image accessibility\"}, \"updated_at\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"Date and time of the last image modification\"}, \"owner\": {\"type\": [\"null\", \"string\"], \"des
cription\": \"Owner of the image\", \"maxLength\": 255}, \"file\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"An image file url\"}, \"min_disk\": {\"type\": \"integer\", \"description\": \"Amo
unt of disk space (in GB) required to boot image.\"}, \"virtual_size\": {\"readOnly\": true, \"type\": [\"null\", \"integer\"], \"description\": \"Virtual size of image in bytes\"}, \"id\": {\"pattern\": \"^([0-
9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$\", \"type\": \"string\", \"description\": \"An identifier for the image\"}, \"size\": {\"readOnly\": true, \"type\": [\"null\", 
\"integer\"], \"description\": \"Size of image file in bytes\"}, \"instance_uuid\": {\"type\": \"string\", \"description\": \"Metadata which can be used to record which instance this image is associated with. (I
nformational only, does not create an instance snapshot.)\", \"is_base\": false}, \"os_distro\": {\"type\": \"string\", \"description\": \"Common name of operating system distribution as specified in http://docs
.openstack.org/trunk/openstack-compute/admin/content/adding-images.html\", \"is_base\": false}, \"name\": {\"type\": [\"null\", \"string\"], \"description\": \"Descriptive name for the image\", \"maxLength\": 25
5}, \"checksum\": {\"readOnly\": true, \"type\": [\"null\", \"string\"], \"description\": \"md5 hash of image contents.\", \"maxLength\": 32}, \"created_at\": {\"readOnly\": true, \"type\": \"string\", \"descrip
tion\": \"Date and time of image registration\"}, \"disk_format\": {\"enum\": [null, \"ami\", \"ari\", \"aki\", \"vhd\", \"vhdx\", \"vmdk\", \"raw\", \"qcow2\", \"vdi\", \"iso\"], \"type\": [\"null\", \"string\"
], \"description\": \"Format of the disk\"}, \"os_version\": {\"type\": \"string\", \"description\": \"Operating system version as specified by the distributor\", \"is_base\": false}, \"protected\": {\"type\": \
"boolean\", \"description\": \"If true, image will not be deletable.\"}, \"architecture\": {\"type\": \"string\", \"description\": \"Operating system architecture as specified in http://docs.openstack.org/trunk/
openstack-compute/admin/content/adding-images.html\", \"is_base\": false}, \"direct_url\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"URL to access the image file kept in external store\"}, \"
self\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"An image self url\"}, \"schema\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"An image schema url\"}}}", 8192, 0, NULL, NUL
L) = 4328


The Patch method was rejected here:

6923  sendto(29, "PATCH /v2/images/02916b0a-0323-4eac-a0d4-c35b70e4efdb HTTP/1.1\r\nHost: 172.17.3.10:9292\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-glancec
lient\r\nX-Roles: admin\r\nX-Auth-Token: 973d5c0bba4c4c5c8fd284a0ae713830\r\nX-Tenant-Id: af6cabae0def4cd0800596303c45612b\r\nX-User-Id: 25085416a11e40d890e1617acadac032\r\nX-Identity-Status: Confirmed\r\nConten
t-Type: application/openstack-images-v2.1-json-patch\r\nContent-Length: 168\r\n\r\n[{\"path\": \"/locations/-\", \"value\": {\"url\": \"rbd://8a92aa38-8bc8-11e6-a94f-525400ddf257/images/02916b0a-0323-4eac-a0d4-c
35b70e4efdb/snap\", \"metadata\": {}}, \"op\": \"add\"}]", 627, 0, NULL, 0) = 627

6923  recvfrom(29, "HTTP/1.1 403 Forbidden\r\nContent-Length: 183\r\nContent-Type: text/html; charset=UTF-8\r\nX-Openstack-Request-Id: req-3fa3d1c3-9599-4132-83f6-92ec60b9f506\r\nDate: Fri, 07 Oct 2016 14:12:32 GMT\r\n\r\n<html>\n <head>\n  <title>403 Forbidden</title>\n </head>\n <body>\n  <h1>403 Forbidden</h1>\n  It's not allowed to add locations if locations are invisible.<br /><br />\n\n\n\n </body>\n</html>", 8192, 0, NULL, NULL) = 373

Comment 3 Attila Fazekas 2016-10-07 15:32:20 UTC

Setting the show_multiple_locations=True in the glance-api.conf on the controller nodes can help.

Comment 4 Attila Fazekas 2016-10-07 18:49:32 UTC

I can't remember to the full history, but the issue might be related to moving to glance v2.

show_image_direct_url itself leads to some security concerns, I did not checked what will be consequences of using show_multiple_locations.

Comment 5 Attila Fazekas 2016-10-11 10:21:04 UTC

Similar issue solved with devstack-plugin-ceph: https://review.openstack.org/#/c/279630/

Comment 6 Erno Kuvaja 2016-10-11 13:54:55 UTC

Taking over from Giulio. This is likely indeed related to how Glance-api gets configured and what Nova tries to do during snapshot.

Comment 10 Paul Grist 2016-11-03 20:53:56 UTC

*** Bug 1388119 has been marked as a duplicate of this bug. ***

Comment 11 Tzach Shefi 2016-11-14 06:38:41 UTC

Verified, on: openstack-tripleo-heat-templates-5.0.0-1.6.el7ost.noarch
Image from an instance created successfully.


Glance uses CEPH 
default_store = rbd

glance --os-image-api-version 2 image-show 02003428-a88b-4798-a350-b4965a99ea15
...
| locations        | [{"url": "rbd://118c1c40-a9bf-                                                   |
|                  | 11e6-9924-525400cbcb06/images/02003428-a88b-4798-

Nova also uses CEPH
[root@compute-0 ~]# grep -i rbd /etc/nova/nova.conf 
images_rbd_pool=vms
rbd_secret_uuid=118c1c40-a9bf-11e6-9924-525400cbcb06
images_type=rbd
images_rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=openstack

Booted an instance called: server1
[stack@undercloud-0 ~]$ nova list
..
| 8c556a44-d190-4c53-bab0-bb20884a8f63 | server1 | ACTIVE | -          | Running     | public=10.0.0.211 |

Created an image from server1:

[stack@undercloud-0 ~]$ openstack server image create 8c556a44-d190-4c53-bab0-bb20884a8f63
+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                        |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2016-11-14T06:25:12.000000                                                                                                                                                   |
| deleted    | False                                                                                                                                                                        |
| id         | c03d0614-05e5-458a-ab82-d472ff3c301b                                                                                                                                         |
| is_public  | False                                                                                                                                                                        |
| min_disk   | 2                                                                                                                                                                            |
| min_ram    | 0                                                                                                                                                                            |
| name       | server1                                                                                                                                                                      |
| owner      | 30c8fba5b4f34392b738e10a48457233                                                                                                                                             |
| properties | base_image_ref='02003428-a88b-4798-a350-b4965a99ea15', image_type='snapshot', instance_uuid='8c556a44-d190-4c53-bab0-bb20884a8f63',                                          |
|            | user_id='6c1c70bc084b45d88d117eabab491c7a'                                                                                                                                   |
| protected  | False                                                                                                                                                                        |
| status     | queued                                                                                                                                                                       |
| updated_at | 2016-11-14T06:25:12.000000                                                                                                                                                   |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Image is created from server1 successfully. 
Before this image-list only showed initial cirros.raw image.


[stack@undercloud-0 ~]$ openstack image list
+--------------------------------------+------------+--------+
| ID                                   | Name       | Status |
+--------------------------------------+------------+--------+
| c03d0614-05e5-458a-ab82-d472ff3c301b | server1    | active |
| 02003428-a88b-4798-a350-b4965a99ea15 | cirros.raw | active |
+--------------------------------------+------------+--------+

I've also booted a second instance (FromServer1Image) from this new image to verify it - also works.  

[stack@undercloud-0 ~]$ nova boot FromServer1Image --flavor good --image server1
...

[stack@undercloud-0 ~]$ nova list
+--------------------------------------+------------------+--------+------------+-------------+-------------------+
| ID                                   | Name             | Status | Task State | Power State | Networks          |
+--------------------------------------+------------------+--------+------------+-------------+-------------------+
| 3081811c-478d-4f89-8ccb-9092b9831de9 | FromServer1Image | ACTIVE | -          | Running     | public=10.0.0.215 |
| 8c556a44-d190-4c53-bab0-bb20884a8f63 | server1          | ACTIVE | -          | Running     | public=10.0.0.211 |
+--------------------------------------+------------------+--------+------------+-------------+-------------------+

Comment 12 Tim Wilkinson 2016-11-14 18:16:07 UTC

This bz appears to focus on the inability to snapshot in the undercloud. Can it also prevent nova from creating a snapshot (nova image-create) in the overcloud?

Comment 14 Tim Wilkinson 2016-11-16 15:55:13 UTC

(In reply to Tim Wilkinson from comment #12)
> Can it also prevent nova from creating a snapshot (nova image-create) in the
> overcloud?

Erno supplied the answer that resolved the issue for me. I was able to make the necessary change to the glance config files and restart the services. Thanks.

Comment 15 Tzach Shefi 2016-11-17 09:12:12 UTC

#c14, Erno answered Tim.

Comment 17 errata-xmlrpc 2016-12-14 16:13:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html

Note You need to log in before you can comment on or make changes to this bug.