Bug 1382737
| Summary: | Nova snapshot fails in ceph direct image setup | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Attila Fazekas <afazekas> | |
| Component: | openstack-tripleo-heat-templates | Assignee: | Erno Kuvaja <ekuvaja> | |
| Status: | CLOSED ERRATA | QA Contact: | Attila Fazekas <afazekas> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 10.0 (Newton) | CC: | ekuvaja, gfidente, jomurphy, jschluet, mburns, pgrist, rhel-osp-director-maint, scohen, tshefi, twilkins | |
| Target Milestone: | rc | Keywords: | Automation, AutomationBlocker, Triaged | |
| Target Release: | 10.0 (Newton) | Flags: | scohen:
needinfo+
|
|
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-tripleo-heat-templates-5.0.0-0.5.0rc3.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1500265 (view as bug list) | Environment: | ||
| Last Closed: | 2016-12-14 16:13:58 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1500265 | |||
Setting the show_multiple_locations=True in the glance-api.conf on the controller nodes can help. I can't remember to the full history, but the issue might be related to moving to glance v2. show_image_direct_url itself leads to some security concerns, I did not checked what will be consequences of using show_multiple_locations. Similar issue solved with devstack-plugin-ceph: https://review.openstack.org/#/c/279630/ Taking over from Giulio. This is likely indeed related to how Glance-api gets configured and what Nova tries to do during snapshot. *** Bug 1388119 has been marked as a duplicate of this bug. *** Verified, on: openstack-tripleo-heat-templates-5.0.0-1.6.el7ost.noarch
Image from an instance created successfully.
Glance uses CEPH
default_store = rbd
glance --os-image-api-version 2 image-show 02003428-a88b-4798-a350-b4965a99ea15
...
| locations | [{"url": "rbd://118c1c40-a9bf- |
| | 11e6-9924-525400cbcb06/images/02003428-a88b-4798-
Nova also uses CEPH
[root@compute-0 ~]# grep -i rbd /etc/nova/nova.conf
images_rbd_pool=vms
rbd_secret_uuid=118c1c40-a9bf-11e6-9924-525400cbcb06
images_type=rbd
images_rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=openstack
Booted an instance called: server1
[stack@undercloud-0 ~]$ nova list
..
| 8c556a44-d190-4c53-bab0-bb20884a8f63 | server1 | ACTIVE | - | Running | public=10.0.0.211 |
Created an image from server1:
[stack@undercloud-0 ~]$ openstack server image create 8c556a44-d190-4c53-bab0-bb20884a8f63
+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2016-11-14T06:25:12.000000 |
| deleted | False |
| id | c03d0614-05e5-458a-ab82-d472ff3c301b |
| is_public | False |
| min_disk | 2 |
| min_ram | 0 |
| name | server1 |
| owner | 30c8fba5b4f34392b738e10a48457233 |
| properties | base_image_ref='02003428-a88b-4798-a350-b4965a99ea15', image_type='snapshot', instance_uuid='8c556a44-d190-4c53-bab0-bb20884a8f63', |
| | user_id='6c1c70bc084b45d88d117eabab491c7a' |
| protected | False |
| status | queued |
| updated_at | 2016-11-14T06:25:12.000000 |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Image is created from server1 successfully.
Before this image-list only showed initial cirros.raw image.
[stack@undercloud-0 ~]$ openstack image list
+--------------------------------------+------------+--------+
| ID | Name | Status |
+--------------------------------------+------------+--------+
| c03d0614-05e5-458a-ab82-d472ff3c301b | server1 | active |
| 02003428-a88b-4798-a350-b4965a99ea15 | cirros.raw | active |
+--------------------------------------+------------+--------+
I've also booted a second instance (FromServer1Image) from this new image to verify it - also works.
[stack@undercloud-0 ~]$ nova boot FromServer1Image --flavor good --image server1
...
[stack@undercloud-0 ~]$ nova list
+--------------------------------------+------------------+--------+------------+-------------+-------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------+--------+------------+-------------+-------------------+
| 3081811c-478d-4f89-8ccb-9092b9831de9 | FromServer1Image | ACTIVE | - | Running | public=10.0.0.215 |
| 8c556a44-d190-4c53-bab0-bb20884a8f63 | server1 | ACTIVE | - | Running | public=10.0.0.211 |
+--------------------------------------+------------------+--------+------------+-------------+-------------------+
This bz appears to focus on the inability to snapshot in the undercloud. Can it also prevent nova from creating a snapshot (nova image-create) in the overcloud? (In reply to Tim Wilkinson from comment #12) > Can it also prevent nova from creating a snapshot (nova image-create) in the > overcloud? Erno supplied the answer that resolved the issue for me. I was able to make the necessary change to the glance config files and restart the services. Thanks. #c14, Erno answered Tim. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html |
Description of problem: Even when I am trying to create a snapshot of server as an admin it fails. Version-Release number of selected component (if applicable): openstack-nova-compute-14.0.0-0.20160929203854.59653c6.el7ost.noarch How reproducible: always Steps to Reproduce: basic setup with ceph. Actual results: #openstack server image create f969bd74-6c74-43f6-8b0a-7ed444318739 +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | checksum | None | | container_format | None | | created_at | 2016-10-07T14:12:22Z | | disk_format | None | | file | /v2/images/02916b0a-0323-4eac-a0d4-c35b70e4efdb/file | | id | 02916b0a-0323-4eac-a0d4-c35b70e4efdb | | min_disk | 0 | | min_ram | 0 | | name | test-snap | | owner | af6cabae0def4cd0800596303c45612b | | properties | base_image_ref='90825c15-8c54-4e01-8670-4a1a32024b1b', image_type='snapshot', instance_uuid='f969bd74-6c74-43f6-8b0a-7ed444318739', user_id='25085416a11e40d890e1617acadac032' | | protected | False | | schema | /v2/schemas/image | | size | None | | status | queued | | tags | | | updated_at | 2016-10-07T14:12:22Z | | virtual_size | None | | visibility | private | +------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ # openstack image show 02916b0a-0323-4eac-a0d4-c35b70e4efdb Could not find resource 02916b0a-0323-4eac-a0d4-c35b70e4efdb Expected results: I get my snapshot. Additional info: glance backend is ceph, and show_image_direct_url = True . In the compute log: 4318739] Cleaning up image 02916b0a-0323-4eac-a0d4-c35b70e4efdb decorated_function /usr/lib/python2.7/site-packages/nova/compute/manager.py:236 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] Traceback (most recent call last): 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 232, in decorated_function 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] *args, **kwargs) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3035, in snapshot_instance 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] task_states.IMAGE_SNAPSHOT) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3065, in _snapshot_instance 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] update_task_state) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1545, in snapshot 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] ignore_errors=True) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] self.force_reraise() 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] six.reraise(self.type_, self.value, self.tb) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1485, in snapshot 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] purge_props=False) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 130, in update 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] purge_props=purge_props) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 742, in update 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] _reraise_translated_image_exception(image_id) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 1069, in _reraise_translated_image_exception 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] six.reraise(new_exc, None, exc_trace) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 740, in update 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] image = self._update_v2(context, sent_service_image_meta, data) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 754, in _update_v2 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] image = self._add_location(context, image_id, location) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 639, in _add_location 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] location, {}) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 177, in call 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] result = getattr(controller, method)(*args, **kwargs) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 311, in add_location 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] self._send_image_update_request(image_id, add_patch) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 296, in _send_image_update_request 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] self.http_client.patch(url, headers=hdrs, data=json.dumps(patch_body)) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 296, in patch 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] return self._request('PATCH', url, **kwargs) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 279, in _request 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] resp, body_iter = self._handle_response(resp) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] raise exc.from_response(resp, resp.content) 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] ImageNotAuthorized: Not authorized for image 02916b0a-0323-4eac-a0d4-c35b70e4efdb. 2016-10-07 14:12:32.931 6923 ERROR nova.compute.manager [instance: f969bd74-6c74-43f6-8b0a-7ed444318739] 2016-10-07 14:12:34.135 6923 DEBUG oslo_messaging._drivers.amqpdriver [req-6b845f23-efb5-4970-aefc-35e1d22d5a9a 25085416a11e40d890e1617acadac032 af6cabae0def4cd0800596303c45612b - - -] CAST unique_id: 097b362b81 794b93899798d2ff63fc4a NOTIFY exchange 'nova' topic 'versioned_notifications.error' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:432 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server [req-6b845f23-efb5-4970-aefc-35e1d22d5a9a 25085416a11e40d890e1617acadac032 af6cabae0def4cd0800596303c45612b - - -] Exception during message handling 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server function_name, call_dict, binary) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server LOG.warning(msg, e, instance=instance) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 242, in decorated_function 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server instance=instance) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 232, in decorated_function 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server *args, **kwargs) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3035, in snapshot_instance 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server task_states.IMAGE_SNAPSHOT) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3065, in _snapshot_instance 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server update_task_state) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1545, in snapshot 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server ignore_errors=True) /python2.7/site-packages/nova/virt/libvirt/driver.py", line 1545, in snapshot 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server ignore_errors=True) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server self.force_reraise() 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1485, in snapshot 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server purge_props=False) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 130, in update 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server purge_props=purge_props) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 742, in update 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 1069, in _reraise_translated_image_exception 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server six.reraise(new_exc, None, exc_trace) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 740, in update 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server image = self._update_v2(context, sent_service_image_meta, data) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 754, in _update_v2 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server image = self._add_location(context, image_id, location) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 639, in _add_location 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server location, {}) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 177, in call 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args, **kwargs) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 311, in add_location 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server self._send_image_update_request(image_id, add_patch) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 296, in _send_image_update_request 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server self.http_client.patch(url, headers=hdrs, data=json.dumps(patch_body)) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 296, in patch 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server return self._request('PATCH', url, **kwargs) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 279, in _request 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server resp, body_iter = self._handle_response(resp) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content) 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server ImageNotAuthorized: Not authorized for image 02916b0a-0323-4eac-a0d4-c35b70e4efdb. 2016-10-07 14:12:34.163 6923 ERROR oslo_messaging.rpc.server ImageNotAuthorized: Not authorized for image 02916b0a-0323-4eac-a0d4-c35b70e4efdb.\n2016-10-07 14:12:32.112 6923 ERROR nova.virt.libvirt.driver Here the creds was ok (n-cpu to glance:9292): 6923 sendto(25, "GET /v2/schemas/image HTTP/1.1\r\nHost: 172.17.3.10:9292\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-glanceclient\r\nX-Roles: admin\r\nX-Aut h-Token: 973d5c0bba4c4c5c8fd284a0ae713830\r\nX-Tenant-Id: af6cabae0def4cd0800596303c45612b\r\nX-User-Id: 25085416a11e40d890e1617acadac032\r\nX-Identity-Status: Confirmed\r\nContent-Type: application/octet-stream \r\n\r\n", 386, 0, NULL, 0) = 386 6923 recvfrom(25, "HTTP/1.1 200 OK\r\nContent-Type: application/json; charset=UTF-8\r\nContent-Length: 4137\r\nX-Openstack-Request-Id: req-154aefab-865f-447a-ace5-7905ebc6244c\r\nDate: Fri, 07 Oct 2016 14:12:31 GMT\r\n\r\n{\"additionalProperties\": {\"type\": \"string\"}, \"name\": \"image\", \"links\": [{\"href\": \"{self}\", \"rel\": \"self\"}, {\"href\": \"{file}\", \"rel\": \"enclosure\"}, {\"href\": \"{schema}\", \"rel\": \"describedby\"}], \"properties\": {\"status\": {\"readOnly\": true, \"enum\": [\"queued\", \"saving\", \"active\", \"killed\", \"deleted\", \"pending_delete\", \"deactivated\"], \"type\": \"string\", \"description\": \"Status of the image\"}, \"tags\": {\"items\": {\"type\": \"string\", \"maxLength\": 255}, \"type\": \"array\", \"description\": \"List of strings related to the image\"}, \"kernel_id\": {\"pat tern\": \"^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$\", \"type\": [\"null\", \"string\"], \"description\": \"ID of image stored in Glance that should be used as the k ernel when booting an AMI-style image.\", \"is_base\": false}, \"container_format\": {\"enum\": [null, \"ami\", \"ari\", \"aki\", \"bare\", \"ovf\", \"ova\", \"docker\"], \"type\": [\"null\", \"string\"], \"desc ription\": \"Format of the container\"}, \"min_ram\": {\"type\": \"integer\", \"description\": \"Amount of ram (in MB) required to boot image.\"}, \"ramdisk_id\": {\"pattern\": \"^([0-9a-fA-F]){8}-([0-9a-fA-F]){ 4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$\", \"type\": [\"null\", \"string\"], \"description\": \"ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.\", \"is_base\": false}, \"locations\": {\"items\": {\"required\": [\"url\", \"metadata\"], \"type\": \"object\", \"properties\": {\"url\": {\"type\": \"string\", \"maxLength\": 255}, \"metadata\": {\"type\": \"obj ect\"}}}, \"type\": \"array\", \"description\": \"A set of URLs to access the image file kept in external store\"}, \"visibility\": {\"enum\": [\"public\", \"private\"], \"type\": \"string\", \"description\": \" Scope of image accessibility\"}, \"updated_at\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"Date and time of the last image modification\"}, \"owner\": {\"type\": [\"null\", \"string\"], \"des cription\": \"Owner of the image\", \"maxLength\": 255}, \"file\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"An image file url\"}, \"min_disk\": {\"type\": \"integer\", \"description\": \"Amo unt of disk space (in GB) required to boot image.\"}, \"virtual_size\": {\"readOnly\": true, \"type\": [\"null\", \"integer\"], \"description\": \"Virtual size of image in bytes\"}, \"id\": {\"pattern\": \"^([0- 9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$\", \"type\": \"string\", \"description\": \"An identifier for the image\"}, \"size\": {\"readOnly\": true, \"type\": [\"null\", \"integer\"], \"description\": \"Size of image file in bytes\"}, \"instance_uuid\": {\"type\": \"string\", \"description\": \"Metadata which can be used to record which instance this image is associated with. (I nformational only, does not create an instance snapshot.)\", \"is_base\": false}, \"os_distro\": {\"type\": \"string\", \"description\": \"Common name of operating system distribution as specified in http://docs .openstack.org/trunk/openstack-compute/admin/content/adding-images.html\", \"is_base\": false}, \"name\": {\"type\": [\"null\", \"string\"], \"description\": \"Descriptive name for the image\", \"maxLength\": 25 5}, \"checksum\": {\"readOnly\": true, \"type\": [\"null\", \"string\"], \"description\": \"md5 hash of image contents.\", \"maxLength\": 32}, \"created_at\": {\"readOnly\": true, \"type\": \"string\", \"descrip tion\": \"Date and time of image registration\"}, \"disk_format\": {\"enum\": [null, \"ami\", \"ari\", \"aki\", \"vhd\", \"vhdx\", \"vmdk\", \"raw\", \"qcow2\", \"vdi\", \"iso\"], \"type\": [\"null\", \"string\" ], \"description\": \"Format of the disk\"}, \"os_version\": {\"type\": \"string\", \"description\": \"Operating system version as specified by the distributor\", \"is_base\": false}, \"protected\": {\"type\": \ "boolean\", \"description\": \"If true, image will not be deletable.\"}, \"architecture\": {\"type\": \"string\", \"description\": \"Operating system architecture as specified in http://docs.openstack.org/trunk/ openstack-compute/admin/content/adding-images.html\", \"is_base\": false}, \"direct_url\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"URL to access the image file kept in external store\"}, \" self\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"An image self url\"}, \"schema\": {\"readOnly\": true, \"type\": \"string\", \"description\": \"An image schema url\"}}}", 8192, 0, NULL, NUL L) = 4328 The Patch method was rejected here: 6923 sendto(29, "PATCH /v2/images/02916b0a-0323-4eac-a0d4-c35b70e4efdb HTTP/1.1\r\nHost: 172.17.3.10:9292\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-glancec lient\r\nX-Roles: admin\r\nX-Auth-Token: 973d5c0bba4c4c5c8fd284a0ae713830\r\nX-Tenant-Id: af6cabae0def4cd0800596303c45612b\r\nX-User-Id: 25085416a11e40d890e1617acadac032\r\nX-Identity-Status: Confirmed\r\nConten t-Type: application/openstack-images-v2.1-json-patch\r\nContent-Length: 168\r\n\r\n[{\"path\": \"/locations/-\", \"value\": {\"url\": \"rbd://8a92aa38-8bc8-11e6-a94f-525400ddf257/images/02916b0a-0323-4eac-a0d4-c 35b70e4efdb/snap\", \"metadata\": {}}, \"op\": \"add\"}]", 627, 0, NULL, 0) = 627 6923 recvfrom(29, "HTTP/1.1 403 Forbidden\r\nContent-Length: 183\r\nContent-Type: text/html; charset=UTF-8\r\nX-Openstack-Request-Id: req-3fa3d1c3-9599-4132-83f6-92ec60b9f506\r\nDate: Fri, 07 Oct 2016 14:12:32 GMT\r\n\r\n<html>\n <head>\n <title>403 Forbidden</title>\n </head>\n <body>\n <h1>403 Forbidden</h1>\n It's not allowed to add locations if locations are invisible.<br /><br />\n\n\n\n </body>\n</html>", 8192, 0, NULL, NULL) = 373