1749379 – server snapshot fails - 403 forbidden - on glance with ceph and show_multiple_locations=False

Bug 1749379 - server snapshot fails - 403 forbidden - on glance with ceph and show_multiple_locations=False

Summary: server snapshot fails - 403 forbidden - on glance with ceph and show_multiple...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-heat-templates
Sub Component:
Version:	15.0 (Stein)
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	ga
Target Release:	15.0 (Stein)
Assignee:	Giulio Fidente
QA Contact:	Pavel Sedlák
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-09-05 13:43 UTC by Pavel Sedlák
Modified:	2019-09-27 10:44 UTC (History)
CC List:	6 users (show)
Fixed In Version:	openstack-tripleo-heat-templates-10.6.1-0.20190905170437.b33b839.el8ost
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-09-21 11:24:31 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1842938	None	None	None	2019-09-05 14:59:50 UTC
OpenStack gerrit	680441	None	MERGED	Revert "Do not forcibly enable Glance multiple locations for RBD backend"	2020-09-14 11:33:09 UTC
Red Hat Product Errata	RHEA-2019:2811	None	None	None	2019-09-21 11:25:34 UTC

Description Pavel Sedlák 2019-09-05 13:43:44 UTC

Performing Server snapshots fails in OSP15 deployment with ceph.
Nova/Glance create image entry in state=SAVING, but after a while when nova attempt to update (PATCH) that image with final rbd location glance rejects it.

Based on https://bugzilla.redhat.com/show_bug.cgi?id=1382737 or comments in https://bugs.launchpad.net/glance/+bug/1595335 and https://review.opendev.org/#/c/279630/ it requires configuration of show_multiple_locations=True.

So seems this functionality gets broken by change https://review.opendev.org/#/c/678335/ done for https://bugzilla.redhat.com/show_bug.cgi?id=1737409.

Can be reproduced automatically using tempest tests. These all fail on image not found for snapshot:

> tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image[id-3731d080-d4c5-4872-b41a-64d0d0021314]
> tempest.api.compute.images.test_images.ImagesTestJSON.test_create_image_from_paused_server[id-71bcb732-0261-11e7-9086-fa163e4fa634]
> .setUpClass (tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON)
> tempest.api.compute.images.test_images.ImagesTestJSON.test_create_image_from_stopped_server[id-aaacd1d0-55a2-4ce8-818a-b5439df8adc9]
> tempest.api.compute.images.test_images.ImagesTestJSON.test_create_image_from_suspended_server[id-8ca07fec-0262-11e7-907e-fa163e4fa634]
> tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_create_backup[id-b963d4f1-94b3-4c40-9e97-7b583f46e470,image]
> tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_shelved_state[id-bb0cb402-09dd-4947-b6e5-5e7e1cfa61ad]
> tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_shelve_paused_server[id-8cf9f450-a871-42cf-9bef-77eba189c0b0]
> tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_shelve_shelved_server[id-443e4f9b-e6bf-4389-b601-3a710f15fddd,negative]

Failure in nova-compute.log looks as:
> 2019-09-05 03:06:28.202 7 DEBUG nova.virt.libvirt.storage.rbd_utils [req-4e065e65-4fdb-4a79-925e-d9d19b1b6f30 515aad1f25604a1c878e3182da4b084e 11212a93a21f449faec59d378261f0a9 - default default] creating snapshot(snap) on rbd image(393792f3-4137-42de-8c0f-d5f0755b6148) create_snap /usr/lib/python3.6/site-packages/nova/virt/libvirt/storage/rbd_utils.py:384
> ...
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver [req-4e065e65-4fdb-4a79-925e-d9d19b1b6f30 515aad1f25604a1c878e3182da4b084e 11212a93a21f449faec59d378261f0a9 - default default] Failed to snapshot image: nova.exception.ImageNotAuthorized: Not authorized for image 393792f3-4137-42de-8c0f-d5f0755b6148.
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last):
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 616, in update
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     image = self._update_v2(context, sent_service_image_meta, data)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 630, in _update_v2
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     image = self._add_location(context, image_id, location)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 492, in _add_location
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     context, 2, 'add_location', args=(image_id, location, {}))
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 193, in call
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     result = getattr(controller, method)(*args, **kwargs)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/v2/images.py", line 451, in add_location
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     response = self._send_image_update_request(image_id, add_patch)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/common/utils.py", line 598, in inner
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     return RequestIdProxy(wrapped(*args, **kwargs))
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/v2/images.py", line 432, in _send_image_update_request
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     data=json.dumps(patch_body))
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 387, in patch
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     return self.request(url, 'PATCH', **kwargs)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/common/http.py", line 377, in request
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     return self._handle_response(resp)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/common/http.py", line 126, in _handle_response
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     raise exc.from_response(resp, resp.content)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver glanceclient.exc.HTTPForbidden: HTTP 403 Forbidden: It&#x27;s not allowed to add locations if locations are invisible.
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver 
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver During handling of the above exception, another exception occurred:
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver 
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver Traceback (most recent call last):
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2037, in snapshot
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     purge_props=False)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/api.py", line 142, in update
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     purge_props=purge_props)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 618, in update
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     _reraise_translated_image_exception(image_id)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 938, in _reraise_translated_image_exception
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     six.reraise(type(new_exc), new_exc, exc_trace)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/six.py", line 692, in reraise
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     raise value.with_traceback(tb)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 616, in update
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     image = self._update_v2(context, sent_service_image_meta, data)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 630, in _update_v2
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     image = self._add_location(context, image_id, location)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 492, in _add_location
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     context, 2, 'add_location', args=(image_id, location, {}))
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 193, in call
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     result = getattr(controller, method)(*args, **kwargs)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/v2/images.py", line 451, in add_location
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     response = self._send_image_update_request(image_id, add_patch)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/common/utils.py", line 598, in inner
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     return RequestIdProxy(wrapped(*args, **kwargs))
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/v2/images.py", line 432, in _send_image_update_request
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     data=json.dumps(patch_body))
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 387, in patch
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     return self.request(url, 'PATCH', **kwargs)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/common/http.py", line 377, in request
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     return self._handle_response(resp)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver   File "/usr/lib/python3.6/site-packages/glanceclient/common/http.py", line 126, in _handle_response
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver     raise exc.from_response(resp, resp.content)
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver nova.exception.ImageNotAuthorized: Not authorized for image 393792f3-4137-42de-8c0f-d5f0755b6148.
> 2019-09-05 03:06:30.364 7 ERROR nova.virt.libvirt.driver 
> 2019-09-05 03:06:30.508 7 DEBUG nova.virt.libvirt.storage.rbd_utils [req-4e065e65-4fdb-4a79-925e-d9d19b1b6f30 515aad1f25604a1c878e3182da4b084e 11212a93a21f449faec59d378261f0a9 - default default] removing snapshot(snap) on rbd image(393792f3-4137-42de-8c0f-d5f0755b6148) remove_snap /usr/lib/python3.6/site-packages/nova/virt/libvirt/storage/rbd_utils.py:411


Packages on the undercloud:
> openstack-tripleo-heat-templates.noarch       10.6.1-0.20190904124632.4e2dddb.el8ost                   @rhelosp-15.0-trunk                 
> openstack-tripleo-common.noarch               10.8.1-0.20190831030439.300785c.el8ost                   @rhelosp-15.0-trunk                 
> openstack-tripleo-common-containers.noarch    10.8.1-0.20190831030439.300785c.el8ost                   @rhelosp-15.0-trunk                 
> python3-tripleo-common.noarch                 10.8.1-0.20190831030439.300785c.el8ost                   @rhelosp-15.0-trunk                 
> rhosp-director-images.noarch                  15.0-20190904.1.el8ost                                   @rhelosp-15.0-trunk

Comment 1 Giulio Fidente 2019-09-05 13:51:22 UTC

Dan, Erno, Lee can you guys help us understand if the change introduced by [1] is valid and or if that option should be set to true whent the Glance backend is rbd regardless?

1. https://bugzilla.redhat.com/show_bug.cgi?id=1737409

Comment 9 Pavel Sedlák 2019-09-06 15:34:03 UTC

Verified, failing tempest tests doing snapshots are now passing again.

version used during test (at undercloud-0):
> openstack-tripleo-heat-templates.noarch       10.6.1-0.20190905170437.b33b839.el8ost                   @rhelosp-15.0-trunk                 

option is switched back to true (at controller-0):
> # grep '^show_multi' var/lib/config-data/puppet-generated/glance_api/etc/glance/glance-api.conf 
> show_multiple_locations=True

Grep of tempets results xml for names of failing tests shows no failure:
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_filter_by_changes_since[id-18bac3ae-da27-436c-92a9-b22474d13aab]" time="0.110"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_filter_by_name[id-33163b73-79f5-4d07-a7ea-9213bcc468ff]" time="0.105"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_filter_by_server_id[id-9f238683-c763-45aa-b848-232ec3ce3105]" time="0.118"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_filter_by_server_ref[id-05a377b8-28cf-4734-a1e6-2ab5c38bf606]" time="0.255"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_filter_by_status[id-a3f5b513-aeb3-42a9-b18e-f091ef73254d]" time="0.136"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_filter_by_type[id-e3356918-4d3e-4756-81d5-abc4524ba29f]" time="0.125"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_limit_results[id-3a484ca9-67ba-451e-b494-7fcf28d32d62]" time="0.112"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_with_detail_filter_by_changes_since[id-7d439e18-ac2e-4827-b049-7e18004712c4]" time="0.167"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_with_detail_filter_by_name[id-644ea267-9bd9-4f3b-af9f-dffa02396a17]" time="0.128"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_with_detail_filter_by_server_ref[id-8c78f822-203b-4bf6-8bba-56ebd551cf84]" time="0.216"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_with_detail_filter_by_status[id-9b0ea018-6185-4f71-948a-a123a107988e]" time="0.104"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_with_detail_filter_by_type[id-888c0cc0-7223-43c5-9db0-b125fd0a393b]" time="0.185"/>
> <testcase classname="tempest.api.compute.images.test_list_image_filters.ListImageFiltersTestJSON" name="test_list_images_with_detail_limit_results[id-ba2fa9a9-b672-47cc-b354-3b4c0600e2cb]" time="0.118"/>

re-needinfo: Question about if it is really needed for server snapshots with ceph confirmed in https://bugzilla.redhat.com/show_bug.cgi?id=1737409#c9

Comment 14 errata-xmlrpc 2019-09-21 11:24:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811

Note You need to log in before you can comment on or make changes to this bug.