Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1852299

Summary:	Snapshot cannot be deleted when the instance is shutdown after multiple snapshots creation if cinder nfs backend is used
Product:	Red Hat OpenStack	Reporter:	Masayuki Igawa <migawa>
Component:	openstack-nova	Assignee:	Lee Yarwood <lyarwood>
Status:	CLOSED ERRATA	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	13.0 (Queens)	CC:	dasmith, eglynn, jhakimra, kchamart, lyarwood, sbauza, sgordon, vromanso
Target Milestone:	beta	Keywords:	Triaged
Target Release:	17.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-nova-23.0.3-0.20210908140341.e39bbdc.el9ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1939411 (view as bug list)		Environment:
Last Closed:	2022-09-21 12:10:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1939411, 1939413, 1939415

Description Masayuki Igawa 2020-06-30 05:25:01 UTC

Description of problem:

A snapshot cannot be deleted when the instance is shutdown, and if the volume has multiple snapshots and cinder nfs backend is used.

When we use nfs backend in cinder and attach a cinder volume to an instance, the instance accesses to the file in nfs share.

When the instance is stopped with "openstack server stop <instance-id>", then take two(or more) snapshots like this:
 "openstack volume snapshot create --force --volume <volume-id> snap1"
 "openstack volume snapshot create --force --volume <volume-id> snap2"

And when we delete the snapshot of them with "openstack volume snapshot delete snap2",
we see an error like this:
~~~
(overcloud) [stack@undercloud-0 ~]$ openstack volume snapshot list
+--------------------------------------+-------+-------------+----------------+------+
| ID                                   | Name  | Description | Status         | Size |
+--------------------------------------+-------+-------------+----------------+------+
| d4581408-1f83-4e10-b405-fbf035bd9a9a | snap2 | None        | error_deleting |    1 |
| 862fa91c-54a5-4276-97b4-1ce97e3dcc21 | snap1 | None        | available      |    1 |
+--------------------------------------+-------+-------------+----------------+------+
~~~

And I can see the error log in the nova-compute.log
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server [req-0792de34-5f50-4537-b8ec-4f911975b48f 100509e7c2514548bcf2dcacee226dd7 eff560ac34024ce0878b9281c181801c - default default] Exception during message handling: DiskNotFound: No disk at volume-88909ff1-8bfc-400f-a
d35-3e4a4ed989f5
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3430, in volume_snapshot_delete
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     snapshot_id, delete_info)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2689, in volume_snapshot_delete
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     context, snapshot_id, 'error_deleting')
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2681, in volume_snapshot_delete
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     snapshot_id, delete_info=delete_info)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2641, in _volume_snapshot_delete
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     rebase_base)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2476, in _rebase_with_qemu_img
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     b_file_fmt = images.qemu_img_info(backing_file).file_format
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 57, in qemu_img_info
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server     raise exception.DiskNotFound(location=path)
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server DiskNotFound: No disk at volume-88909ff1-8bfc-400f-ad35-3e4a4ed989f5
2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server 
~~~


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create a volume in cinder nfs backend
2. Create a bfv instance with the volume
3. Stop the instance
4. Take two or more snapshots of the volume
5. Delete the snapshot

Actual results:

DiskNotFound error occurs.

Expected results:

No error occurs.

Additional info:

Comment 1 Lee Yarwood 2020-07-03 13:42:05 UTC

For some reason we aren't checking if the domain is actually running within n-cpu when handling this...

Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server [req-570281c6-566e-44a3-9953-eeb634513778 req-0fbbe87f-fd1d-4861-9fb3-21b8eb011e55 service nova] Exception during message handling: libvirt.libvirtError: Requested operation is not valid: domain is not >
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 78, in wrapped
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3916, in volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     snapshot_id, delete_info)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3129, in volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     context, snapshot_id, 'error_deleting')
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3122, in volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     snapshot_id, delete_info=delete_info)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3107, in _volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = dev.commit(commit_base, commit_top, relative=True)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 809, in commit
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self._disk, base, top, self.COMMIT_DEFAULT_BANDWIDTH, flags=flags)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 190, in doit
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 129, in execute
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 83, in tworker
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib64/python3.7/site-packages/libvirt.py", line 728, in blockCommit
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server libvirt.libvirtError: Requested operation is not valid: domain is not running

Comment 2 Masayuki Igawa 2020-07-27 05:08:51 UTC

Hi, any updates on this? I can share further information if it's needed.

Comment 15 errata-xmlrpc 2022-09-21 12:10:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543