Description of problem: A snapshot cannot be deleted when the instance is shutdown, and if the volume has multiple snapshots and cinder nfs backend is used. When we use nfs backend in cinder and attach a cinder volume to an instance, the instance accesses to the file in nfs share. When the instance is stopped with "openstack server stop <instance-id>", then take two(or more) snapshots like this: "openstack volume snapshot create --force --volume <volume-id> snap1" "openstack volume snapshot create --force --volume <volume-id> snap2" And when we delete the snapshot of them with "openstack volume snapshot delete snap2", we see an error like this: ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack volume snapshot list +--------------------------------------+-------+-------------+----------------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+-------+-------------+----------------+------+ | d4581408-1f83-4e10-b405-fbf035bd9a9a | snap2 | None | error_deleting | 1 | | 862fa91c-54a5-4276-97b4-1ce97e3dcc21 | snap1 | None | available | 1 | +--------------------------------------+-------+-------------+----------------+------+ ~~~ And I can see the error log in the nova-compute.log 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server [req-0792de34-5f50-4537-b8ec-4f911975b48f 100509e7c2514548bcf2dcacee226dd7 eff560ac34024ce0878b9281c181801c - default default] Exception during message handling: DiskNotFound: No disk at volume-88909ff1-8bfc-400f-a d35-3e4a4ed989f5 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 226, in inner 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server return func(*args, **kwargs) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server function_name, call_dict, binary) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3430, in volume_snapshot_delete 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server snapshot_id, delete_info) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2689, in volume_snapshot_delete 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server context, snapshot_id, 'error_deleting') 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2681, in volume_snapshot_delete 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server snapshot_id, delete_info=delete_info) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2641, in _volume_snapshot_delete 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server rebase_base) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2476, in _rebase_with_qemu_img 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server b_file_fmt = images.qemu_img_info(backing_file).file_format 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 57, in qemu_img_info 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server raise exception.DiskNotFound(location=path) 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server DiskNotFound: No disk at volume-88909ff1-8bfc-400f-ad35-3e4a4ed989f5 2020-06-30 12:35:23.236 1 ERROR oslo_messaging.rpc.server ~~~ Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create a volume in cinder nfs backend 2. Create a bfv instance with the volume 3. Stop the instance 4. Take two or more snapshots of the volume 5. Delete the snapshot Actual results: DiskNotFound error occurs. Expected results: No error occurs. Additional info:
For some reason we aren't checking if the domain is actually running within n-cpu when handling this... Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server [req-570281c6-566e-44a3-9953-eeb634513778 req-0fbbe87f-fd1d-4861-9fb3-21b8eb011e55 service nova] Exception during message handling: libvirt.libvirtError: Requested operation is not valid: domain is not > Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 241, in inner Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 78, in wrapped Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server self.force_reraise() Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server raise value Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3916, in volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server snapshot_id, delete_info) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3129, in volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server context, snapshot_id, 'error_deleting') Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server self.force_reraise() Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server raise value Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3122, in volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server snapshot_id, delete_info=delete_info) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3107, in _volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server result = dev.commit(commit_base, commit_top, relative=True) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 809, in commit Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server self._disk, base, top, self.COMMIT_DEFAULT_BANDWIDTH, flags=flags) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 190, in doit Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 148, in proxy_call Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 129, in execute Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server raise value Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 83, in tworker Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.7/site-packages/libvirt.py", line 728, in blockCommit Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server libvirt.libvirtError: Requested operation is not valid: domain is not running
Hi, any updates on this? I can share further information if it's needed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543