Bug 1947840

Summary: [OSP 16.2] virDomainBlockCommit called when deleting a snapshot via os-assisted-volume-snapshots even when instance is shutoff
Product: Red Hat OpenStack Reporter: Lee Yarwood <lyarwood>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.2 (Train)CC: dasmith, eglynn, jhakimra, kchamart, nova-maint, sbauza, sgordon, stephenfin, vromanso
Target Milestone: betaKeywords: Patch, Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-nova-20.6.1-2.20210419130324.366f938.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1939422 Environment:
Last Closed: 2021-09-15 07:13:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1939422    
Bug Blocks:    

Description Lee Yarwood 2021-04-09 11:34:08 UTC 
+++ This bug was initially created as a clone of Bug #1939422 +++

Description of problem:

As noted in bug #1852299 attempting to delete a NFS volume snapshot (via c-api and the the os-assisted-volume-snapshots n-api) of a volume attached to a SHUTOFF instance currently results in n-cpu attempting to fire off a virDomainBlockCommit command even though the instance isn't running.

Version-Release number of selected component (if applicable):
All.

How reproducible:
Always.

Steps to Reproduce:
1. Create multiple volume snapshots against a volume.
2. Attach the volume to an ACTIVE instance.
3. Stop the instance and ensure it is SHUTOFF.
4. Attempt to delete the latest snapshot.

Actual results:
virDomainBlockCommit is called even though the domain isn't running.

Expected results:
qemu-img commit or qemu-img rebase should be used to handle this offline.

Additional info:

Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server [req-570281c6-566e-44a3-9953-eeb634513778 req-0fbbe87f-fd1d-4861-9fb3-21b8eb011e55 service nova] Exception during message handling: libvirt.libvirtError: Requested operation is not valid: domain is not >
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 78, in wrapped
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 3916, in volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     snapshot_id, delete_info)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3129, in volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     context, snapshot_id, 'error_deleting')
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3122, in volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     snapshot_id, delete_info=delete_info)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3107, in _volume_snapshot_delete
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = dev.commit(commit_base, commit_top, relative=True)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 809, in commit
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     self._disk, base, top, self.COMMIT_DEFAULT_BANDWIDTH, flags=flags)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 190, in doit
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 148, in proxy_call
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 129, in execute
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     raise value
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 83, in tworker
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib64/python3.7/site-packages/libvirt.py", line 728, in blockCommit
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self)
Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server libvirt.libvirtError: Requested operation is not valid: domain is not running
Comment 14 errata-xmlrpc 2021-09-15 07:13:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483