Bug 1816136

Summary: [OSP 16.1] Unable to extend an attached encypted volume while in use
Product: Red Hat OpenStack Reporter: Lee Yarwood <lyarwood>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: high Docs Contact:
Priority: high    
Version: 16.0 (Train)CC: amodi, bkopilov, dasmith, eglynn, jhakimra, kchamart, lyarwood, mbooth, nova-maint, sbauza, sgordon, tshefi, vromanso
Target Milestone: betaKeywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-20.1.2-0.20200320003447.dd16430.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1795507 Environment:
Last Closed: 2020-07-29 07:51:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1795507    
Bug Blocks:    

Description Lee Yarwood 2020-03-23 12:15:42 UTC 
+++ This bug was initially created as a clone of Bug #1795507 +++

Description of problem:
Hi, 
OpenStack cloud with backend - 3parfc

Encrypted volume attached to an instance,
When extending the volume, cinder reports that action was done successfully and the size changed BUT inside the VM the lsblk reports that the size was not changed.

We found a traceback from nova about failures in resize:

var/log/containers/nova/nova-compute.log:71:2020-01-28 07:14:17.394 7 ERROR nova.virt.libvirt.driver [instance: f93e235e-30e9-4bdc-ad48-8e972552bbb5] 
/var/log/containers/nova/nova-compute.log:73:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server [req-ef88174f-6762-4842-a64c-76074a848ac7 b753e5c55ba94950b5463ae41bb623ab 08d5a55b6cad4413910abc863b4a2b15 - default default] Exception during message handling: libvirt.libvirtError: internal error: unable to execute QEMU command 'block_resize': Cannot grow device files
/var/log/containers/nova/nova-compute.log:74:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
/var/log/containers/nova/nova-compute.log:75:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
/var/log/containers/nova/nova-compute.log:76:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
/var/log/containers/nova/nova-compute.log:77:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
/var/log/containers/nova/nova-compute.log:78:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
/var/log/containers/nova/nova-compute.log:79:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
/var/log/containers/nova/nova-compute.log:80:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
/var/log/containers/nova/nova-compute.log:81:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
/var/log/containers/nova/nova-compute.log:82:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
/var/log/containers/nova/nova-compute.log:83:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:84:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:85:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:86:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:87:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:88:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:89:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
/var/log/containers/nova/nova-compute.log:90:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
/var/log/containers/nova/nova-compute.log:91:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9223, in external_instance_event
/var/log/containers/nova/nova-compute.log:92:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.extend_volume(context, instance, event.tag)
/var/log/containers/nova/nova-compute.log:93:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function
/var/log/containers/nova/nova-compute.log:94:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:95:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function
/var/log/containers/nova/nova-compute.log:96:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
/var/log/containers/nova/nova-compute.log:97:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:98:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:99:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:100:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:101:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:102:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:103:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function
/var/log/containers/nova/nova-compute.log:104:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:105:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9081, in extend_volume
/var/log/containers/nova/nova-compute.log:106:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     bdm.volume_size * units.Gi)
/var/log/containers/nova/nova-compute.log:107:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2048, in extend_volume
/var/log/containers/nova/nova-compute.log:108:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     instance=instance)
/var/log/containers/nova/nova-compute.log:109:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:110:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:111:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:112:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:113:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:114:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:115:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2037, in extend_volume
/var/log/containers/nova/nova-compute.log:116:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     dev.resize(new_size // units.Ki)
/var/log/containers/nova/nova-compute.log:117:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 809, in resize
/var/log/containers/nova/nova-compute.log:118:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self._guest._domain.blockResize(self._disk, size_kb)
/var/log/containers/nova/nova-compute.log:119:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
/var/log/containers/nova/nova-compute.log:120:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:121:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
/var/log/containers/nova/nova-compute.log:122:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:123:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
/var/log/containers/nova/nova-compute.log:124:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
/var/log/containers/nova/nova-compute.log:125:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:126:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:127:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
/var/log/containers/nova/nova-compute.log:128:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
/var/log/containers/nova/nova-compute.log:129:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1006, in blockResize
/var/log/containers/nova/nova-compute.log:130:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainBlockResize() failed', dom=self)
/var/log/containers/nova/nova-compute.log:131:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: unable to execute QEMU command 'block_resize': Cannot grow device files
/var/log/containers/nova/nova-compute.log:132:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server 
/var/log/containers/libvirt/libvirtd.log:2:2020-01-28 07:14:17.392+0000: 4425: error : qemuMonitorJSONCheckError:418 : internal error: unable to execute QEMU command 'block_resize': Cannot grow device files



Thanks,
Benny

--- Additional comment from Tzach Shefi on 2020-01-28 07:33:51 UTC ---

Version:

openstack-nova-compute-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
openstack-nova-migration-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
python3-novaclient-15.1.0-0.20190919143437.cd396b8.el8ost.noarch
python3-nova-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
puppet-nova-15.4.1-0.20191126042922.b1bb388.el8ost.noarch
openstack-nova-common-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch



If this isn't supported, than this BZ should turn into a doc bug no support for extending an attached encrypted volume.

--- Additional comment from Tzach Shefi on 2020-01-28 08:29:00 UTC ---

FYI retested, 
This time encrypted volume was detached before extending and reattached after. 
Extending worked fine in this case, lsblk reported correct extended size.

This we know extending an attached volume doesn't support encrypted volumes,
Workaround detach the volume extend and reattach it.
Comment 5 Alex McLeod 2020-06-16 12:28:29 UTC 
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.
Comment 8 errata-xmlrpc 2020-07-29 07:51:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3148