Bug 1816136 - [OSP 16.1] Unable to extend an attached encypted volume while in use
Summary: [OSP 16.1] Unable to extend an attached encypted volume while in use
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 16.0 (Train)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: beta
: 16.1 (Train on RHEL 8.2)
Assignee: Lee Yarwood
QA Contact: OSP DFG:Compute
URL:
Whiteboard:
Depends On: 1795507
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-23 12:15 UTC by Lee Yarwood
Modified: 2023-03-21 19:28 UTC (History)
13 users (show)

Fixed In Version: openstack-nova-20.1.2-0.20200320003447.dd16430.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of: 1795507
Environment:
Last Closed: 2020-07-29 07:51:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:3148 0 None None None 2020-07-29 07:51:16 UTC

Description Lee Yarwood 2020-03-23 12:15:42 UTC
+++ This bug was initially created as a clone of Bug #1795507 +++

Description of problem:
Hi, 
OpenStack cloud with backend - 3parfc

Encrypted volume attached to an instance,
When extending the volume, cinder reports that action was done successfully and the size changed BUT inside the VM the lsblk reports that the size was not changed.

We found a traceback from nova about failures in resize:

var/log/containers/nova/nova-compute.log:71:2020-01-28 07:14:17.394 7 ERROR nova.virt.libvirt.driver [instance: f93e235e-30e9-4bdc-ad48-8e972552bbb5] 
/var/log/containers/nova/nova-compute.log:73:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server [req-ef88174f-6762-4842-a64c-76074a848ac7 b753e5c55ba94950b5463ae41bb623ab 08d5a55b6cad4413910abc863b4a2b15 - default default] Exception during message handling: libvirt.libvirtError: internal error: unable to execute QEMU command 'block_resize': Cannot grow device files
/var/log/containers/nova/nova-compute.log:74:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
/var/log/containers/nova/nova-compute.log:75:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
/var/log/containers/nova/nova-compute.log:76:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
/var/log/containers/nova/nova-compute.log:77:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
/var/log/containers/nova/nova-compute.log:78:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
/var/log/containers/nova/nova-compute.log:79:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
/var/log/containers/nova/nova-compute.log:80:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
/var/log/containers/nova/nova-compute.log:81:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
/var/log/containers/nova/nova-compute.log:82:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
/var/log/containers/nova/nova-compute.log:83:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:84:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:85:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:86:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:87:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:88:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:89:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
/var/log/containers/nova/nova-compute.log:90:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
/var/log/containers/nova/nova-compute.log:91:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9223, in external_instance_event
/var/log/containers/nova/nova-compute.log:92:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.extend_volume(context, instance, event.tag)
/var/log/containers/nova/nova-compute.log:93:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function
/var/log/containers/nova/nova-compute.log:94:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:95:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function
/var/log/containers/nova/nova-compute.log:96:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
/var/log/containers/nova/nova-compute.log:97:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:98:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:99:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:100:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:101:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:102:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:103:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function
/var/log/containers/nova/nova-compute.log:104:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:105:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9081, in extend_volume
/var/log/containers/nova/nova-compute.log:106:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     bdm.volume_size * units.Gi)
/var/log/containers/nova/nova-compute.log:107:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2048, in extend_volume
/var/log/containers/nova/nova-compute.log:108:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     instance=instance)
/var/log/containers/nova/nova-compute.log:109:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:110:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:111:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:112:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:113:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:114:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:115:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2037, in extend_volume
/var/log/containers/nova/nova-compute.log:116:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     dev.resize(new_size // units.Ki)
/var/log/containers/nova/nova-compute.log:117:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 809, in resize
/var/log/containers/nova/nova-compute.log:118:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self._guest._domain.blockResize(self._disk, size_kb)
/var/log/containers/nova/nova-compute.log:119:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
/var/log/containers/nova/nova-compute.log:120:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:121:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
/var/log/containers/nova/nova-compute.log:122:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:123:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
/var/log/containers/nova/nova-compute.log:124:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
/var/log/containers/nova/nova-compute.log:125:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:126:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:127:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
/var/log/containers/nova/nova-compute.log:128:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
/var/log/containers/nova/nova-compute.log:129:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1006, in blockResize
/var/log/containers/nova/nova-compute.log:130:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainBlockResize() failed', dom=self)
/var/log/containers/nova/nova-compute.log:131:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: unable to execute QEMU command 'block_resize': Cannot grow device files
/var/log/containers/nova/nova-compute.log:132:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server 
/var/log/containers/libvirt/libvirtd.log:2:2020-01-28 07:14:17.392+0000: 4425: error : qemuMonitorJSONCheckError:418 : internal error: unable to execute QEMU command 'block_resize': Cannot grow device files



Thanks,
Benny

--- Additional comment from Tzach Shefi on 2020-01-28 07:33:51 UTC ---

Version:

openstack-nova-compute-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
openstack-nova-migration-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
python3-novaclient-15.1.0-0.20190919143437.cd396b8.el8ost.noarch
python3-nova-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
puppet-nova-15.4.1-0.20191126042922.b1bb388.el8ost.noarch
openstack-nova-common-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch



If this isn't supported, than this BZ should turn into a doc bug no support for extending an attached encrypted volume.

--- Additional comment from Tzach Shefi on 2020-01-28 08:29:00 UTC ---

FYI retested, 
This time encrypted volume was detached before extending and reattached after. 
Extending worked fine in this case, lsblk reported correct extended size.

This we know extending an attached volume doesn't support encrypted volumes,
Workaround detach the volume extend and reattach it.

Comment 5 Alex McLeod 2020-06-16 12:28:29 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.

Comment 8 errata-xmlrpc 2020-07-29 07:51:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3148


Note You need to log in before you can comment on or make changes to this bug.