1795507 – Unable to extend an attached encypted volume while in use

Bug 1795507 - Unable to extend an attached encypted volume while in use

Summary: Unable to extend an attached encypted volume while in use

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	16.0 (Train)
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	beta
Target Release:	17.0
Assignee:	Lee Yarwood
QA Contact:	James Parker
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1816136
TreeView+	depends on / blocked

Reported:	2020-01-28 07:26 UTC by bkopilov
Modified:	2022-09-21 12:10 UTC (History)
CC List:	11 users (show)
Fixed In Version:	openstack-nova-23.0.3-0.20210908140341.e39bbdc.el9ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1816136 (view as bug list)
Environment:
Last Closed:	2022-09-21 12:09:13 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
Nova libvirt logs (20.62 KB, application/gzip) 2020-01-28 07:33 UTC, Tzach Shefi	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1861071	None	None	None	2020-01-28 07:51:47 UTC
OpenStack gerrit	706880	None	MERGED	imageutils: Report format specific details when using JSON output format	2020-12-15 22:52:03 UTC
OpenStack gerrit	706897	None	MERGED	images: Move qemu-img info calls into privsep	2020-12-15 22:52:35 UTC
OpenStack gerrit	706898	None	MERGED	images: Allow the output format of qemu-img info to be controlled	2020-12-15 22:52:35 UTC
OpenStack gerrit	706900	None	MERGED	libvirt: Correctly resize encrypted LUKSv1 volumes	2020-12-15 22:52:05 UTC
Red Hat Issue Tracker	OSP-3141	None	None	None	2022-04-13 20:09:40 UTC
Red Hat Product Errata	RHEA-2022:6543	None	None	None	2022-09-21 12:10:59 UTC

Description bkopilov 2020-01-28 07:26:53 UTC

Description of problem:
Hi, 
OpenStack cloud with backend - 3parfc

Encrypted volume attached to an instance,
When extending the volume, cinder reports that action was done successfully and the size changed BUT inside the VM the lsblk reports that the size was not changed.

We found a traceback from nova about failures in resize:

var/log/containers/nova/nova-compute.log:71:2020-01-28 07:14:17.394 7 ERROR nova.virt.libvirt.driver [instance: f93e235e-30e9-4bdc-ad48-8e972552bbb5] 
/var/log/containers/nova/nova-compute.log:73:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server [req-ef88174f-6762-4842-a64c-76074a848ac7 b753e5c55ba94950b5463ae41bb623ab 08d5a55b6cad4413910abc863b4a2b15 - default default] Exception during message handling: libvirt.libvirtError: internal error: unable to execute QEMU command 'block_resize': Cannot grow device files
/var/log/containers/nova/nova-compute.log:74:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
/var/log/containers/nova/nova-compute.log:75:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
/var/log/containers/nova/nova-compute.log:76:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
/var/log/containers/nova/nova-compute.log:77:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
/var/log/containers/nova/nova-compute.log:78:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
/var/log/containers/nova/nova-compute.log:79:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
/var/log/containers/nova/nova-compute.log:80:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
/var/log/containers/nova/nova-compute.log:81:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
/var/log/containers/nova/nova-compute.log:82:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
/var/log/containers/nova/nova-compute.log:83:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:84:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:85:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:86:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:87:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:88:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:89:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
/var/log/containers/nova/nova-compute.log:90:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
/var/log/containers/nova/nova-compute.log:91:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9223, in external_instance_event
/var/log/containers/nova/nova-compute.log:92:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.extend_volume(context, instance, event.tag)
/var/log/containers/nova/nova-compute.log:93:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function
/var/log/containers/nova/nova-compute.log:94:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:95:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function
/var/log/containers/nova/nova-compute.log:96:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
/var/log/containers/nova/nova-compute.log:97:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:98:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:99:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:100:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:101:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:102:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:103:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function
/var/log/containers/nova/nova-compute.log:104:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:105:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9081, in extend_volume
/var/log/containers/nova/nova-compute.log:106:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     bdm.volume_size * units.Gi)
/var/log/containers/nova/nova-compute.log:107:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2048, in extend_volume
/var/log/containers/nova/nova-compute.log:108:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     instance=instance)
/var/log/containers/nova/nova-compute.log:109:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-compute.log:110:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/nova/nova-compute.log:111:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-compute.log:112:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-compute.log:113:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:114:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:115:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2037, in extend_volume
/var/log/containers/nova/nova-compute.log:116:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     dev.resize(new_size // units.Ki)
/var/log/containers/nova/nova-compute.log:117:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 809, in resize
/var/log/containers/nova/nova-compute.log:118:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     self._guest._domain.blockResize(self._disk, size_kb)
/var/log/containers/nova/nova-compute.log:119:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
/var/log/containers/nova/nova-compute.log:120:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:121:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
/var/log/containers/nova/nova-compute.log:122:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
/var/log/containers/nova/nova-compute.log:123:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
/var/log/containers/nova/nova-compute.log:124:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
/var/log/containers/nova/nova-compute.log:125:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/nova/nova-compute.log:126:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/nova/nova-compute.log:127:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
/var/log/containers/nova/nova-compute.log:128:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
/var/log/containers/nova/nova-compute.log:129:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1006, in blockResize
/var/log/containers/nova/nova-compute.log:130:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainBlockResize() failed', dom=self)
/var/log/containers/nova/nova-compute.log:131:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: unable to execute QEMU command 'block_resize': Cannot grow device files
/var/log/containers/nova/nova-compute.log:132:2020-01-28 07:14:17.778 7 ERROR oslo_messaging.rpc.server 
/var/log/containers/libvirt/libvirtd.log:2:2020-01-28 07:14:17.392+0000: 4425: error : qemuMonitorJSONCheckError:418 : internal error: unable to execute QEMU command 'block_resize': Cannot grow device files



Thanks,
Benny

Comment 1 Tzach Shefi 2020-01-28 07:33:51 UTC

Created attachment 1655889 [details]
Nova libvirt logs

Version:

openstack-nova-compute-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
openstack-nova-migration-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
python3-novaclient-15.1.0-0.20190919143437.cd396b8.el8ost.noarch
python3-nova-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch
puppet-nova-15.4.1-0.20191126042922.b1bb388.el8ost.noarch
openstack-nova-common-20.0.2-0.20191230035951.27bfd0b.el8ost.noarch



If this isn't supported, than this BZ should turn into a doc bug no support for extending an attached encrypted volume.

Comment 2 Tzach Shefi 2020-01-28 08:29:00 UTC

FYI retested, 
This time encrypted volume was detached before extending and reattached after. 
Extending worked fine in this case, lsblk reported correct extended size.

This we know extending an attached volume doesn't support encrypted volumes,
Workaround detach the volume extend and reattach it.

Comment 10 errata-xmlrpc 2022-09-21 12:09:13 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543

Note You need to log in before you can comment on or make changes to this bug.