Bug 1934513

Summary: Attempting to start or hard reboot a users instance as an admin with encrypted rbd volumes leaves the instance unbootable
Product: Red Hat OpenStack Reporter: Lee Yarwood <lyarwood>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 17.0 (Wallaby)CC: dasmith, eglynn, jhakimra, kchamart, sbauza, sgordon, vromanso
Target Milestone: betaKeywords: Patch, Triaged
Target Release: 17.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-23.0.3-0.20210908140341.e39bbdc.el9ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1998556 (view as bug list) Environment:
Last Closed: 2022-09-21 12:14:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1998556, 2000220    

Description Lee Yarwood 2021-03-03 11:53:59 UTC 
Description of problem:

$subject, by default admins do not have access to user created barbican secrets. As a result admins cannot hard reboot or stop/start instances as this deletes local libvirt secrets, refetches secrets from Barbican and recreates the local secrets.

However this initial attempt by an admin will destroy the local secrets *before* failing to access anything in Barbican.

As a result any request by the owner of the instance to hard reboot or stop/start the instance can fail as the _detach_encryptor logic fails to find any local secret and assumes that native LUKSv1 encryption isn't being used. This causes the os-brick encryptors to be loaded that can fail if the underlying volume type isn't supported, such as rbd.

This was initially seen in https://bugzilla.redhat.com/show_bug.cgi?id=1933268 and results in a trace like the following:

2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server [req-fe304872-e35f-4cb3-8760-4fd1eed745bc fef8c04ca63ab77e9a37b9d79367fd49747d2016352759f6faa8475fbf6f63c1 4127275f099844f28fde120064aa4753 - 1d485afd913b4c489730f79d83044080 1d485afd913b4c489730f79d83044080] Exception during message handling: os_brick.exception.VolumeEncryptionNotSupported: Volume encryption is not supported for rbd volume d9817c6a-9c84-472a-8fc8-58ad73b389aa.
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     raise value
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 191, in decorated_function
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     raise value
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 161, in decorated_function
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     raise value
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3140, in start_instance
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3110, in _power_on
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     block_device_info)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3459, in power_on
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3306, in _hard_reboot
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     block_device_info=block_device_info)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1316, in destroy
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     destroy_disks)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1389, in cleanup
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     cleanup_instance_disks=cleanup_instance_disks)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1474, in _cleanup
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     instance=instance)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     raise value
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1461, in _cleanup
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     self._disconnect_volume(context, connection_info, instance)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1687, in _disconnect_volume
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     self._detach_encryptor(context, connection_info, encryption=encryption)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1813, in _detach_encryptor
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     encryption)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1730, in _get_volume_encryptor
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     **encryption)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/encryptors/__init__.py", line 93, in get_volume_encryptor
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     **kwargs)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/importutils.py", line 44, in import_object
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     return import_class(import_str)(*args, **kwargs)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/encryptors/luks.py", line 61, in __init__
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     *args, **kwargs)
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/os_brick/encryptors/cryptsetup.py", line 55, in __init__
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server     volume_type=connection_info['driver_volume_type'])
2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server os_brick.exception.VolumeEncryptionNotSupported: Volume encryption is not supported for rbd volume d9817c6a-9c84-472a-8fc8-58ad73b389aa.

Version-Release number of selected component (if applicable):
13, 16.{0,1,2} and 17

How reproducible:
Always

Steps to Reproduce:
1. As an non-admin user create an instance with encrypted rbd volumes attached
2. Attempt to hard reboot or stop/start the instance as an admin
3. Attempt to hard reboot or stop/start the instance as the owner

Actual results:
The request by the admin to hard reboot or stop/start the instance fails.
The request by the owner to hard reboot or stop/start the instance fails due to os_brick.exception.VolumeEncryptionNotSupported being raised.

Expected results:
The request by the admin to hard reboot or stop/start the instance fails.
The request by the owner to hard reboot or stop/start the instance succeeds.

Additional info:
Comment 6 errata-xmlrpc 2022-09-21 12:14:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543