+++ This bug was initially created as a clone of Bug #1934513 +++ Description of problem: $subject, by default admins do not have access to user created barbican secrets. As a result admins cannot hard reboot or stop/start instances as this deletes local libvirt secrets, refetches secrets from Barbican and recreates the local secrets. However this initial attempt by an admin will destroy the local secrets *before* failing to access anything in Barbican. As a result any request by the owner of the instance to hard reboot or stop/start the instance can fail as the _detach_encryptor logic fails to find any local secret and assumes that native LUKSv1 encryption isn't being used. This causes the os-brick encryptors to be loaded that can fail if the underlying volume type isn't supported, such as rbd. This was initially seen in https://bugzilla.redhat.com/show_bug.cgi?id=1933268 and results in a trace like the following: 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server [req-fe304872-e35f-4cb3-8760-4fd1eed745bc fef8c04ca63ab77e9a37b9d79367fd49747d2016352759f6faa8475fbf6f63c1 4127275f099844f28fde120064aa4753 - 1d485afd913b4c489730f79d83044080 1d485afd913b4c489730f79d83044080] Exception during message handling: os_brick.exception.VolumeEncryptionNotSupported: Volume encryption is not supported for rbd volume d9817c6a-9c84-472a-8fc8-58ad73b389aa. 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server raise value 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 191, in decorated_function 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server raise value 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 161, in decorated_function 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server raise value 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3140, in start_instance 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server self._power_on(context, instance) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 3110, in _power_on 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server block_device_info) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3459, in power_on 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server self._hard_reboot(context, instance, network_info, block_device_info) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3306, in _hard_reboot 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server block_device_info=block_device_info) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1316, in destroy 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server destroy_disks) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1389, in cleanup 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server cleanup_instance_disks=cleanup_instance_disks) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1474, in _cleanup 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server instance=instance) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server raise value 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1461, in _cleanup 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server self._disconnect_volume(context, connection_info, instance) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1687, in _disconnect_volume 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server self._detach_encryptor(context, connection_info, encryption=encryption) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1813, in _detach_encryptor 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server encryption) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1730, in _get_volume_encryptor 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server **encryption) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/encryptors/__init__.py", line 93, in get_volume_encryptor 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server **kwargs) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/importutils.py", line 44, in import_object 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server return import_class(import_str)(*args, **kwargs) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/encryptors/luks.py", line 61, in __init__ 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server *args, **kwargs) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/os_brick/encryptors/cryptsetup.py", line 55, in __init__ 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server volume_type=connection_info['driver_volume_type']) 2021-02-23 17:07:50.453 7 ERROR oslo_messaging.rpc.server os_brick.exception.VolumeEncryptionNotSupported: Volume encryption is not supported for rbd volume d9817c6a-9c84-472a-8fc8-58ad73b389aa. Version-Release number of selected component (if applicable): 13, 16.{0,1,2} and 17 How reproducible: Always Steps to Reproduce: 1. As an non-admin user create an instance with encrypted rbd volumes attached 2. Attempt to hard reboot or stop/start the instance as an admin 3. Attempt to hard reboot or stop/start the instance as the owner Actual results: The request by the admin to hard reboot or stop/start the instance fails. The request by the owner to hard reboot or stop/start the instance fails due to os_brick.exception.VolumeEncryptionNotSupported being raised. Expected results: The request by the admin to hard reboot or stop/start the instance fails. The request by the owner to hard reboot or stop/start the instance succeeds. Additional info:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1001