+++ This bug was initially created as a clone of Bug #1840060 +++ Description of problem: Instances booted from encrypted volumes fails to boot post compute reboot. Version-Release number of selected component (if applicable): Red Hat Openstack 16.1+Barbican Actual results: Instances boot up fails post reboot of compute node Expected results: Instance boot up should be successful post compute reboot Additional info: 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty. 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]
Hello Team, One of the Customer is testing compute node reboot with parameter "resume_guests_state_on_host_boot=True". Environment = RHOSP16.1 + Barbican The VMs with non-encrypted volumes come bac properly where with encrypted one fails with below error in in nova-compute.logs ~~~ 2020-08-20 11:30:12.271 7 WARNING stevedore.named [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] Could not load castellan.key_manager.barbican_key_manager.BarbicanKeyManager 2020-08-20 11:30:12.272 7 WARNING castellan.key_manager [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] Deprecation Warning : castellan.key_manager.barbican_key_manager.BarbicanKeyManager is not a stevedore based driver, trying to load it as a class: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'castellan.key_manager.barbican_key_manager.BarbicanKeyManager' 2020-08-20 11:30:12.272 7 DEBUG barbicanclient.client [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] Creating Client object Client /usr/lib/python3.6/site-packages/barbicanclient/client.py:156 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] Failure attaching encryptor; rolling back volume con nection: keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty. 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] Traceback (most recent call last): 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1641, in _connect_volume 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] self._attach_encryptor(context, connection_info, encryption) 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1760, in _attach_encryptor 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] key = keymgr.get(context, encryption['encryption_key_id']) 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 575, in get 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] secret = self._get_secret(context, managed_object_id) 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 545, in _ge t_secret 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] barbican_client = self._get_barbican_client(context) 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 142, in _ge t_barbican_client 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] self._barbican_endpoint) 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 214, in _cr eate_base_url 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] service_type='key-manager') 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File "/usr/lib/python3.6/site-packages/keystoneauth1/access/service_catalog.py", line 425, in endpoint_ data_for 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] raise exceptions.EmptyCatalog('The service catalog is empty.') 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty. 2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] 2020-08-20 11:30:12.275 7 WARNING nova.compute.manager [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] Failed to resume instance: keystoneauth1.exceptions.ca talog.EmptyCatalog: The service catalog is empty. ~~~ This bugs seems to be similar like the one we have for RHOSP13: https://bugzilla.redhat.com/show_bug.cgi?id=1840060.