Bug 1870927

Summary:	[OSP 16.1] Service catalog empty when nova.compute.manager.ComputeManager._resume_guests_state is called
Product:	Red Hat OpenStack	Reporter:	Shailesh Chhabdiya <schhabdi>
Component:	openstack-nova	Assignee:	Lee Yarwood <lyarwood>
Status:	CLOSED DUPLICATE	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.1 (Train)	CC:	bretm, dasmith, eglynn, jhakimra, kchamart, lyarwood, mchappel, nova-maint, pkundal, sbauza, sgordon, vromanso
Target Milestone:	async	Keywords:	Patch, Triaged, ZStream
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1840060	Environment:
Last Closed:	2021-01-06 09:20:39 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1840060

Description Shailesh Chhabdiya 2020-08-21 04:16:20 UTC

+++ This bug was initially created as a clone of Bug #1840060 +++

Description of problem:

Instances booted from encrypted volumes fails to boot post compute reboot.


Version-Release number of selected component (if applicable):

Red Hat Openstack 16.1+Barbican


Actual results:
Instances boot up fails post reboot of compute node


Expected results:
Instance boot up should be successful post compute reboot


Additional info:
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty.
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]

Comment 1 Shailesh Chhabdiya 2020-08-21 04:18:38 UTC

Hello Team,

One of the Customer is testing compute node reboot with parameter "resume_guests_state_on_host_boot=True". 

Environment = RHOSP16.1 + Barbican

The VMs with non-encrypted volumes come bac properly where with encrypted one fails with below error in in nova-compute.logs

~~~
2020-08-20 11:30:12.271 7 WARNING stevedore.named [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] Could not load castellan.key_manager.barbican_key_manager.BarbicanKeyManager
2020-08-20 11:30:12.272 7 WARNING castellan.key_manager [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] Deprecation Warning : castellan.key_manager.barbican_key_manager.BarbicanKeyManager is not a stevedore
 based driver, trying to load it as a class: stevedore.exception.NoMatches: No 'castellan.drivers' driver found, looking for 'castellan.key_manager.barbican_key_manager.BarbicanKeyManager'
2020-08-20 11:30:12.272 7 DEBUG barbicanclient.client [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] Creating Client object Client /usr/lib/python3.6/site-packages/barbicanclient/client.py:156
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] Failure attaching encryptor; rolling back volume con
nection: keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty.
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] Traceback (most recent call last):
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1641, in _connect_volume
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     self._attach_encryptor(context, connection_info, encryption)
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1760, in _attach_encryptor
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     key = keymgr.get(context, encryption['encryption_key_id'])
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 575, in get
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     secret = self._get_secret(context, managed_object_id)
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 545, in _ge
t_secret
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     barbican_client = self._get_barbican_client(context)
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 142, in _ge
t_barbican_client
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     self._barbican_endpoint)
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 214, in _cr
eate_base_url
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     service_type='key-manager')
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]   File "/usr/lib/python3.6/site-packages/keystoneauth1/access/service_catalog.py", line 425, in endpoint_
data_for
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf]     raise exceptions.EmptyCatalog('The service catalog is empty.')
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty.
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] 
2020-08-20 11:30:12.275 7 WARNING nova.compute.manager [req-36b6c199-295f-4621-8aa8-b5011bcdc07b - - - - -] [instance: c5b3e7d4-99ea-409c-aba6-d32751f93ccf] Failed to resume instance: keystoneauth1.exceptions.ca
talog.EmptyCatalog: The service catalog is empty.
~~~

This bugs seems to be similar like the one we have for RHOSP13: https://bugzilla.redhat.com/show_bug.cgi?id=1840060.