Description of problem: I've created the lab with barbican and a encrypted cinder volume: (overcloud) [stack@undercloud-0 ~]$ openstack volume type list +--------------------------------------+----------------------------+-----------+ | ID | Name | Is Public | +--------------------------------------+----------------------------+-----------+ | 502e53b8-3b83-4bff-bde3-5559b5cc3b91 | tripleo_ceph_volumes_new | True | | 151cec8d-7592-48b9-809d-32f9cad9b85c | LuksEncryptor-Template-256 | True | | 097fb345-e7af-4d41-bb4b-1c81ddba36f0 | tripleo_lvm | True | | 1f5cf5ff-4ded-4283-9837-0fa9f2cc1999 | tripleo_nfs | True | | 535c37f4-a1c0-4fd6-a257-d4adf36eff4d | tripleo_ceph | True | | 1bd2cf2a-0a98-454c-abd6-c929fccaf461 | tripleo_hpelefthand | True | +--------------------------------------+----------------------------+-----------+ (overcloud) [stack@undercloud-0 ~]$ openstack volume type show 151cec8d-7592-48b9-809d-32f9cad9b85c +--------------------+--------------------------------------+ | Field | Value | +--------------------+--------------------------------------+ | access_project_ids | None | | description | None | | id | 151cec8d-7592-48b9-809d-32f9cad9b85c | | is_public | True | | name | LuksEncryptor-Template-256 | | properties | volume_backend_name='tripleo_ceph' | | qos_specs_id | None | +--------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack volume list +--------------------------------------+-------------------------+----------------+------+---------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+-------------------------+----------------+------+---------------------------------------------------------------+ | 1cc68c1a-62cc-4bd3-8f40-d3b44bcb652d | test_boot_encrypted_vol | in-use | 2 | Attached to test_enc_inst on /dev/vda | (overcloud) [stack@undercloud-0 ~]$ openstack volume show 1cc68c1a-62cc-4bd3-8f40-d3b44bcb652d --fit-width +--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [{u'server_id': u'43494b5a-b29f-444e-9484-de58430d5a0c', u'attachment_id': u'cad7ab82-a837-41cc-978f-32c8dba2bf9b', u'attached_at': u'2020-05-13T15:18:39.000000', | | | u'host_name': u'compute-1.localdomain', u'volume_id': u'1cc68c1a-62cc-4bd3-8f40-d3b44bcb652d', u'device': u'/dev/vda', u'id': u'1cc68c1a-62cc-4bd3-8f40-d3b44bcb652d'}] | | availability_zone | nova | | bootable | true | | consistencygroup_id | None | | created_at | 2020-05-13T14:46:44.000000 | | description | None | | encrypted | True | | id | 1cc68c1a-62cc-4bd3-8f40-d3b44bcb652d | | migration_status | None | | multiattach | False | | name | test_boot_encrypted_vol | | os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | a578afa9dad2434182d053c07ffba4b8 | | properties | attached_mode='rw' | | replication_status | None | | size | 2 | | snapshot_id | None | | source_volid | None | | status | in-use | | type | LuksEncryptor-Template-256 | | updated_at | 2020-05-13T15:18:40.000000 | | user_id | f9ca5a979d024d58aba8804b29f9c804 | | volume_image_metadata | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'image_name': u'cirros_0.3.4', u'image_id': u'b12db32e-ac60-4e77-8fbf-c09ad3fc8d6d', u'checksum': | | | u'ee1eca47dc88f4879d8a229cc70a07c6', u'min_disk': u'0', u'size': u'13287936'} | | 43494b5a-b29f-444e-9484-de58430d5a0c | test_enc_inst | ACTIVE | - | Running | ten=172.10.10.22 | So with the instance running on the compute node; I emulated a crash with: [root@compute-1 ~]# echo 'c' > /proc/sysrq-trigger Before that; I set the config in nova to resume the guest state upon a reboot: [root@compute-1 ~]# egrep -v '^$|^#' /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf | grep -i resume resume_guests_state_on_host_boot=True The instance went into error state once the compute node came back up after the crash: | 43494b5a-b29f-444e-9484-de58430d5a0c | test_enc_inst | ERROR | - | NOSTATE | ten=172.10.10.22 | +--------------------------------------+--------------------+--------+------------+-------------+------------------------------+ (overcloud) [stack@undercloud-0 ~]$ nova show test_enc_inst +--------------------------------------+----------------------------------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | compute-1.localdomain | | OS-EXT-SRV-ATTR:hostname | test-enc-inst | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-0000002c | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-5n36t7eg | | OS-EXT-SRV-ATTR:root_device_name | /dev/vda | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | 2020-05-13T15:18:56.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2020-05-13T15:18:29Z | | description | test_enc_inst | | flavor:disk | 10 | | flavor:ephemeral | 0 | | flavor:extra_specs | {} | | flavor:original_name | m1.tiny.test | | flavor:ram | 512 | | flavor:swap | 0 | | flavor:vcpus | 1 | | hostId | 62dfca41bc62701e028be3f12c7ace0c7f7f4a010a3f6e5953b2cd20 | | host_status | UP | | id | 43494b5a-b29f-444e-9484-de58430d5a0c | | image | Attempt to boot from volume - no image supplied | | key_name | - | | locked | False | | metadata | {} | | name | test_enc_inst | | os-extended-volumes:volumes_attached | [{"id": "1cc68c1a-62cc-4bd3-8f40-d3b44bcb652d", "delete_on_termination": false}] | | security_groups | default | | status | ERROR | | tags | [] | | ten network | 172.10.10.22 | | tenant_id | a578afa9dad2434182d053c07ffba4b8 | | updated | 2020-05-19T05:28:00Z | | user_id | f9ca5a979d024d58aba8804b29f9c804 | +--------------------------------------+----------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ In the nova_compute container logs; I can see the same trace as yours: +++ 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [req-38be3556-05a7-47aa-b7b0-9d8da8e1ba3e - - - - -] [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] Failure attaching encryptor; rolling back volume connection: NotFound: (http://172.17.1.17:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-ee512420-3211-4309-8185-880e2bb055ee) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] Traceback (most recent call last): 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1265, in _connect_volume 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] context, connection_info, encryption, allow_native_luks) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1380, in _attach_encryptor 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] key = keymgr.get(context, encryption['encryption_key_id']) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 556, in get 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] secret = self._get_secret(context, managed_object_id) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 526, in _get_secret 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] barbican_client = self._get_barbican_client(context) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 136, in _get_barbican_client 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] self._barbican_endpoint) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 198, in _create_base_url 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] discovery = auth.get_discovery(sess, url=endpoint) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 556, in get_discovery 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] authenticated=authenticated) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 1171, in get_discovery 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] disc = Discover(session, url, authenticated=authenticated) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 401, in __init__ 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] authenticated=authenticated) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 99, in get_version_data 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] resp = session.get(url, headers=headers, authenticated=authenticated) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 840, in get 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return self.request(url, 'GET', **kwargs) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 573, in request 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] auth_headers = self.get_auth_headers(auth) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 900, in get_auth_headers 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return auth.get_headers(self, **kwargs) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] token = self.get_token(session) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return self.get_access(session).auth_token 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] self.auth_ref = self.get_auth_ref(session) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return self._plugin.get_auth_ref(session, **kwargs) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] authenticated=False, log=False) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 848, in post 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return self.request(url, 'POST', **kwargs) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 737, in request 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] raise exceptions.from_response(resp, method, url) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] NotFound: (http://172.17.1.17:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-ee512420-3211-4309-8185-880e2bb055ee) 2020-05-19 05:27:41.274 7 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [req-5d2cfce1-a8a1-455a-b7e6-123ddb49dcb5 - - - - -] [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] Failure attaching encryptor; rolling back volume connection: NotFound: (http://172.17.1.17:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-0d6b35b6-7616-4985-a0f3-e3a7fff4ebe8) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] Traceback (most recent call last): 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1265, in _connect_volume 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] context, connection_info, encryption, allow_native_luks) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1380, in _attach_encryptor 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] key = keymgr.get(context, encryption['encryption_key_id']) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 556, in get 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] secret = self._get_secret(context, managed_object_id) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 526, in _get_secret 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] barbican_client = self._get_barbican_client(context) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 136, in _get_barbican_client 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] self._barbican_endpoint) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 198, in _create_base_url 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] discovery = auth.get_discovery(sess, url=endpoint) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 556, in get_discovery 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] authenticated=authenticated) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 1171, in get_discovery 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] disc = Discover(session, url, authenticated=authenticated) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 401, in __init__ 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] authenticated=authenticated) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/discover.py", line 99, in get_version_data 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] resp = session.get(url, headers=headers, authenticated=authenticated) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 840, in get 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return self.request(url, 'GET', **kwargs) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 573, in request 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] auth_headers = self.get_auth_headers(auth) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 900, in get_auth_headers 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return auth.get_headers(self, **kwargs) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 95, in get_headers 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] token = self.get_token(session) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 88, in get_token 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return self.get_access(session).auth_token 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] self.auth_ref = self.get_auth_ref(session) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 201, in get_auth_ref 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return self._plugin.get_auth_ref(session, **kwargs) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 63, in get_auth_ref 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] authenticated=False, log=False) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 848, in post 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] return self.request(url, 'POST', **kwargs) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 737, in request 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] raise exceptions.from_response(resp, method, url) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] NotFound: (http://172.17.1.17:5000/v2.0/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-0d6b35b6-7616-4985-a0f3-e3a7fff4ebe8) 2020-05-19 05:27:45.258 8 ERROR nova.virt.libvirt.driver [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] 2020-05-19 05:27:49.425 7 DEBUG nova.compute.manager [req-bb584163-bdca-44ef-8ffc-6e7636a2c2ca - - - - -] [instance: 43494b5a-b29f-444e-9484-de58430d5a0c] Instance is in error state. _init_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:811 [root@compute-1 ~]# +++ I think this is because libvirt does not have the secret to decrypt the volume; the only secret present in libvirt post a reboot is the ceph key: [root@compute-1 ~]# virsh secret-list UUID Usage -------------------------------------------------------------------------------- 8d93628a-5f7f-11e9-bf47-525400a08118 ceph client.openstack secret Version-Release number of selected component (if applicable): [root@controller-1 ~]# docker ps | grep -e cinder -e barbican e5cef74e46c5 192.168.24.1:8787/rhosp13/openstack-cinder-api:13.0-104 "dumb-init --singl..." 4 weeks ago Up 4 weeks cinder_api_cron 726f2e96ce39 192.168.24.1:8787/rhosp13/openstack-cinder-scheduler:13.0-105 "dumb-init --singl..." 4 weeks ago Up 4 weeks (healthy) cinder_scheduler f5c8a8f12069 192.168.24.1:8787/rhosp13/openstack-cinder-api:13.0-104 "dumb-init --singl..." 4 weeks ago Up 4 weeks (healthy) cinder_api baf6f10893b9 192.168.24.1:8787/rhosp13/openstack-barbican-worker:13.0-94 "dumb-init --singl..." 4 months ago Up 5 weeks (healthy) barbican_worker ca79adb3e4a9 192.168.24.1:8787/rhosp13/openstack-barbican-keystone-listener:13.0-94 "dumb-init --singl..." 4 months ago Up 5 weeks (healthy) barbican_keystone_listener 5182721482e3 192.168.24.1:8787/rhosp13/openstack-barbican-api:13.0-92 "dumb-init --singl..." 4 months ago Up 5 weeks (healthy) barbican_api [root@compute-0 ~]# docker ps | grep -e nova 15519337292d 192.168.24.1:8787/rhosp13/openstack-nova-compute:13.0-119.1577111040 "dumb-init --singl..." 4 months ago Up 4 months (healthy) nova_compute ef2631c36bda 192.168.24.1:8787/rhosp13/openstack-nova-compute:13.0-119.1577111040 "dumb-init --singl..." 4 months ago Up 4 months (healthy) nova_migration_target 5839d5fe9e4a 192.168.24.1:8787/rhosp13/openstack-nova-libvirt:13.0-123.1578407520 "dumb-init --singl..." 4 months ago Up 4 months nova_libvirt 15e1b0228bb4 192.168.24.1:8787/rhosp13/openstack-nova-libvirt:13.0-123.1578407520 "dumb-init --singl..." 4 months ago Up 4 months nova_virtlogd How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
The issue isn't with the libvirt secret, it's with the admin context being used by nova.compute.manager.ComputeManager.init_host. I thought this had been resolved but it looks like we can't find the auth token referenced by the context here? Can you share the nova-compute.log somewhere and I'll try to reproduce locally.
FWIW I can reproduce this upstream still even with service users and end up with an empty catalog within the context: May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR castellan.key_manager.barbican_key_manager [None req-bdfed3a4-3d83-4084-8013-fa6b03759ef7 None None] Error creating Barbican client: The service catalog is empty.: keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty. May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [None req-bdfed3a4-3d83-4084-8013-fa6b03759ef7 None None] [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] Failure attaching encryptor; rolling back volume connection: castellan.common.exception.KeyManagerError: Key manager err> May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] Traceback (most recent call last): May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] File "/usr/local/lib/python3.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 115, in _get_barbican_client May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] self._barbican_endpoint = self._get_barbican_endpoint(auth, sess) May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] File "/usr/local/lib/python3.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 187, in _get_barbican_endpoint May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] service_type='key-manager') May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] File "/usr/local/lib/python3.7/site-packages/keystoneauth1/access/service_catalog.py", line 425, in endpoint_data_for May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] raise exceptions.EmptyCatalog('The service catalog is empty.') May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty. May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] During handling of the above exception, another exception occurred: May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] Traceback (most recent call last): May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1577, in _connect_volume May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] self._attach_encryptor(context, connection_info, encryption) May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1696, in _attach_encryptor May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] key = keymgr.get(context, encryption['encryption_key_id']) May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] File "/usr/local/lib/python3.7/site-packages/castellan/key_manager/migration.py", line 56, in get May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] context, managed_object_id) May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] File "/usr/local/lib/python3.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 562, in get May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] secret = self._get_secret(context, managed_object_id) May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] File "/usr/local/lib/python3.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 532, in _get_secret May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] barbican_client = self._get_barbican_client(context) May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] File "/usr/local/lib/python3.7/site-packages/castellan/key_manager/barbican_key_manager.py", line 125, in _get_barbican_client May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] raise exception.KeyManagerError(reason=e) May 28 10:11:24 localhost.localdomain nova-compute[342332]: ERROR nova.virt.libvirt.driver [instance: f3a9899f-c35b-4ddd-bfdf-bbf970a47d0a] castellan.common.exception.KeyManagerError: Key manager error: The service catalog is empty.
Hello Lee, Thanks for the help. Since you are able to reproduce the issue; do you still need access to the lab where I tested this or the nova compute logs ? Regards, Punit
Apologies for the delay, there are two issues here: #1 The EmptyCatalog exception raised as the bare bones admin context we are using during nova-compute startup doesn't contain a service catalog #2 The fact that even if the service catalog is populated an admin by default still wouldn't be able to gain access to the user secrets required to attach an encrypted volume We can potentially fix #1 as there may be other flows in the future where we need to reach out to external services as the admin user while relaunching instances. #2 however is different, we can't override the default policy of the key manager (Barbican) from Nova so unless the secrets were previously created by the admin *or* the default policy is changed in the environment attempting to resume instances on host start up will continue to fail even with #1 fixed. I'm going to switch this bug to tracking #1 while also documenting #2 as a known issue at the moment. Let me know if you or the customer has any questions regarding this.
*** This bug has been marked as a duplicate of bug 1905017 ***