Description of problem: Unable live/cold migrate between 2 computes Version-Release number of selected component (if applicable): Red Hat OpenStack Platform release 16.0.0 Beta (Train) How reproducible: Always Steps to Reproduce: 1. openstack server migrate 0289b07c-c4f8-4a84-9467-e79342f3b785 --live-migration --host compute-02.osp-001.lab.iad2.dc.redhat.com --wait --debug 2. no errors, but not migrated 3. Actual results: no errors, but not migrated Expected results: should be migrated to compute-02.osp-001.lab.iad2.dc.redhat.com Additional info: There is a recent sosreport from both compute available from both computes (source and destination), with the debug log attached available on supportshell. I am unsure if this is a nova or barbican problem, but looking at the errors, I believe it's more barbican related These are the errors: This is on compute 1 (source): ~~~ nova-compute.log:2020-06-03 14:25:14.359 7 ERROR nova.compute.manager [-] [instance: 0289b07c-c4f8-4a84-9467-e79342f3b785] Pre live migration failed at compute-02.osp-001.lab.iad2.dc.redhat.com: oslo_messaging.rpc.client.RemoteError: Remote error: KeyManagerError Key manager error: Forbidden: Secret retrieval attempt not allowed - please review your user/project privileges ~~~ This is on compute 2 (destination): ~~~ nova-compute.log:2020-06-03 14:25:14.074 7 ERROR nova.virt.libvirt.driver [req-f33be0f2-ae60-46e1-877b-44e0a98db3cc 523357c5ad9245fc994c8bc66577fbe8 778458988fa34607bafdeeed39116b93 - default default] [instance: 0289b07c-c4f8-4a84-9467-e79342f3b785] Failure attaching encryptor; rolling back volume connection: castellan.common.exception.KeyManagerError: Key manager error: Forbidden: Secret retrieval attempt not allowed - please review your user/project privileges nova-compute.log:2020-06-03 14:25:14.074 7 ERROR nova.virt.libvirt.driver [instance: 0289b07c-c4f8-4a84-9467-e79342f3b785] castellan.common.exception.KeyManagerError: Key manager error: Forbidden: Secret retrieval attempt not allowed - please review your user/project privileges nova-compute.log:2020-06-03 14:25:14.074 7 ERROR nova.virt.libvirt.driver [instance: 0289b07c-c4f8-4a84-9467-e79342f3b785] nova-compute.log:2020-06-03 14:25:14.575 7 INFO nova.virt.block_device [req-f33be0f2-ae60-46e1-877b-44e0a98db3cc 523357c5ad9245fc994c8bc66577fbe8 778458988fa34607bafdeeed39116b93 - default default] [instance: 0289b07c-c4f8-4a84-9467-e79342f3b785] Attempting to driver detach volume 0d9613ca-fc03-4a3e-b8be-c4487fb11f89 from mountpoint /dev/vda nova-compute.log:2020-06-03 14:25:14.576 7 WARNING nova.virt.block_device [req-f33be0f2-ae60-46e1-877b-44e0a98db3cc 523357c5ad9245fc994c8bc66577fbe8 778458988fa34607bafdeeed39116b93 - default default] [instance: 0289b07c-c4f8-4a84-9467-e79342f3b785] Detaching volume from unknown instance nova-compute.log:2020-06-03 14:25:14.640 7 WARNING nova.virt.libvirt.driver [req-f33be0f2-ae60-46e1-877b-44e0a98db3cc 523357c5ad9245fc994c8bc66577fbe8 778458988fa34607bafdeeed39116b93 - default default] [instance: 0289b07c-c4f8-4a84-9467-e79342f3b785] During detach_volume, instance disappeared.: nova.exception.InstanceNotFound: Instance 0289b07c-c4f8-4a84-9467-e79342f3b785 could not be found. ~~~
This does appear to be a Barbican configuration issue. Nova is simply surfacing the errors that Barbican is reporting (via Castellan). There is possibly a bug here in how Nova handles that error, but the first issue to resolve is the policy one. Are both Nova and Barbican using the standard policy configuration files? If not, can you share these?
In this scenario the default Barbican policy is (correctly) denying access to the secret because the credentials used to make the request are for a different project than the project that owns the secret. In this particular case it is because this deployment has an Operations team that uses its own Keystone project (tenant) for identity purposes. The use case is: > As a member of the operations team, I need to be able to migrate an end user's VM which requires access to a Barbican key owned by the end user when the VM has an encrypted volume. Although the default Barbican policy does not allow for this use case, it can be enabled by using a custom policy, for example: { "operations_team": "role:admin and project_id:<OPERATIONS_PROJECT_UUID>", "secret_meta:get": "rule:all_but_audit or rule:operations_team", "secret:decrypt": "rule:secret_decrypt_non_private_read or rule:secret_project_creator or rule:secret_project_admin or rule:secret_acl_read or rule:operations_team", "secret:get": "rule:secret_non_private_read or rule:secret_project_creator or rule:secret_project_admin or rule:secret_acl_read or rule:operations_team", "secrets:get": "rule:all_but_audit or rule:operations_team" } With this example policy, any user that is granted the "admin" role on the operations project with ID <OPERATIONS_PROJECT_UUID> will be able to retrieve any secret from Barbican regardless of which project owns the secret.