Bug 1772344 - [OSP17] Unable to boot an instance from an encrypted volume snapshot
Summary: [OSP17] Unable to boot an instance from an encrypted volume snapshot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 16.0 (Train)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: beta
: 17.0
Assignee: Lee Yarwood
QA Contact: James Parker
URL:
Whiteboard:
Depends On:
Blocks: 1803086
TreeView+ depends on / blocked
 
Reported: 2019-11-14 08:20 UTC by bkopilov
Modified: 2022-09-21 12:09 UTC (History)
9 users (show)

Fixed In Version: openstack-nova-23.0.3-0.20210908140341.e39bbdc.el9ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1803086 (view as bug list)
Environment:
Last Closed: 2022-09-21 12:07:58 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
compute-0 logs (21.94 KB, application/gzip)
2019-11-14 08:24 UTC, bkopilov 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 694497 0 'None' MERGED block_device: Copy original volume_type when missing for snapshot based volumes 2020-11-25 11:12:50 UTC
OpenStack gerrit 695549 0 'None' NEW Add boot from encrypted snapshot test 2020-11-25 11:12:52 UTC
Red Hat Issue Tracker OSP-10064 0 None None None 2022-04-13 20:04:45 UTC
Red Hat Product Errata RHEA-2022:6543 0 None None None 2022-09-21 12:09:15 UTC

Description bkopilov 2019-11-14 08:20:48 UTC 
Description of problem:
RHOS16 , ceph storage backend for nova , glance and cinder.


Steps to reproduce:
Create encrypted volume (LUKS).
Create a snapshot from the encrypted volume.
Boot an instance from snapshot - > Fails.

nova --os-compute-api-version 2.79  boot test  --snapshot 38b2206d-d416-4bc6-99ab-f1980129f648 --flavor test   --nic net-id=d63818db-fb01-4895-96fe-2a9b0023e640







(overcloud) [stack@undercloud-0 ~]$ cinder show 99500e5a-b377-4006-ab33-221bb6905a18
+--------------------------------+----------------------------------------------------------+
| Property                       | Value                                                    |
+--------------------------------+----------------------------------------------------------+
| attached_servers               | []                                                       |
| attachment_ids                 | []                                                       |
| availability_zone              | nova                                                     |
| bootable                       | true                                                     |
| consistencygroup_id            | None                                                     |
| created_at                     | 2019-11-14T08:02:48.000000                               |
| description                    | None                                                     |
| encrypted                      | True                                                     |
| id                             | 99500e5a-b377-4006-ab33-221bb6905a18                     |
| metadata                       |                                                          |
| migration_status               | None                                                     |
| multiattach                    | False                                                    |
| name                           | tempest-TestEncryptedCinderVolumes-volume-1667857809     |
| os-vol-host-attr:host          | hostgroup@tripleo_ceph#tripleo_ceph                      |
| os-vol-mig-status-attr:migstat | None                                                     |
| os-vol-mig-status-attr:name_id | None                                                     |
| os-vol-tenant-attr:tenant_id   | b9983a4424624d73b13bcece1b466405                         |
| replication_status             | None                                                     |
| size                           | 1                                                        |
| snapshot_id                    | None                                                     |
| source_volid                   | None                                                     |
| status                         | available                                                |
| updated_at                     | 2019-11-14T08:03:07.000000                               |
| user_id                        | 659d15f72d3145afa06c1406e0f18f4c                         |
| volume_image_metadata          | checksum : ba3cd24377dde5dfdd58728894004abb              |
|                                | container_format : bare                                  |
|                                | disk_format : raw                                        |
|                                | image_id : c02f4ac6-ba03-4390-bb90-b732115db3ab          |
|                                | image_name : cirros                                      |
|                                | min_disk : 0                                             |
|                                | min_ram : 0                                              |
|                                | signature_verified : False                               |
|                                | size : 46137344                                          |
| volume_type                    | tempest-volume-typeTestEncryptedCinderVolumes-1226376028 |
+--------------------------------+----------------------------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-show 38b2206d-d416-4bc6-99ab-f1980129f648
+--------------------------------------------+-------------------------------------------------------+
| Property                                   | Value                                                 |
+--------------------------------------------+-------------------------------------------------------+
| created_at                                 | 2019-11-14T08:03:07.000000                            |
| description                                | None                                                  |
| id                                         | 38b2206d-d416-4bc6-99ab-f1980129f648                  |
| metadata                                   | {}                                                    |
| name                                       | tempest-TestEncryptedCinderVolumes-snapshot-551989603 |
| os-extended-snapshot-attributes:progress   | 100%                                                  |
| os-extended-snapshot-attributes:project_id | b9983a4424624d73b13bcece1b466405                      |
| size                                       | 1                                                     |
| status                                     | available                                             |
| updated_at                                 | 2019-11-14T08:03:08.000000                            |
| volume_id                                  | 99500e5a-b377-4006-ab33-221bb6905a18                  |
+--------------------------------------------+-------------------------------------------------------+
Here is the traceback:
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [req-57a53c0d-44ae-4c07-a170-b3b2b3010aaa 24bb32b71eaa42d4a3d3044a3d3ce728 346ae2925a2c4fcd8fb162feedc9b8e7 - default default] [instance: 78a9a850-540b-4922-84a4-d26190d298b8] Instance failed block device setup: nova.exception.InvalidInput: Invalid input received: Invalid input received: Invalid volume_type provided: 2cb26675-ad55-459d-8be5-184fd0fe24f9 (requested type is not compatible; recommend omitting the type argument). (HTTP 400) (Request-ID: req-619530c5-8767-4449-bc5b-9a7ee20f080f)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8] Traceback (most recent call last):
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 397, in wrapper
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     res = method(self, ctx, *args, **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 419, in wrapper
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     res = method(self, ctx, size, *args, **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 661, in create
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     item = client.volumes.create(size, **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/v3/volumes.py", line 125, in create
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     return self._create('/volumes', body, 'volume')
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/base.py", line 302, in _create
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     resp, body = self.api.client.post(url, body=body)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 217, in post
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     return self._cs_request(url, 'POST', **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 205, in _cs_request
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     return self.request(url, method, **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 191, in request
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     raise exceptions.from_response(resp, body)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8] cinderclient.exceptions.BadRequest: Invalid input received: Invalid volume_type provided: 2cb26675-ad55-459d-8be5-184fd0fe24f9 (requested type is not compatible; recommend omitting the type argument). (HTTP 400) (Request-ID: req-619530c5-8767-4449-bc5b-9a7ee20f080f)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8] 
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8] During handling of the above exception, another exception occurred:
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8] 
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8] Traceback (most recent call last):
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 1887, in _prep_block_device
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     wait_func=self._await_block_device_map_created)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 875, in attach_block_devices
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     _log_and_attach(device)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 872, in _log_and_attach
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     bdm.attach(*attach_args, **attach_kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 740, in attach
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     wait_func=wait_func, snapshot=snapshot)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 364, in _create_volume
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     availability_zone=av_zone, **create_kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 405, in wrapper
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     _reraise(exception.InvalidInput(reason=err_msg))
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 483, in _reraise
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     six.reraise(type(desired_exc), desired_exc, sys.exc_info()[2])
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/six.py", line 692, in reraise
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     raise value.with_traceback(tb)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 397, in wrapper
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     res = method(self, ctx, *args, **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 419, in wrapper
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     res = method(self, ctx, size, *args, **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 661, in create
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     item = client.volumes.create(size, **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/v3/volumes.py", line 125, in create
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     return self._create('/volumes', body, 'volume')
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/base.py", line 302, in _create
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     resp, body = self.api.client.post(url, body=body)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 217, in post
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     return self._cs_request(url, 'POST', **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 205, in _cs_request
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     return self.request(url, method, **kwargs)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]   File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 191, in request
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8]     raise exceptions.from_response(resp, body)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8] nova.exception.InvalidInput: Invalid input received: Invalid input received: Invalid volume_type provided: 2cb26675-ad55-459d-8be5-184fd0fe24f9 (requested type is not compatible; recommend omitting the type argument). (HTTP 400) (Request-ID: req-619530c5-8767-4449-bc5b-9a7ee20f080f)
2019-11-14 08:10:15.590 7 ERROR nova.compute.manager [instance: 78a9a850-540b-4922-84a4-d26190d298b8] 


Attaching logs.

Thanks,
Benny
Comment 1 bkopilov 2019-11-14 08:24:23 UTC 
Created attachment 1636039 [details]
compute-0 logs
Comment 2 Lee Yarwood 2019-11-15 09:17:13 UTC 
Can we get c-api logs to see why it returned the 400?
Comment 3 Lee Yarwood 2019-11-15 09:52:47 UTC 
(In reply to Lee Yarwood from comment #2)
> Can we get c-api logs to see why it returned the 400?

Actually ignore that request. There's enough in the n-cpu trace to see that c-api is hitting this:

https://github.com/openstack/cinder/blob/5ad452fcd403554cce784c2ce03368dc764002a6/cinder/volume/api.py#L269-L278

Both python-novaclient and n-cpu don't appear to be copying the volume_type from the original volume causing this failure.

There's an obvious workaround of creating the volume directly with cinder but we should probably try to handle this either in the client and/or in nova/virt/block_device.py.
Comment 12 errata-xmlrpc 2022-09-21 12:07:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543
Note You need to log in before you can comment on or make changes to this bug.