Created attachment 1242029 [details] sos report from my test environment with the debug logs Description of problem: nova boot fails while using bootable cinder volume as a backend when that cinder volume is created from the snapshot of an instance Version-Release number of selected component (if applicable): [root@dell-per320-1 ~]# rpm -qa | grep cinder python-cinder-8.1.1-4.el7ost.noarch openstack-cinder-8.1.1-4.el7ost.noarch python-cinderclient-1.6.0-1.el7ost.noarch [root@dell-per320-1 ~]# rpm -qa | grep glance python-glanceclient-2.0.0-1.el7ost.noarch python-glance-12.0.0-2.el7ost.noarch python-glance-store-0.13.1-2.el7ost.noarch openstack-glance-12.0.0-2.el7ost.noarch How reproducible: Always Steps to Reproduce: 1. Set glance_api_version = 1 in /etc/cinder/cinder.conf and restart the cinder services [root@dell-per320-1 cinder(keystone_admin)]# grep glance_api_version /etc/cinder/cinder.conf glance_api_version = 1 2. Create a normal glance image root@dell-per320-1 cinder(keystone_admin)]# glance image-list +--------------------------------------+------------+ | ID | Name | +--------------------------------------+------------+ | bf956b1b-6eac-41d8-89b1-df49da099a75 | cirros | | abb1f188-f0b5-4464-baea-1ac15e5c6ac7 | inst1-snap | +--------------------------------------+------------+ 3. Create a bootable cinder volume from the glance image [root@dell-per320-1 cinder(keystone_admin)]# cinder create --image cirros --display-name vol3 3 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2017-01-17T23:52:25.000000 | | description | None | | encrypted | False | | id | 3f2a2ed9-63d1-4735-8f38-96d4e9f67930 | | metadata | {} | | migration_status | None | | multiattach | False | | name | vol3 | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | feb4db429eb748d9b16333c2cf2f9147 | | replication_status | disabled | | size | 3 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | None | | user_id | 5f3228ca573d484aa65ca58daf7ae533 | | volume_type | None | +--------------------------------+--------------------------------------+ [root@dell-per320-1 cinder(keystone_admin)]# cinder list +--------------------------------------+-------------+------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-------------+------+------+-------------+----------+--------------------------------------+ | 0999f502-d02a-4d22-9832-ab90af4332fa | error | vol2 | 3 | - | false | | | 3f2a2ed9-63d1-4735-8f38-96d4e9f67930 | available | vol3 | 3 | - | true | | | 6ca596d9-d3a3-4ff8-afdd-dd364cd0b41e | in-use | vol1 | 3 | - | true | 01614fb9-6348-41d0-b037-963968121c43 | +--------------------------------------+-------------+------+------+-------------+----------+--------------------------------------+ 4.Boot a nova instance from the cinder volume [root@dell-per320-1 cinder(keystone_admin)]# nova boot --poll --flavor m1.tiny --nic net-name=private --block-device-mapping vda=3f2a2ed9-63d1-4735-8f38-96d4e9f67930:::0 inst3 +--------------------------------------+--------------------------------------------------+ | Property | Value | +--------------------------------------+--------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | instance-00000002 | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | P5pVVsoFMDXs | | config_drive | | | created | 2017-01-17T23:53:55Z | | flavor | m1.tiny (1) | | hostId | | | id | 6174c48e-253d-4f60-960c-eee912ad3e53 | | image | Attempt to boot from volume - no image supplied | | key_name | - | | metadata | {} | | name | inst3 | | os-extended-volumes:volumes_attached | [{"id": "3f2a2ed9-63d1-4735-8f38-96d4e9f67930"}] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | feb4db429eb748d9b16333c2cf2f9147 | | updated | 2017-01-17T23:53:55Z | | user_id | 5f3228ca573d484aa65ca58daf7ae533 | +--------------------------------------+--------------------------------------------------+ Server building... 100% complete Finished [root@dell-per320-1 cinder(keystone_admin)]# nova list +--------------------------------------+-------+---------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+---------+------------+-------------+------------------+ | 01614fb9-6348-41d0-b037-963968121c43 | inst1 | SHUTOFF | - | Shutdown | private=10.0.0.3 | | 6174c48e-253d-4f60-960c-eee912ad3e53 | inst3 | ACTIVE | - | Running | private=10.0.0.4 | +--------------------------------------+-------+---------+------------+-------------+------------------+ 5. Stop the running instance and take a snapshot of it [root@dell-per320-1 cinder(keystone_admin)]# nova stop inst3 Request to stop server inst3 has been accepted. [root@dell-per320-1 cinder(keystone_admin)]# nova list +--------------------------------------+-------+---------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+---------+------------+-------------+------------------+ | 01614fb9-6348-41d0-b037-963968121c43 | inst1 | SHUTOFF | - | Shutdown | private=10.0.0.3 | | 6174c48e-253d-4f60-960c-eee912ad3e53 | inst3 | SHUTOFF | - | Shutdown | private=10.0.0.4 | +--------------------------------------+-------+---------+------------+-------------+------------------+ [root@dell-per320-1 cinder(keystone_admin)]# nova image-create inst3 inst3-snap [root@dell-per320-1 cinder(keystone_admin)]# glance image-list +--------------------------------------+------------+ | ID | Name | +--------------------------------------+------------+ | bf956b1b-6eac-41d8-89b1-df49da099a75 | cirros | | abb1f188-f0b5-4464-baea-1ac15e5c6ac7 | inst1-snap | | ea4ce365-c8b6-4911-9c8f-860b119bbfb2 | inst3-snap | +--------------------------------------+------------+ 6. Now create a cinder volume from the newly created snapshot [root@dell-per320-1 cinder(keystone_admin)]# cinder create --image inst3-snap --display-name vol4 3 +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2017-01-17T23:56:12.000000 | | description | None | | encrypted | False | | id | c9e07690-fa38-4eca-97b4-9be66edf0c2d | | metadata | {} | | migration_status | None | | multiattach | False | | name | vol4 | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | feb4db429eb748d9b16333c2cf2f9147 | | replication_status | disabled | | size | 3 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | None | | user_id | 5f3228ca573d484aa65ca58daf7ae533 | | volume_type | None | +--------------------------------+--------------------------------------+ [root@dell-per320-1 cinder(keystone_admin)]# cinder list +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ | 0999f502-d02a-4d22-9832-ab90af4332fa | error | vol2 | 3 | - | false | | | 3f2a2ed9-63d1-4735-8f38-96d4e9f67930 | in-use | vol3 | 3 | - | true | 6174c48e-253d-4f60-960c-eee912ad3e53 | | 6ca596d9-d3a3-4ff8-afdd-dd364cd0b41e | in-use | vol1 | 3 | - | true | 01614fb9-6348-41d0-b037-963968121c43 | | c9e07690-fa38-4eca-97b4-9be66edf0c2d | available | vol4 | 3 | - | true | | +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ 7. Now try to boot a nova instance from the newly created volume: [root@dell-per320-1 cinder(keystone_admin)]# nova boot --poll --flavor m1.tiny --nic net-name=private --block-device-mapping vda=c9e07690-fa38-4eca-97b4-9be66edf0c2d:::0 inst4 Actual results: nova boot command fails with the below error: ERROR (BadRequest): Invalid image metadata. Error: A list is required in field img_block_device_mapping, not a unicode (HTTP 400) (Request-ID: req-3b2254ca-64cd-476a-8000-8c8ee0c02404) Expected results: nova boot command should complete successfully without any errors. Additional info: This issue is also seen on the following package versions: # rpm -qa | grep cinder | sort openstack-cinder-8.1.1-2.el7ost.noarch python-cinder-8.1.1-2.el7ost.noarch python-cinderclient-1.6.0-1.el7ost.noarch # rpm -qa | grep glance | sort openstack-glance-12.0.0-1.el7ost.noarch python-glance-12.0.0-1.el7ost.noarch python-glanceclient-2.0.0-1.el7ost.noarch python-glance-store-0.13.1-2.el7ost.noarch
Thanks for the report, I've not gone through the cinder code in any detail yet but I wanted to quickly comment that this doesn't reproduce if you use nova directly to create the volume from the snapshot image : # nova boot --flavor 1 --block-device id=1d237a89-066a-44ef-94f9-d1e9bcd3f62b,source=image,dest=volume,size=1,bootindex=0 test-boot-from-volume # nova stop test-boot-from-volume # nova image-create test-boot-from-volume test-boot-from-volume-snapshot # nova boot --flavor 1 --block-device id=dd47897b-6c33-434a-b337-aacfa5540044,source=image,dest=volume,size=1,bootindex=0 test-boot-from-snapshot
Moving to openstack-cinder, the issue here is with Cinder and the way it is storing and then returning image metadata associated with a volume created from a snapshot image. When using the v1 glance API to create a volume from an image snapshot it stores block_device_mapping in the following DB table : MariaDB [cinder]> describe volume_glance_metadata; +-------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------+----------------+ | created_at | datetime | YES | | NULL | | | updated_at | datetime | YES | | NULL | | | deleted_at | datetime | YES | | NULL | | | deleted | tinyint(1) | YES | | NULL | | | id | int(11) | NO | PRI | NULL | auto_increment | | volume_id | varchar(36) | YES | MUL | NULL | | | snapshot_id | varchar(36) | YES | MUL | NULL | | | key | varchar(255) | YES | | NULL | | | value | text | YES | | NULL | | +-------------+--------------+------+-----+---------+----------------+ 9 rows in set (0.00 sec) When Nova then queries Cinder about this volume any image metadata referring to it is then added onto the detailed response by the VolumeImageMetadataController in the API. 2017-01-20 07:59:50.702 16419 DEBUG keystoneauth.session [req-c540ec39-7946-4764-930b-dfc04cb1d742 b1604e269ff048bd9fd04997e93a8a2f 62d753fb4c054171a5012be67e9b298a - - -] REQ: curl -g -i -X GET http://192.168.122.89:8776/v2/62d753fb4c054171a5012be67e9b298a/volumes/9cf33d1b-77d1-4fd6-a604-5eba67abf2dc -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}171e42113ff5b4d94c2e14505a77a1a36be5fbeb" _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:248 2017-01-20 07:59:51.194 16419 DEBUG keystoneauth.session [req-c540ec39-7946-4764-930b-dfc04cb1d742 b1604e269ff048bd9fd04997e93a8a2f 62d753fb4c054171a5012be67e9b298a - - -] RESP: [200] X-Compute-Request-Id: req-8f47a68c-a219-4679-a7e7-ffb21256652b Content-Type: application/json Content-Length: 1632 X-Openstack-Request-Id: req-8f47a68c-a219-4679-a7e7-ffb21256652b Date: Fri, 20 Jan 2017 12:59:51 GMT Connection: keep-alive RESP BODY: {"volume": {"migration_status": null, "attachments": [], "links": [{"href": "http://192.168.122.89:8776/v2/62d753fb4c054171a5012be67e9b298a/volumes/9cf33d1b-77d1-4fd6-a604-5eba67abf2dc", "rel": "self"}, {"href": "http://192.168.122.89:8776/62d753fb4c054171a5012be67e9b298a/volumes/9cf33d1b-77d1-4fd6-a604-5eba67abf2dc", "rel": "bookmark"}], "availability_zone": "nova", "os-vol-host-attr:host": "packstack.example.com@lvm#lvm", "encrypted": false, "updated_at": "2017-01-19T20:18:20.000000", "replication_status": "disabled", "snapshot_id": null, "id": "9cf33d1b-77d1-4fd6-a604-5eba67abf2dc", "size": 1, "user_id": "b1604e269ff048bd9fd04997e93a8a2f", "os-vol-tenant-attr:tenant_id": "62d753fb4c054171a5012be67e9b298a", "os-vol-mig-status-attr:migstat": null, "metadata": {}, "status": "available", "volume_image_metadata": {"block_device_mapping": "[{u'guest_format': None, u'boot_index': 0, u'no_device': None, u'volume_id': None, u'volume_size': 1, u'device_name': u'/dev/vda', u'disk_bus': u'virtio', u'image_id': None, u'source_type': u'snapshot', u'device_type': u'disk', u'snapshot_id': u'423edd63-ec11-4e83-86f6-4c27d8607781', u'destination_type': u'volume', u'delete_on_termination': False}]", "min_ram": "0", "image_name": "test-boot-from-volume-snapshot", "bdm_v2": "True", "image_id": "03f19e1e-59ad-44bc-9df6-2c1df1191678", "root_device_name": "/dev/vda", "min_disk": "1", "size": "0"}, "description": null, "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": null, "bootable": "true", "created_at": "2017-01-19T20:18:18.000000", "volume_type": null}} As this is just taking the value directly from the DB we end up with a string value for block_device_mapping causing Nova to bork when attempting to create an ImageMeta object. The following patch corrects this in my test OSP 9 environment by converting the string back into valid JSON before it is passed over to Nova : # diff -u /usr/lib/python2.7/site-packages/cinder/volume/api.py.orig /usr/lib/python2.7/site-packages/cinder/volume/api.py --- /usr/lib/python2.7/site-packages/cinder/volume/api.py.orig 2017-01-20 08:39:21.121277998 -0500 +++ /usr/lib/python2.7/site-packages/cinder/volume/api.py 2017-01-20 08:42:34.147966327 -0500 @@ -23,6 +23,7 @@ from oslo_config import cfg from oslo_log import log as logging +from oslo_serialization import jsonutils from oslo_utils import excutils from oslo_utils import strutils from oslo_utils import timeutils @@ -1134,6 +1135,8 @@ volume_id_list) results = collections.defaultdict(dict) for meta_entry in db_data: + if meta_entry['key'] == 'block_device_mapping': + meta_entry['value'] = jsonutils.loads(meta_entry['value']) results[meta_entry['volume_id']].update({meta_entry['key']: meta_entry['value']}) return results I'll retain ownership for now and see if there's any appetite for this upstream even with the Glance V1 API now being deprecated.
This no longer reproduces upstream as Cinder doesn't stash the block_device_mapping from the snapshot during the creation of the volume. As we have a valid workaround here of using Nova to orchestrate the creation of this volume from the snapshot I'm going to close this out as WONTFIX. Please feel free to reopen (adding a needinfo against me) if there is a valid use case that I'm missing here where the use of Nova isn't acceptable to workaround this.
Seems this has to be reopened. If you create a new server from a snapshot of a volume, and then create an image from this server, you can not create a new server from this image, because block_device_mapping in the image metadata is a string. If I try to create a server from such an image, I get the following error: "Invalid image metadata. Error: A list is required in field img_block_device_mapping, not a str", because block_device_mapping is a string and not a list (missing JSON decoding). If you need further information, just contact me.