Bug 1414226 - nova boot fails while using bootable cinder volume as a backend when that cinder volume is created from the snapshot of an instance
Summary: nova boot fails while using bootable cinder volume as a backend when that cin...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 9.0 (Mitaka)
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: ---
: ---
Assignee: Lee Yarwood
QA Contact: Tzach Shefi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-18 05:47 UTC by Punit Kundal
Modified: 2020-06-23 12:21 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-25 12:15:54 UTC
Target Upstream Version:
tshefi: automate_bug-


Attachments (Terms of Use)
sos report from my test environment with the debug logs (14.04 MB, application/x-xz)
2017-01-18 05:47 UTC, Punit Kundal
no flags Details

Description Punit Kundal 2017-01-18 05:47:19 UTC
Created attachment 1242029 [details]
sos report from my test environment with the debug logs

Description of problem:

nova boot fails while using bootable cinder volume as a backend when that cinder volume is created from the snapshot of an instance

Version-Release number of selected component (if applicable):

[root@dell-per320-1 ~]# rpm -qa | grep cinder
python-cinder-8.1.1-4.el7ost.noarch
openstack-cinder-8.1.1-4.el7ost.noarch
python-cinderclient-1.6.0-1.el7ost.noarch

[root@dell-per320-1 ~]# rpm -qa | grep glance 
python-glanceclient-2.0.0-1.el7ost.noarch
python-glance-12.0.0-2.el7ost.noarch
python-glance-store-0.13.1-2.el7ost.noarch
openstack-glance-12.0.0-2.el7ost.noarch


How reproducible:
Always

Steps to Reproduce:
1. Set glance_api_version = 1 in /etc/cinder/cinder.conf and restart the cinder services

[root@dell-per320-1 cinder(keystone_admin)]# grep glance_api_version /etc/cinder/cinder.conf 
glance_api_version = 1

2. Create a normal glance image

root@dell-per320-1 cinder(keystone_admin)]# glance image-list
+--------------------------------------+------------+
| ID                                   | Name       |
+--------------------------------------+------------+
| bf956b1b-6eac-41d8-89b1-df49da099a75 | cirros     |
| abb1f188-f0b5-4464-baea-1ac15e5c6ac7 | inst1-snap |
+--------------------------------------+------------+

3. Create a bootable cinder volume from the glance image

[root@dell-per320-1 cinder(keystone_admin)]# cinder create --image cirros --display-name vol3 3
+--------------------------------+--------------------------------------+
|            Property            |                Value                 |
+--------------------------------+--------------------------------------+
|          attachments           |                  []                  |
|       availability_zone        |                 nova                 |
|            bootable            |                false                 |
|      consistencygroup_id       |                 None                 |
|           created_at           |      2017-01-17T23:52:25.000000      |
|          description           |                 None                 |
|           encrypted            |                False                 |
|               id               | 3f2a2ed9-63d1-4735-8f38-96d4e9f67930 |
|            metadata            |                  {}                  |
|        migration_status        |                 None                 |
|          multiattach           |                False                 |
|              name              |                 vol3                 |
|     os-vol-host-attr:host      |                 None                 |
| os-vol-mig-status-attr:migstat |                 None                 |
| os-vol-mig-status-attr:name_id |                 None                 |
|  os-vol-tenant-attr:tenant_id  |   feb4db429eb748d9b16333c2cf2f9147   |
|       replication_status       |               disabled               |
|              size              |                  3                   |
|          snapshot_id           |                 None                 |
|          source_volid          |                 None                 |
|             status             |               creating               |
|           updated_at           |                 None                 |
|            user_id             |   5f3228ca573d484aa65ca58daf7ae533   |
|          volume_type           |                 None                 |
+--------------------------------+--------------------------------------+

[root@dell-per320-1 cinder(keystone_admin)]# cinder list
+--------------------------------------+-------------+------+------+-------------+----------+--------------------------------------+
|                  ID                  |    Status   | Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-------------+------+------+-------------+----------+--------------------------------------+
| 0999f502-d02a-4d22-9832-ab90af4332fa |    error    | vol2 |  3   |      -      |  false   |                                      |
| 3f2a2ed9-63d1-4735-8f38-96d4e9f67930 | available | vol3 |  3   |      -      |   true   |                                      |
| 6ca596d9-d3a3-4ff8-afdd-dd364cd0b41e |    in-use   | vol1 |  3   |      -      |   true   | 01614fb9-6348-41d0-b037-963968121c43 |
+--------------------------------------+-------------+------+------+-------------+----------+--------------------------------------+

4.Boot a nova instance from the cinder volume

[root@dell-per320-1 cinder(keystone_admin)]# nova boot --poll --flavor m1.tiny --nic net-name=private --block-device-mapping vda=3f2a2ed9-63d1-4735-8f38-96d4e9f67930:::0 inst3
+--------------------------------------+--------------------------------------------------+
| Property                             | Value                                            |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          |                                                  |
| OS-EXT-SRV-ATTR:host                 | -                                                |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000002                                |
| OS-EXT-STS:power_state               | 0                                                |
| OS-EXT-STS:task_state                | scheduling                                       |
| OS-EXT-STS:vm_state                  | building                                         |
| OS-SRV-USG:launched_at               | -                                                |
| OS-SRV-USG:terminated_at             | -                                                |
| accessIPv4                           |                                                  |
| accessIPv6                           |                                                  |
| adminPass                            | P5pVVsoFMDXs                                     |
| config_drive                         |                                                  |
| created                              | 2017-01-17T23:53:55Z                             |
| flavor                               | m1.tiny (1)                                      |
| hostId                               |                                                  |
| id                                   | 6174c48e-253d-4f60-960c-eee912ad3e53             |
| image                                | Attempt to boot from volume - no image supplied  |
| key_name                             | -                                                |
| metadata                             | {}                                               |
| name                                 | inst3                                            |
| os-extended-volumes:volumes_attached | [{"id": "3f2a2ed9-63d1-4735-8f38-96d4e9f67930"}] |
| progress                             | 0                                                |
| security_groups                      | default                                          |
| status                               | BUILD                                            |
| tenant_id                            | feb4db429eb748d9b16333c2cf2f9147                 |
| updated                              | 2017-01-17T23:53:55Z                             |
| user_id                              | 5f3228ca573d484aa65ca58daf7ae533                 |
+--------------------------------------+--------------------------------------------------+

Server building... 100% complete
Finished
[root@dell-per320-1 cinder(keystone_admin)]# nova list
+--------------------------------------+-------+---------+------------+-------------+------------------+
| ID                                   | Name  | Status  | Task State | Power State | Networks         |
+--------------------------------------+-------+---------+------------+-------------+------------------+
| 01614fb9-6348-41d0-b037-963968121c43 | inst1 | SHUTOFF | -          | Shutdown    | private=10.0.0.3 |
| 6174c48e-253d-4f60-960c-eee912ad3e53 | inst3 | ACTIVE  | -          | Running     | private=10.0.0.4 |
+--------------------------------------+-------+---------+------------+-------------+------------------+

5. Stop the running instance and take a snapshot of it

[root@dell-per320-1 cinder(keystone_admin)]# nova stop inst3
Request to stop server inst3 has been accepted.

[root@dell-per320-1 cinder(keystone_admin)]# nova list
+--------------------------------------+-------+---------+------------+-------------+------------------+
| ID                                   | Name  | Status  | Task State | Power State | Networks         |
+--------------------------------------+-------+---------+------------+-------------+------------------+
| 01614fb9-6348-41d0-b037-963968121c43 | inst1 | SHUTOFF | -          | Shutdown    | private=10.0.0.3 |
| 6174c48e-253d-4f60-960c-eee912ad3e53 | inst3 | SHUTOFF | -          | Shutdown    | private=10.0.0.4 |
+--------------------------------------+-------+---------+------------+-------------+------------------+
[root@dell-per320-1 cinder(keystone_admin)]# nova image-create inst3 inst3-snap

[root@dell-per320-1 cinder(keystone_admin)]# glance image-list
+--------------------------------------+------------+
| ID                                   | Name       |
+--------------------------------------+------------+
| bf956b1b-6eac-41d8-89b1-df49da099a75 | cirros     |
| abb1f188-f0b5-4464-baea-1ac15e5c6ac7 | inst1-snap |
| ea4ce365-c8b6-4911-9c8f-860b119bbfb2 | inst3-snap |
+--------------------------------------+------------+ 

6. Now create a cinder volume from the newly created snapshot

[root@dell-per320-1 cinder(keystone_admin)]# cinder create --image inst3-snap --display-name vol4 3
+--------------------------------+--------------------------------------+
|            Property            |                Value                 |
+--------------------------------+--------------------------------------+
|          attachments           |                  []                  |
|       availability_zone        |                 nova                 |
|            bootable            |                false                 |
|      consistencygroup_id       |                 None                 |
|           created_at           |      2017-01-17T23:56:12.000000      |
|          description           |                 None                 |
|           encrypted            |                False                 |
|               id               | c9e07690-fa38-4eca-97b4-9be66edf0c2d |
|            metadata            |                  {}                  |
|        migration_status        |                 None                 |
|          multiattach           |                False                 |
|              name              |                 vol4                 |
|     os-vol-host-attr:host      |                 None                 |
| os-vol-mig-status-attr:migstat |                 None                 |
| os-vol-mig-status-attr:name_id |                 None                 |
|  os-vol-tenant-attr:tenant_id  |   feb4db429eb748d9b16333c2cf2f9147   |
|       replication_status       |               disabled               |
|              size              |                  3                   |
|          snapshot_id           |                 None                 |
|          source_volid          |                 None                 |
|             status             |               creating               |
|           updated_at           |                 None                 |
|            user_id             |   5f3228ca573d484aa65ca58daf7ae533   |
|          volume_type           |                 None                 |
+--------------------------------+--------------------------------------+


[root@dell-per320-1 cinder(keystone_admin)]# cinder list
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  | Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
| 0999f502-d02a-4d22-9832-ab90af4332fa |   error   | vol2 |  3   |      -      |  false   |                                      |
| 3f2a2ed9-63d1-4735-8f38-96d4e9f67930 |   in-use  | vol3 |  3   |      -      |   true   | 6174c48e-253d-4f60-960c-eee912ad3e53 |
| 6ca596d9-d3a3-4ff8-afdd-dd364cd0b41e |   in-use  | vol1 |  3   |      -      |   true   | 01614fb9-6348-41d0-b037-963968121c43 |
| c9e07690-fa38-4eca-97b4-9be66edf0c2d | available | vol4 |  3   |      -      |   true   |                                      |
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+

7. Now try to boot a nova instance from the newly created volume:

[root@dell-per320-1 cinder(keystone_admin)]# nova boot --poll --flavor m1.tiny --nic net-name=private --block-device-mapping vda=c9e07690-fa38-4eca-97b4-9be66edf0c2d:::0 inst4

Actual results:

nova boot command fails with the below error:

ERROR (BadRequest): Invalid image metadata. Error: A list is required in field img_block_device_mapping, not a unicode (HTTP 400) (Request-ID: req-3b2254ca-64cd-476a-8000-8c8ee0c02404)

Expected results:

nova boot command should complete successfully without any errors.

Additional info:
This issue is also seen on the following package versions:

# rpm -qa | grep cinder | sort
openstack-cinder-8.1.1-2.el7ost.noarch
python-cinder-8.1.1-2.el7ost.noarch
python-cinderclient-1.6.0-1.el7ost.noarch

# rpm -qa | grep glance | sort
openstack-glance-12.0.0-1.el7ost.noarch
python-glance-12.0.0-1.el7ost.noarch
python-glanceclient-2.0.0-1.el7ost.noarch
python-glance-store-0.13.1-2.el7ost.noarch

Comment 1 Lee Yarwood 2017-01-19 20:27:13 UTC
Thanks for the report, I've not gone through the cinder code in any detail yet but I wanted to quickly comment that this doesn't reproduce if you use nova directly to create the volume from the snapshot image :

# nova boot --flavor 1 --block-device id=1d237a89-066a-44ef-94f9-d1e9bcd3f62b,source=image,dest=volume,size=1,bootindex=0 test-boot-from-volume
# nova stop test-boot-from-volume
# nova image-create test-boot-from-volume test-boot-from-volume-snapshot
# nova boot --flavor 1 --block-device id=dd47897b-6c33-434a-b337-aacfa5540044,source=image,dest=volume,size=1,bootindex=0 test-boot-from-snapshot

Comment 2 Lee Yarwood 2017-01-20 14:57:37 UTC
Moving to openstack-cinder, the issue here is with Cinder and the way it is storing and then returning image metadata associated with a volume created from a snapshot image. 

When using the v1 glance API to create a volume from an image snapshot it stores block_device_mapping in the following DB table :

MariaDB [cinder]> describe volume_glance_metadata;
+-------------+--------------+------+-----+---------+----------------+
| Field       | Type         | Null | Key | Default | Extra          |
+-------------+--------------+------+-----+---------+----------------+
| created_at  | datetime     | YES  |     | NULL    |                |
| updated_at  | datetime     | YES  |     | NULL    |                |
| deleted_at  | datetime     | YES  |     | NULL    |                |
| deleted     | tinyint(1)   | YES  |     | NULL    |                |
| id          | int(11)      | NO   | PRI | NULL    | auto_increment |
| volume_id   | varchar(36)  | YES  | MUL | NULL    |                |
| snapshot_id | varchar(36)  | YES  | MUL | NULL    |                |
| key         | varchar(255) | YES  |     | NULL    |                |
| value       | text         | YES  |     | NULL    |                |
+-------------+--------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)

When Nova then queries Cinder about this volume any image metadata referring to it is then added onto the detailed response by the VolumeImageMetadataController in the API.

2017-01-20 07:59:50.702 16419 DEBUG keystoneauth.session [req-c540ec39-7946-4764-930b-dfc04cb1d742 b1604e269ff048bd9fd04997e93a8a2f 62d753fb4c054171a5012be67e9b298a - - -] REQ: curl -g -i -X GET http://192.168.122.89:8776/v2/62d753fb4c054171a5012be67e9b298a/volumes/9cf33d1b-77d1-4fd6-a604-5eba67abf2dc -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}171e42113ff5b4d94c2e14505a77a1a36be5fbeb" _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:248
2017-01-20 07:59:51.194 16419 DEBUG keystoneauth.session [req-c540ec39-7946-4764-930b-dfc04cb1d742 b1604e269ff048bd9fd04997e93a8a2f 62d753fb4c054171a5012be67e9b298a - - -] RESP: [200] X-Compute-Request-Id: req-8f47a68c-a219-4679-a7e7-ffb21256652b Content-Type: application/json Content-Length: 1632 X-Openstack-Request-Id: req-8f47a68c-a219-4679-a7e7-ffb21256652b Date: Fri, 20 Jan 2017 12:59:51 GMT Connection: keep-alive
RESP BODY: {"volume": {"migration_status": null, "attachments": [], "links": [{"href": "http://192.168.122.89:8776/v2/62d753fb4c054171a5012be67e9b298a/volumes/9cf33d1b-77d1-4fd6-a604-5eba67abf2dc", "rel": "self"}, {"href": "http://192.168.122.89:8776/62d753fb4c054171a5012be67e9b298a/volumes/9cf33d1b-77d1-4fd6-a604-5eba67abf2dc", "rel": "bookmark"}], "availability_zone": "nova", "os-vol-host-attr:host": "packstack.example.com@lvm#lvm", "encrypted": false, "updated_at": "2017-01-19T20:18:20.000000", "replication_status": "disabled", "snapshot_id": null, "id": "9cf33d1b-77d1-4fd6-a604-5eba67abf2dc", "size": 1, "user_id": "b1604e269ff048bd9fd04997e93a8a2f", "os-vol-tenant-attr:tenant_id": "62d753fb4c054171a5012be67e9b298a", "os-vol-mig-status-attr:migstat": null, "metadata": {}, "status": "available", "volume_image_metadata": {"block_device_mapping": "[{u'guest_format': None, u'boot_index': 0, u'no_device': None, u'volume_id': None, u'volume_size': 1, u'device_name': u'/dev/vda', u'disk_bus': u'virtio', u'image_id': None, u'source_type': u'snapshot', u'device_type': u'disk', u'snapshot_id': u'423edd63-ec11-4e83-86f6-4c27d8607781', u'destination_type': u'volume', u'delete_on_termination': False}]", "min_ram": "0", "image_name": "test-boot-from-volume-snapshot", "bdm_v2": "True", "image_id": "03f19e1e-59ad-44bc-9df6-2c1df1191678", "root_device_name": "/dev/vda", "min_disk": "1", "size": "0"}, "description": null, "multiattach": false, "source_volid": null, "consistencygroup_id": null, "os-vol-mig-status-attr:name_id": null, "name": null, "bootable": "true", "created_at": "2017-01-19T20:18:18.000000", "volume_type": null}}

As this is just taking the value directly from the DB we end up with a string value for block_device_mapping causing Nova to bork when attempting to create an ImageMeta object.

The following patch corrects this in my test OSP 9 environment by converting the string back into valid JSON before it is passed over to Nova :

# diff -u /usr/lib/python2.7/site-packages/cinder/volume/api.py.orig /usr/lib/python2.7/site-packages/cinder/volume/api.py                                                                                                                    
--- /usr/lib/python2.7/site-packages/cinder/volume/api.py.orig  2017-01-20 08:39:21.121277998 -0500
+++ /usr/lib/python2.7/site-packages/cinder/volume/api.py       2017-01-20 08:42:34.147966327 -0500
@@ -23,6 +23,7 @@
 
 from oslo_config import cfg
 from oslo_log import log as logging
+from oslo_serialization import jsonutils 
 from oslo_utils import excutils
 from oslo_utils import strutils
 from oslo_utils import timeutils
@@ -1134,6 +1135,8 @@
                                                           volume_id_list)
         results = collections.defaultdict(dict)
         for meta_entry in db_data:
+           if meta_entry['key'] == 'block_device_mapping':                         
+                meta_entry['value'] = jsonutils.loads(meta_entry['value'])
             results[meta_entry['volume_id']].update({meta_entry['key']:
                                                      meta_entry['value']})
         return results

I'll retain ownership for now and see if there's any appetite for this upstream even with the Glance V1 API now being deprecated.

Comment 3 Lee Yarwood 2017-01-25 12:15:54 UTC
This no longer reproduces upstream as Cinder doesn't stash the block_device_mapping from the snapshot during the creation of the volume. As we have a valid workaround here of using Nova to orchestrate the creation of this volume from the snapshot I'm going to close this out as WONTFIX.

Please feel free to reopen (adding a needinfo against me) if there is a valid use case that I'm missing here where the use of Nova isn't acceptable to workaround this.

Comment 6 Pascal Poschenrieder 2020-06-23 12:21:04 UTC
Seems this has to be reopened.

If you create a new server from a snapshot of a volume, and then create an image from this server, you can not create a new server from this image, because block_device_mapping in the image metadata is a string.

If I try to create a server from such an image, I get the following error: "Invalid image metadata. Error: A list is required in field img_block_device_mapping, not a str", because block_device_mapping is a string and not a list (missing JSON decoding).

If you need further information, just contact me.


Note You need to log in before you can comment on or make changes to this bug.