Bug 1414222 - Cinder volume creation fails from the snapshot of an instance that has a bootable cinder volume attached to it as a backend
Summary: Cinder volume creation fails from the snapshot of an instance that has a boot...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: async
: ---
Assignee: Eric Harney
QA Contact: Tzach Shefi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-18 05:12 UTC by Punit Kundal
Modified: 2020-03-11 15:37 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-15 16:19:25 UTC
Target Upstream Version:
tshefi: automate_bug-


Attachments (Terms of Use)
sos report from my test environment with the debug logs (10.55 MB, application/x-xz)
2017-01-18 05:12 UTC, Punit Kundal
no flags Details
PDF for additional details regarding the issue. (451.60 KB, application/pdf)
2017-01-30 04:31 UTC, Punit Kundal
no flags Details
Reproducetion logs (341.11 KB, application/x-gzip)
2017-02-02 08:46 UTC, Tzach Shefi
no flags Details

Description Punit Kundal 2017-01-18 05:12:23 UTC
Created attachment 1242025 [details]
sos report from my test environment with the debug logs

Description of problem:

Cinder volume creation fails from the snapshot of an instance that has a bootable cinder volume attached to it as a backend

Version-Release number of selected component (if applicable):

[root@dell-per320-1 ~]# rpm -qa | grep cinder
python-cinder-8.1.1-4.el7ost.noarch
openstack-cinder-8.1.1-4.el7ost.noarch
python-cinderclient-1.6.0-1.el7ost.noarch

[root@dell-per320-1 ~]# rpm -qa | grep glance 
python-glanceclient-2.0.0-1.el7ost.noarch
python-glance-12.0.0-2.el7ost.noarch
python-glance-store-0.13.1-2.el7ost.noarch
openstack-glance-12.0.0-2.el7ost.noarch


How reproducible:
Always

Steps to Reproduce:
1. Set the glance_api_version to 2 in /etc/cinder/cinder.conf

[root@dell-per320-1 ~]# grep glance_api_version /etc/cinder/cinder.conf 
glance_api_version = 2


2. Create a normal glance image

[root@dell-per320-1 ~(keystone_admin)]# glance image-list
+--------------------------------------+--------+
| ID                                   | Name   |
+--------------------------------------+--------+
| bf956b1b-6eac-41d8-89b1-df49da099a75 | cirros |
+--------------------------------------+--------+


3. Create a cinder volume from the glance image

[root@dell-per320-1 ~(keystone_admin)]# cinder create --image cirros --display-name vol1 3
+--------------------------------+--------------------------------------+
|            Property            |                Value                 |
+--------------------------------+--------------------------------------+
|          attachments           |                  []                  |
|       availability_zone        |                 nova                 |
|            bootable            |                false                 |
|      consistencygroup_id       |                 None                 |
|           created_at           |      2017-01-17T08:21:00.000000      |
|          description           |                 None                 |
|           encrypted            |                False                 |
|               id               | 6ca596d9-d3a3-4ff8-afdd-dd364cd0b41e |
|            metadata            |                  {}                  |
|        migration_status        |                 None                 |
|          multiattach           |                False                 |
|              name              |                 vol1                 |
|     os-vol-host-attr:host      |                 None                 |
| os-vol-mig-status-attr:migstat |                 None                 |
| os-vol-mig-status-attr:name_id |                 None                 |
|  os-vol-tenant-attr:tenant_id  |   feb4db429eb748d9b16333c2cf2f9147   |
|       replication_status       |               disabled               |
|              size              |                  3                   |
|          snapshot_id           |                 None                 |
|          source_volid          |                 None                 |
|             status             |               creating               |
|           updated_at           |                 None                 |
|            user_id             |   5f3228ca573d484aa65ca58daf7ae533   |
|          volume_type           |                 None                 |
+--------------------------------+--------------------------------------+

[root@dell-per320-1 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 6ca596d9-d3a3-4ff8-afdd-dd364cd0b41e | available | vol1 |  3   |      -      |   true   |             |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+

4. Boot a nova instance with the above cinder volume as a backend for the instance

[root@dell-per320-1 ~(keystone_admin)]# nova boot --poll --flavor m1.tiny --nic net-name=private --block-device-mapping vda=6ca596d9-d3a3-4ff8-afdd-dd364cd0b41e:::0 inst1
+--------------------------------------+--------------------------------------------------+
| Property                             | Value                                            |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          |                                                  |
| OS-EXT-SRV-ATTR:host                 | -                                                |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000001                                |
| OS-EXT-STS:power_state               | 0                                                |
| OS-EXT-STS:task_state                | scheduling                                       |
| OS-EXT-STS:vm_state                  | building                                         |
| OS-SRV-USG:launched_at               | -                                                |
| OS-SRV-USG:terminated_at             | -                                                |
| accessIPv4                           |                                                  |
| accessIPv6                           |                                                  |
| adminPass                            | avYTQhH2yt5n                                     |
| config_drive                         |                                                  |
| created                              | 2017-01-17T08:24:51Z                             |
| flavor                               | m1.tiny (1)                                      |
| hostId                               |                                                  |
| id                                   | 01614fb9-6348-41d0-b037-963968121c43             |
| image                                | Attempt to boot from volume - no image supplied  |
| key_name                             | -                                                |
| metadata                             | {}                                               |
| name                                 | inst1                                            |
| os-extended-volumes:volumes_attached | [{"id": "6ca596d9-d3a3-4ff8-afdd-dd364cd0b41e"}] |
| progress                             | 0                                                |
| security_groups                      | default                                          |
| status                               | BUILD                                            |
| tenant_id                            | feb4db429eb748d9b16333c2cf2f9147                 |
| updated                              | 2017-01-17T08:24:51Z                             |
| user_id                              | 5f3228ca573d484aa65ca58daf7ae533                 |
+--------------------------------------+--------------------------------------------------+

Server building... 100% complete
Finished

[root@dell-per320-1 ~(keystone_admin)]# nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks         |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 01614fb9-6348-41d0-b037-963968121c43 | inst1 | ACTIVE | -          | Running     | private=10.0.0.3 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

5. Stop the nova instance and take a snapshot of the instance with the nova image-create command

[root@dell-per320-1 ~(keystone_admin)]# nova stop inst1
Request to stop server inst1 has been accepted.

[root@dell-per320-1 ~(keystone_admin)]# nova list
+--------------------------------------+-------+---------+------------+-------------+------------------+
| ID                                   | Name  | Status  | Task State | Power State | Networks         |
+--------------------------------------+-------+---------+------------+-------------+------------------+
| 01614fb9-6348-41d0-b037-963968121c43 | inst1 | SHUTOFF | -          | Shutdown    | private=10.0.0.3 |
+--------------------------------------+-------+---------+------------+-------------+------------------+

[root@dell-per320-1 ~(keystone_admin)]# nova image-create inst1 inst1-snap

[root@dell-per320-1 ~(keystone_admin)]# glance image-list
+--------------------------------------+------------+
| ID                                   | Name       |
+--------------------------------------+------------+
| bf956b1b-6eac-41d8-89b1-df49da099a75 | cirros     |
| abb1f188-f0b5-4464-baea-1ac15e5c6ac7 | inst1-snap |
+--------------------------------------+------------+

6. Now try to create a cinder volume from the above instance snapshot

[root@dell-per320-1 ~(keystone_admin)]# cinder create --image inst1-snap --display-name vol2 3
+--------------------------------+--------------------------------------+
|            Property            |                Value                 |
+--------------------------------+--------------------------------------+
|          attachments           |                  []                  |
|       availability_zone        |                 nova                 |
|            bootable            |                false                 |
|      consistencygroup_id       |                 None                 |
|           created_at           |      2017-01-17T08:26:57.000000      |
|          description           |                 None                 |
|           encrypted            |                False                 |
|               id               | 0999f502-d02a-4d22-9832-ab90af4332fa |
|            metadata            |                  {}                  |
|        migration_status        |                 None                 |
|          multiattach           |                False                 |
|              name              |                 vol2                 |
|     os-vol-host-attr:host      |                 None                 |
| os-vol-mig-status-attr:migstat |                 None                 |
| os-vol-mig-status-attr:name_id |                 None                 |
|  os-vol-tenant-attr:tenant_id  |   feb4db429eb748d9b16333c2cf2f9147   |
|       replication_status       |               disabled               |
|              size              |                  3                   |
|          snapshot_id           |                 None                 |
|          source_volid          |                 None                 |
|             status             |               creating               |
|           updated_at           |                 None                 |
|            user_id             |   5f3228ca573d484aa65ca58daf7ae533   |
|          volume_type           |                 None                 |
+--------------------------------+--------------------------------------+

[root@dell-per320-1 ~(keystone_admin)]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 0999f502-d02a-4d22-9832-ab90af4332fa | error  | vol2 |  3   |      -      |  false   |                                      |
| 6ca596d9-d3a3-4ff8-afdd-dd364cd0b41e | in-use | vol1 |  3   |      -      |   true   | 01614fb9-6348-41d0-b037-963968121c43 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+


Actual results:

As can be seen above the cinder volume creation failed

Expected results:

Cinder volume should be created successfully

Additional info:

This issue is also seen on the following package versions:

# rpm -qa | grep cinder | sort
openstack-cinder-8.1.1-2.el7ost.noarch
python-cinder-8.1.1-2.el7ost.noarch
python-cinderclient-1.6.0-1.el7ost.noarch

# rpm -qa | grep glance | sort
openstack-glance-12.0.0-1.el7ost.noarch
python-glance-12.0.0-1.el7ost.noarch
python-glanceclient-2.0.0-1.el7ost.noarch
python-glance-store-0.13.1-2.el7ost.noarch

Comment 2 Lee Yarwood 2017-01-20 16:12:56 UTC
FYI the workaround from RHBZ#1414226 of using nova commands works here also.

Comment 5 Punit Kundal 2017-01-30 04:26:51 UTC
Hello,

Many thanks for the workaround proposed in RHBZ#1414226 [0] but the customer is not convinced with it and wants to do the following:

1. Create a new cinder volume from a snapshot of an instance that has a bootable cinder volume attached to it as a backend.

2. The boot a new instance from the newly created cinder

The customer is using Nimble storage backend.

So when a new instance is booted directly from a snapshot image (not creating a boot cinder volume separately first), the Nimble backend driver creates a 'cloned volume' in the physical storage, which is what the customer wants to avoid.

The customer does not want to create cloned volume because maximum number of cloned volumes is limited in Nimble Storage.

The customer has found that if they create a new cinder volume from a snapshot of an instance first, the Nimble driver creates a normal volume, not a cloned volume.

For more in-depth details. I am attaching the pdf file "Case01770883" to the BZ ticket. Please refer the same.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1414226#c1

Thanks for all your help till now. Any guidance/help from your side will be highly appreciated.

Comment 6 Punit Kundal 2017-01-30 04:31:54 UTC
Created attachment 1245751 [details]
PDF for additional details regarding the issue.

Comment 8 Elise Gafford 2017-02-01 14:25:55 UTC
Hi Tzach,

Please evaluate whether this bug exists against a supported backend (Ceph) and sync with Sean for partner followup.

Thanks!

Comment 9 Manabu Ori 2017-02-01 14:40:34 UTC
Hi Elise,

I experienced this symptom with LVM and Nibmle backend, so I think it's independent of backends, FYI.

Comment 10 Tzach Shefi 2017-02-02 08:32:41 UTC
Reproduced issue, OSP9 with internal CEPH.
Following steps I reached same bad result: 

$ cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status |     Name     | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| ad35a831-e70b-46aa-966d-9d8097a5cd48 | error  |     vol2     |  2   |      -      |  false   |                                      |
| e36f9205-0402-4e26-950e-7ddc7fd46c3a | in-use | VolFromImage |  2   |      -      |   true   | d96ce34c-935e-42fc-b757-22a576134012 |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+

Comment 11 Tzach Shefi 2017-02-02 08:46:08 UTC
Created attachment 1247036 [details]
Reproducetion logs

These are the IDs to look for in the logs:

Glance image 
| addc9753-497a-45c7-8a66-3cc3f6ecb71a | cirros.raw

Cinder volume from ^ image
e36f9205-0402-4e26-950e-7ddc7fd46c3a | available | VolFromImage |  2 

Nova instance boot from ^ volume
d96ce34c-935e-42fc-b757-22a576134012 | inst1 | ACTIVE

Glance snap-shot from ^ instance
4d3a52e8-3a9f-440c-a415-9523a852d4c8 | inst1-snap |  

Failed to create Cinder volume from ^ snapshot
ad35a831-e70b-46aa-966d-9d8097a5cd48 | error  |     vol2

Comment 14 Eric Harney 2017-02-15 16:19:25 UTC
We can't fix this because the images generated by Nova with the image-create operation are not intended to be used by Cinder.  This sequence of operations is not supported.

The suggested alternative is to use Cinder volume cloning to make a temporary volume, and then use Cinder's upload-to-image functionality to store that image in Glance.

Since the root of this issue is that the Nimble Cinder driver has limitations which it should not really have, this seems like the most viable approach.

Comment 15 Tzach Shefi 2018-09-23 05:09:34 UTC
Due to #14, adding that we don't have Nimble to test this. 
Nothing for Qe to test/automate in terms close loop.


Note You need to log in before you can comment on or make changes to this bug.