Bug 1456355

Summary: Volume created from snapshot can not be migrated/retyped to different backend after snapshot got deleted
Product: Red Hat OpenStack Reporter: Martin Schuppert <mschuppe>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED ERRATA QA Contact: Tzach Shefi <tshefi>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: dhill, eharney, justinas.balciunas, lkuchlan, pgrist, rcernin, srevivo
Target Milestone: z4Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)Flags: lkuchlan: automate_bug+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-9.1.4-5.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-06 17:16:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Cinder verification logs none

Description Martin Schuppert 2017-05-29 08:17:39 UTC 
Description of problem:

Right now we have the following documented limitation:

https://docs.openstack.org/admin-guide/blockstorage-volume-migration.html

~~~
Migrating volumes that have snapshots are currently not allowed.
~~~

But when we:

1. Create volume
2. Create snapshot from volume
3. Create volume from the snapshot
4. Delete snapshot
5. Re-type newly created volume (from the snapshot) which includes storage migration

We fail with:
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager SnapshotNotFound: Snapshot 48cf30e7-942a-4c99-a432-cb09a6334d1c could not be found.

The issue seem to be that there is still a reference to the deleted snapshot:
                         id: 144e1805-f9b5-45e8-bea0-12ffe901c5b7
             snapshot_id: 48cf30e7-942a-4c99-a432-cb09a6334d1c

Version-Release number of selected component (if applicable):
OSP10 - python-cinder-9.1.4-3.el7ost

How reproducible:
always

Steps to Reproduce:
requ: env with 2 cinder backends
1. Create volume
2. Create snapshot from volume
3. Create volume from the snapshot
4. Delete snapshot
5. Re-type newly created volume (from the snapshot) which includes storage migration

Actual results:
[root@osp10-allinone ~(keystone_admin)]# cinder type-list
+--------------------------------------+--------+-------------+-----------+
| ID                                   | Name   | Description | Is_Public |
+--------------------------------------+--------+-------------+-----------+
| 3ced6c1f-d1aa-43c2-a78b-312b5649026a | lvm1   | -           | True      | <- lvm1 backend
| 5ae498b5-0fc9-4bd0-8bf0-71fe617ae2b7 | lvm1-1 | -           | True      | <- lvm1 backend
| 893ff87a-841d-41bd-a152-428c38005aec | nfs    | -           | True      | <- nfs backend
| 8c6f82c1-8e3c-4880-8cde-a51f30ce6cfd | ceph2  | -           | True      | <- ceph backend
| d6ba0690-5d38-4d91-8b14-6ce49cdf28ed | ceph   | -           | True      | <- ceph backend
| df7627bb-f1da-4db2-8f02-7adc042b459b | lvm2   | -           | True      | <- lvm2 backend
+--------------------------------------+--------+-------------+-----------+

[root@osp10-allinone ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+
| ID                                   | Status    | Name    | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+
| bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1    | lvm1        | false    |             |
+--------------------------------------+-----------+---------+------+-------------+----------+-------------+

* create snapshot
[root@osp10-allinone ~(keystone_admin)]# cinder snapshot-create my-test
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2017-05-29T07:37:16.846194           |
| description | None                                 |
| id          | 48cf30e7-942a-4c99-a432-cb09a6334d1c |
| metadata    | {}                                   |
| name        | None                                 |
| size        | 1                                    |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | bcfa0eda-299c-4286-9f9f-edcecbac197a |
+-------------+--------------------------------------+
[root@osp10-allinone ~(keystone_admin)]# cinder snapshot-list
+--------------------------------------+--------------------------------------+-----------+------+------+
| ID                                   | Volume ID                            | Status    | Name | Size |
+--------------------------------------+--------------------------------------+-----------+------+------+
| 48cf30e7-942a-4c99-a432-cb09a6334d1c | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | -    | 1    |
+--------------------------------------+--------------------------------------+-----------+------+------+

* create volume from snapshot

[root@osp10-allinone ~(keystone_admin)]# cinder create --snapshot 48cf30e7-942a-4c99-a432-cb09a6334d1c --name from-snapshot
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2017-05-29T07:38:05.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | from-snapshot                        |
| os-vol-host-attr:host          | osp10-allinone@lvm1#lvm1             |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 9f8e512dfae8410fba47bdb4b3095fc1     |
| replication_status             | disabled                             |
| size                           | 1                                    |
| snapshot_id                    | 48cf30e7-942a-4c99-a432-cb09a6334d1c |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | 2017-05-29T07:38:05.000000           |
| user_id                        | 6fd3eed05d9d494abcba7a47c7d3ef85     |
| volume_type                    | lvm1                                 |
+--------------------------------+--------------------------------------+

[root@osp10-allinone ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name          | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1    | lvm1        | false    |             |
| bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test       | 1    | lvm1        | false    |             |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+

* delete snapshot

[root@osp10-allinone ~(keystone_admin)]# cinder snapshot-delete 48cf30e7-942a-4c99-a432-cb09a6334d1c
[root@osp10-allinone ~(keystone_admin)]# cinder snapshot-list
+----+-----------+--------+------+------+
| ID | Volume ID | Status | Name | Size |
+----+-----------+--------+------+------+
+----+-----------+--------+------+------+


1) retype with same backend (lvm1):

[root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot lvm1-1
[root@osp10-allinone ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name          | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1    | lvm1-1      | false    |             |
| bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test       | 1    | lvm1        | false    |             |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+

2) retype with backend migration lvm1 -> lvm2

[root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot lvm2 --migration-policy on-demand
[root@osp10-allinone ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name          | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1    | lvm1-1      | false    |             |
| bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test       | 1    | lvm1        | false    |             |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+

2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager Traceback (most recent call last):
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     result = task.execute(**arguments)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 832, in execute
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     **volume_spec)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 438, in _create_from_snapshot
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     snapshot = objects.Snapshot.get_by_id(context, snapshot_id)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/objects/base.py", line 287, in get_by_id
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 1624, in get_by_id
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     return IMPL.get_by_id(context, model, id, *args, **kwargs)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 196, in wrapper
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 6204, in get_by_id
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     return _GET_METHODS[model](context, id, *args, **kwargs)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 196, in wrapper
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2601, in snapshot_get
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     return _snapshot_get(context, snapshot_id)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 196, in wrapper
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     return f(*args, **kwargs)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2594, in _snapshot_get
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager     raise exception.SnapshotNotFound(snapshot_id=snapshot_id)
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager SnapshotNotFound: Snapshot 48cf30e7-942a-4c99-a432-cb09a6334d1c could not be found.
2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager 

MariaDB [cinder]> select * from volumes where id='144e1805-f9b5-45e8-bea0-12ffe901c5b7'\G;
*************************** 1. row ***************************
                 created_at: 2017-05-29 07:38:05
                 updated_at: 2017-05-29 07:39:39
                 deleted_at: NULL
                    deleted: 0
                         id: 144e1805-f9b5-45e8-bea0-12ffe901c5b7
                     ec2_id: NULL
                    user_id: 6fd3eed05d9d494abcba7a47c7d3ef85
                 project_id: 9f8e512dfae8410fba47bdb4b3095fc1
                       host: osp10-allinone@lvm1#lvm1
                       size: 1
          availability_zone: nova
                     status: available
              attach_status: detached
               scheduled_at: 2017-05-29 07:38:05
                launched_at: 2017-05-29 07:38:08
              terminated_at: NULL
               display_name: from-snapshot
        display_description: NULL
          provider_location: NULL
              provider_auth: NULL
-->             snapshot_id: 48cf30e7-942a-4c99-a432-cb09a6334d1c
             volume_type_id: 5ae498b5-0fc9-4bd0-8bf0-71fe617ae2b7
               source_volid: NULL
                   bootable: 0
          provider_geometry: NULL
                   _name_id: NULL
          encryption_key_id: NULL
           migration_status: error
         replication_status: disabled
replication_extended_status: NULL
    replication_driver_data: NULL
        consistencygroup_id: NULL
                provider_id: NULL
                multiattach: 0
            previous_status: available
               cluster_name: NULL
                   group_id: NULL

MariaDB [cinder]> update volumes set snapshot_id=NULL where id='144e1805-f9b5-45e8-bea0-12ffe901c5b7';
Query OK, 1 row affected (0.04 sec)
Rows matched: 1  Changed: 1  Warnings: 0

[root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot lvm2 --migration-policy on-demand
[root@osp10-allinone ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name          | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | retyping  | from-snapshot | 1    | lvm1-1      | false    |             |
| 65aaede3-eeb3-4c5b-93b5-8055542d3c31 | available | from-snapshot | 1    | lvm2        | false    |             |
| bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test       | 1    | lvm1        | false    |             |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+

[root@osp10-allinone ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name          | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1    | lvm2        | false    |             |
| bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test       | 1    | lvm1        | false    |             |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+

3) retype with backend migration lvm2 -> ceph
[root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot ceph --migration-policy on-demand
[root@osp10-allinone ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name          | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1    | ceph        | false    |             |
| bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test       | 1    | lvm1        | false    |             |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+

4) retype with backend migration ceph -> lvm1
[root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot lvm1 --migration-policy on-demand
[root@osp10-allinone ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name          | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | retyping  | from-snapshot | 1    | ceph        | false    |             |
| 1e79860d-b0ef-4153-b97b-33747a07b0b4 | available | from-snapshot | 1    | lvm1        | false    |             |
| bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test       | 1    | lvm1        | false    |             |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
[root@osp10-allinone ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name          | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1    | lvm1        | false    |             |
| bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test       | 1    | lvm1        | false    |             |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+

Expected results:
volume can be retyped/migrated to different backend after snapshot got deleted.

Additional info:
https://bugs.launchpad.net/mos/+bug/1682417
Comment 1 Eric Harney 2017-06-12 18:00:59 UTC 
I've reproduced this issue on RDO Newton w/ openstack-cinder-9.1.4-1.el7.noarch.
Comment 4 Tzach Shefi 2017-08-14 06:50:15 UTC 
Verified on:
Openstack-cinder-9.1.4-6.el7ost.noarch

Retype of a volume, created from a deleted snapshot, was successfully completed. 
Details steps below:


1. Uploaded Cirros image
# openstack image create cirros --container-format bare --disk-format qcow2 --file cirros-0.3.5-x86_64-disk.img 

2. Created lvm volume from image, used image only to have data not to test boot flag.
#cinder create --volume-type tripleo_iscsi  --name lvm  1 --image cirros
#cinder list
...
| 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm  | 1    | tripleo_iscsi | true     |             |

3. Created snapshot of lvm volume
# cinder snapshot-create lvm --name lvm_snap
# cinder snapshot-list                                                                                                                                                                                                 
...                                                                                                                             
| feba3829-803c-436c-875b-d0a6df7879b7 | 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm_snap | 1    |                                                                                                                                

4. Create new volume from snap
# cinder create 1 --snapshot-id feba3829-803c-436c-875b-d0a6df7879b7 --name lvm2_vol_from_snap

Review both volumes which are available. 
$ cinder list
+--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+
| ID                                   | Status    | Name               | Size | Volume Type   | Bootable | Attached to |
+--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+
| 0155ffe8-c48f-419d-a2bd-944e7852091f | available | lvm2_vol_from_snap | 1    | tripleo_iscsi | true     |             |
| 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm                | 1    | tripleo_iscsi | true     |             |
+--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+


5. Delete cinder volume:
#cinder snapshot-delete feba3829-803c-436c-875b-d0a6df7879b7
#cinder snapshot-list
+----+-----------+--------+------+------+
| ID | Volume ID | Status | Name | Size |
+----+-----------+--------+------+------+
+----+-----------+--------+------+------+


6. Migrate volume (created from the now delete snap) from lvm Triple0_iscsi to NFS1 nfs backend. 
# cinder retype lvm2_vol_from_snap nfs1 --migration-policy on-demand 

# cinder list
+--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+
| ID                                   | Status    | Name               | Size | Volume Type   | Bootable | Attached to |
+--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+
| 0155ffe8-c48f-419d-a2bd-944e7852091f | retyping  | lvm2_vol_from_snap | 1    | tripleo_iscsi | true     |             |
| 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm                | 1    | tripleo_iscsi | true     |             |
| 47565313-b04b-4dd4-ba40-5812be160b12 | available | lvm2_vol_from_snap | 1    | nfs1          | true     |             |
+--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+

Once migration finished, cinder list shows migration was completed successfully as expected. 

# cinder list
+--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+
| ID                                   | Status    | Name               | Size | Volume Type   | Bootable | Attached to |
+--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+
| 0155ffe8-c48f-419d-a2bd-944e7852091f | available | lvm2_vol_from_snap | 1    | nfs1          | true     |             |
| 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm                | 1    | tripleo_iscsi | true     |             |
+--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+
Comment 5 Tzach Shefi 2017-08-14 06:51:36 UTC 
Created attachment 1312968 [details]
Cinder verification logs
Comment 7 errata-xmlrpc 2017-09-06 17:16:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2660
Comment 8 Alan Bishop 2020-05-12 21:13:32 UTC 
*** Bug 1834929 has been marked as a duplicate of this bug. ***