Description of problem: Right now we have the following documented limitation: https://docs.openstack.org/admin-guide/blockstorage-volume-migration.html ~~~ Migrating volumes that have snapshots are currently not allowed. ~~~ But when we: 1. Create volume 2. Create snapshot from volume 3. Create volume from the snapshot 4. Delete snapshot 5. Re-type newly created volume (from the snapshot) which includes storage migration We fail with: 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager SnapshotNotFound: Snapshot 48cf30e7-942a-4c99-a432-cb09a6334d1c could not be found. The issue seem to be that there is still a reference to the deleted snapshot: id: 144e1805-f9b5-45e8-bea0-12ffe901c5b7 snapshot_id: 48cf30e7-942a-4c99-a432-cb09a6334d1c Version-Release number of selected component (if applicable): OSP10 - python-cinder-9.1.4-3.el7ost How reproducible: always Steps to Reproduce: requ: env with 2 cinder backends 1. Create volume 2. Create snapshot from volume 3. Create volume from the snapshot 4. Delete snapshot 5. Re-type newly created volume (from the snapshot) which includes storage migration Actual results: [root@osp10-allinone ~(keystone_admin)]# cinder type-list +--------------------------------------+--------+-------------+-----------+ | ID | Name | Description | Is_Public | +--------------------------------------+--------+-------------+-----------+ | 3ced6c1f-d1aa-43c2-a78b-312b5649026a | lvm1 | - | True | <- lvm1 backend | 5ae498b5-0fc9-4bd0-8bf0-71fe617ae2b7 | lvm1-1 | - | True | <- lvm1 backend | 893ff87a-841d-41bd-a152-428c38005aec | nfs | - | True | <- nfs backend | 8c6f82c1-8e3c-4880-8cde-a51f30ce6cfd | ceph2 | - | True | <- ceph backend | d6ba0690-5d38-4d91-8b14-6ce49cdf28ed | ceph | - | True | <- ceph backend | df7627bb-f1da-4db2-8f02-7adc042b459b | lvm2 | - | True | <- lvm2 backend +--------------------------------------+--------+-------------+-----------+ [root@osp10-allinone ~(keystone_admin)]# cinder list +--------------------------------------+-----------+---------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------+------+-------------+----------+-------------+ | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1 | lvm1 | false | | +--------------------------------------+-----------+---------+------+-------------+----------+-------------+ * create snapshot [root@osp10-allinone ~(keystone_admin)]# cinder snapshot-create my-test +-------------+--------------------------------------+ | Property | Value | +-------------+--------------------------------------+ | created_at | 2017-05-29T07:37:16.846194 | | description | None | | id | 48cf30e7-942a-4c99-a432-cb09a6334d1c | | metadata | {} | | name | None | | size | 1 | | status | creating | | updated_at | None | | volume_id | bcfa0eda-299c-4286-9f9f-edcecbac197a | +-------------+--------------------------------------+ [root@osp10-allinone ~(keystone_admin)]# cinder snapshot-list +--------------------------------------+--------------------------------------+-----------+------+------+ | ID | Volume ID | Status | Name | Size | +--------------------------------------+--------------------------------------+-----------+------+------+ | 48cf30e7-942a-4c99-a432-cb09a6334d1c | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | - | 1 | +--------------------------------------+--------------------------------------+-----------+------+------+ * create volume from snapshot [root@osp10-allinone ~(keystone_admin)]# cinder create --snapshot 48cf30e7-942a-4c99-a432-cb09a6334d1c --name from-snapshot +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2017-05-29T07:38:05.000000 | | description | None | | encrypted | False | | id | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | | metadata | {} | | migration_status | None | | multiattach | False | | name | from-snapshot | | os-vol-host-attr:host | osp10-allinone@lvm1#lvm1 | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 9f8e512dfae8410fba47bdb4b3095fc1 | | replication_status | disabled | | size | 1 | | snapshot_id | 48cf30e7-942a-4c99-a432-cb09a6334d1c | | source_volid | None | | status | creating | | updated_at | 2017-05-29T07:38:05.000000 | | user_id | 6fd3eed05d9d494abcba7a47c7d3ef85 | | volume_type | lvm1 | +--------------------------------+--------------------------------------+ [root@osp10-allinone ~(keystone_admin)]# cinder list +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1 | lvm1 | false | | | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1 | lvm1 | false | | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ * delete snapshot [root@osp10-allinone ~(keystone_admin)]# cinder snapshot-delete 48cf30e7-942a-4c99-a432-cb09a6334d1c [root@osp10-allinone ~(keystone_admin)]# cinder snapshot-list +----+-----------+--------+------+------+ | ID | Volume ID | Status | Name | Size | +----+-----------+--------+------+------+ +----+-----------+--------+------+------+ 1) retype with same backend (lvm1): [root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot lvm1-1 [root@osp10-allinone ~(keystone_admin)]# cinder list +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1 | lvm1-1 | false | | | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1 | lvm1 | false | | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ 2) retype with backend migration lvm1 -> lvm2 [root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot lvm2 --migration-policy on-demand [root@osp10-allinone ~(keystone_admin)]# cinder list +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1 | lvm1-1 | false | | | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1 | lvm1 | false | | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager Traceback (most recent call last): 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager result = task.execute(**arguments) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 832, in execute 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager **volume_spec) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 438, in _create_from_snapshot 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager snapshot = objects.Snapshot.get_by_id(context, snapshot_id) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/objects/base.py", line 287, in get_by_id 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 1624, in get_by_id 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager return IMPL.get_by_id(context, model, id, *args, **kwargs) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 196, in wrapper 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager return f(*args, **kwargs) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 6204, in get_by_id 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager return _GET_METHODS[model](context, id, *args, **kwargs) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 196, in wrapper 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager return f(*args, **kwargs) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2601, in snapshot_get 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager return _snapshot_get(context, snapshot_id) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 196, in wrapper 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager return f(*args, **kwargs) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2594, in _snapshot_get 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager raise exception.SnapshotNotFound(snapshot_id=snapshot_id) 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager SnapshotNotFound: Snapshot 48cf30e7-942a-4c99-a432-cb09a6334d1c could not be found. 2017-05-29 03:39:38.413 4196 ERROR cinder.volume.manager MariaDB [cinder]> select * from volumes where id='144e1805-f9b5-45e8-bea0-12ffe901c5b7'\G; *************************** 1. row *************************** created_at: 2017-05-29 07:38:05 updated_at: 2017-05-29 07:39:39 deleted_at: NULL deleted: 0 id: 144e1805-f9b5-45e8-bea0-12ffe901c5b7 ec2_id: NULL user_id: 6fd3eed05d9d494abcba7a47c7d3ef85 project_id: 9f8e512dfae8410fba47bdb4b3095fc1 host: osp10-allinone@lvm1#lvm1 size: 1 availability_zone: nova status: available attach_status: detached scheduled_at: 2017-05-29 07:38:05 launched_at: 2017-05-29 07:38:08 terminated_at: NULL display_name: from-snapshot display_description: NULL provider_location: NULL provider_auth: NULL --> snapshot_id: 48cf30e7-942a-4c99-a432-cb09a6334d1c volume_type_id: 5ae498b5-0fc9-4bd0-8bf0-71fe617ae2b7 source_volid: NULL bootable: 0 provider_geometry: NULL _name_id: NULL encryption_key_id: NULL migration_status: error replication_status: disabled replication_extended_status: NULL replication_driver_data: NULL consistencygroup_id: NULL provider_id: NULL multiattach: 0 previous_status: available cluster_name: NULL group_id: NULL MariaDB [cinder]> update volumes set snapshot_id=NULL where id='144e1805-f9b5-45e8-bea0-12ffe901c5b7'; Query OK, 1 row affected (0.04 sec) Rows matched: 1 Changed: 1 Warnings: 0 [root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot lvm2 --migration-policy on-demand [root@osp10-allinone ~(keystone_admin)]# cinder list +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | retyping | from-snapshot | 1 | lvm1-1 | false | | | 65aaede3-eeb3-4c5b-93b5-8055542d3c31 | available | from-snapshot | 1 | lvm2 | false | | | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1 | lvm1 | false | | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ [root@osp10-allinone ~(keystone_admin)]# cinder list +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1 | lvm2 | false | | | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1 | lvm1 | false | | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ 3) retype with backend migration lvm2 -> ceph [root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot ceph --migration-policy on-demand [root@osp10-allinone ~(keystone_admin)]# cinder list +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1 | ceph | false | | | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1 | lvm1 | false | | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ 4) retype with backend migration ceph -> lvm1 [root@osp10-allinone ~(keystone_admin)]# cinder retype from-snapshot lvm1 --migration-policy on-demand [root@osp10-allinone ~(keystone_admin)]# cinder list +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | retyping | from-snapshot | 1 | ceph | false | | | 1e79860d-b0ef-4153-b97b-33747a07b0b4 | available | from-snapshot | 1 | lvm1 | false | | | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1 | lvm1 | false | | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ [root@osp10-allinone ~(keystone_admin)]# cinder list +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ | 144e1805-f9b5-45e8-bea0-12ffe901c5b7 | available | from-snapshot | 1 | lvm1 | false | | | bcfa0eda-299c-4286-9f9f-edcecbac197a | available | my-test | 1 | lvm1 | false | | +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+ Expected results: volume can be retyped/migrated to different backend after snapshot got deleted. Additional info: https://bugs.launchpad.net/mos/+bug/1682417
I've reproduced this issue on RDO Newton w/ openstack-cinder-9.1.4-1.el7.noarch.
Verified on: Openstack-cinder-9.1.4-6.el7ost.noarch Retype of a volume, created from a deleted snapshot, was successfully completed. Details steps below: 1. Uploaded Cirros image # openstack image create cirros --container-format bare --disk-format qcow2 --file cirros-0.3.5-x86_64-disk.img 2. Created lvm volume from image, used image only to have data not to test boot flag. #cinder create --volume-type tripleo_iscsi --name lvm 1 --image cirros #cinder list ... | 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm | 1 | tripleo_iscsi | true | | 3. Created snapshot of lvm volume # cinder snapshot-create lvm --name lvm_snap # cinder snapshot-list ... | feba3829-803c-436c-875b-d0a6df7879b7 | 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm_snap | 1 | 4. Create new volume from snap # cinder create 1 --snapshot-id feba3829-803c-436c-875b-d0a6df7879b7 --name lvm2_vol_from_snap Review both volumes which are available. $ cinder list +--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+ | 0155ffe8-c48f-419d-a2bd-944e7852091f | available | lvm2_vol_from_snap | 1 | tripleo_iscsi | true | | | 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm | 1 | tripleo_iscsi | true | | +--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+ 5. Delete cinder volume: #cinder snapshot-delete feba3829-803c-436c-875b-d0a6df7879b7 #cinder snapshot-list +----+-----------+--------+------+------+ | ID | Volume ID | Status | Name | Size | +----+-----------+--------+------+------+ +----+-----------+--------+------+------+ 6. Migrate volume (created from the now delete snap) from lvm Triple0_iscsi to NFS1 nfs backend. # cinder retype lvm2_vol_from_snap nfs1 --migration-policy on-demand # cinder list +--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+ | 0155ffe8-c48f-419d-a2bd-944e7852091f | retyping | lvm2_vol_from_snap | 1 | tripleo_iscsi | true | | | 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm | 1 | tripleo_iscsi | true | | | 47565313-b04b-4dd4-ba40-5812be160b12 | available | lvm2_vol_from_snap | 1 | nfs1 | true | | +--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+ Once migration finished, cinder list shows migration was completed successfully as expected. # cinder list +--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+ | 0155ffe8-c48f-419d-a2bd-944e7852091f | available | lvm2_vol_from_snap | 1 | nfs1 | true | | | 1b0f021d-c9d7-4399-ab10-4adb585f71a7 | available | lvm | 1 | tripleo_iscsi | true | | +--------------------------------------+-----------+--------------------+------+---------------+----------+-------------+
Created attachment 1312968 [details] Cinder verification logs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2660
*** Bug 1834929 has been marked as a duplicate of this bug. ***