Bug 1794249
| Summary: | cinder retype(with migration) from unencrypted to encrypted volume type in Ceph backend fails because of encryption header takes more space. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Shatadru Bandyopadhyay <sbandyop> |
| Component: | python-os-brick | Assignee: | Cinder Bugs List <cinder-bugs> |
| Status: | CLOSED MIGRATED | QA Contact: | Evelina Shames <eshames> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 17.0 (Wallaby) | CC: | apevec, eharney, gcharot, gfidente, jobernar, jschluet, jveiraca, lhh, ltoscano, pgrist, shtiwari, udesale |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-01-05 09:36:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1931621 | ||
encrypted volume creation in ceph backend works fine. (overcloud) [stack@undercloud-0 ~]$ openstack volume create --size 1 --type LuksEncryptor-Template-256 'Encrypted-Test-Volume' +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-01-23T04:00:00.000000 | | description | None | | encrypted | True | | id | 300e44af-bd71-4bf5-b0d4-aa97928892c5 | | migration_status | None | | multiattach | False | | name | Encrypted-Test-Volume | | properties | | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | type | LuksEncryptor-Template-256 | | updated_at | None | | user_id | f9ca5a979d024d58aba8804b29f9c804 | +---------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack volume show 'Encrypted-Test-Volume' +--------------------------------+--------------------------------------+ | Field | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-01-23T04:00:00.000000 | | description | None | | encrypted | True | | id | 300e44af-bd71-4bf5-b0d4-aa97928892c5 | | migration_status | None | | multiattach | False | | name | Encrypted-Test-Volume | | os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | a578afa9dad2434182d053c07ffba4b8 | | properties | | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | type | LuksEncryptor-Template-256 | | updated_at | 2020-01-23T04:00:19.000000 | | user_id | f9ca5a979d024d58aba8804b29f9c804 | +--------------------------------+--------------------------------------+ I have set volume_backend_name='tripleo_ceph' for `LuksEncryptor-Template-256` type. (overcloud) [stack@undercloud-0 ~]$ openstack volume type show LuksEncryptor-Template-256 +--------------------+--------------------------------------+ | Field | Value | +--------------------+--------------------------------------+ | access_project_ids | None | | description | None | | id | 151cec8d-7592-48b9-809d-32f9cad9b85c | | is_public | True | | name | LuksEncryptor-Template-256 | | properties | volume_backend_name='tripleo_ceph' | | qos_specs_id | None | +--------------------+--------------------------------------+ openstack secret list +------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration | +------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ | http://10.0.0.103:9311/v1/secrets/11b71833-01dc-4177-8f2a-844a0666f77c | None | 2020-01-23T03:59:58+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | symmetric | None | None | | http://10.0.0.103:9311/v1/secrets/1b5b6150-2db0-416f-a44c-fdc33924892f | None | 2020-01-21T14:17:16+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | symmetric | None | None | | http://10.0.0.103:9311/v1/secrets/5aa0fe07-1a23-4e92-87f2-8268b6252ea8 | None | 2020-01-21T14:22:20+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | symmetric | None | None | | http://10.0.0.103:9311/v1/secrets/7128a241-dc3d-4e00-91ca-4cb82b333f9b | None | 2020-01-21T14:35:19+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | symmetric | None | None | | http://10.0.0.103:9311/v1/secrets/97ff5917-81c5-4eaf-a715-7f5c16827fd8 | None | 2020-01-21T14:25:27+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | symmetric | None | None | | http://10.0.0.103:9311/v1/secrets/bc6c77ac-48fc-417f-9979-34214f733dd0 | None | 2020-01-21T14:43:46+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes | 256 | symmetric | None | None | +------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+ yet to narrow down but I see the 'connection_info' is wrong which us being passed to os.path.basename
from log :
~~~
2020-01-23 13:31:21.488 714897 DEBUG os_brick.initiator.connectors.rbd [req-8700d37b-b7ee-4d2d-b330-10edfe915e7c f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] ==> disconnect_volume: call u"{'connection_properties': {u'secret_type': u'ceph', 'device_path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7fd3e710da90>, u'name': u'volumes/volume-872ad53b-3ca7-4b99-a913-4a1313a3715b', u'encrypted': True, u'keyring': None, u'cluster_name': u'ceph', u'secret_uuid': None, u'qos_specs': None, u'auth_enabled': True, u'hosts': [u'172.17.3.15', u'172.17.3.27', u'172.17.3.32'], u'volume_id': u'872ad53b-3ca7-4b99-a913-4a1313a3715b', u'discard': True, u'access_mode': u'rw', u'auth_username': u'openstack', u'ports': [u'6789', u'6789', u'6789']}, 'self': <os_brick.initiator.connectors.rbd.RBDConnector object at 0x7fd3e6fa6950>, 'force': True, 'device_info': {'path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7fd3e710da90>}, 'ignore_errors': False}" trace_logging_wrapper /usr/lib/python2.7/site-packages/os_brick/utils.py:146
~~~
Not sure why rbd has a device_path set and that to a object rather than a string.
~~~
'device_path': <os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7fd3e710da90>
~~~
~~~
(Pdb) connection_info['data']['device_path']
<os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x7fba641df450>
~~~
and this object being passed to os.path.basename which is causing the error.
57 self.symlink_path = connection_info['data']['device_path']
58
59 # a unique name for the volume -- e.g., the iSCSI participant name
60 self.dev_name = 'crypt-%s' % os.path.basename(self.symlink_path) < ----
|
Description of problem: retype from non encrypted to encrypted volume in ceph backend fails Version-Release number of selected component (if applicable): RHOSP13 latest minor version docker exec -it $(docker ps|grep -i volume |awk '{print $1}' ) bash ()[root@controller-0 /]# rpm -qa|grep -i cinder python-cinder-12.0.8-3.el7ost.noarch puppet-cinder-12.4.1-5.el7ost.noarch openstack-cinder-12.0.8-3.el7ost.noarch python2-cinderclient-3.5.0-1.el7ost.noarch ()[root@controller-0 /]# rpm -qa|grep -i os-brick python2-os-brick-2.3.8-1.el7ost.noarch How reproducible: Always Steps to Reproduce: 1. deploy RHOSP with cinder backend ceph, deploy barbican 2. create volume in ceph backend 3. create volume type for encryption openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LuksEncryptor-Template-256 4. try to retype #cinder retype test-encrypt LuksEncryptor-Template-256 --migration-policy on-demand cinder type-list +--------------------------------------+----------------------------+-------------+-----------+ | ID | Name | Description | Is_Public | +--------------------------------------+----------------------------+-------------+-----------+ | 097fb345-e7af-4d41-bb4b-1c81ddba36f0 | tripleo_lvm | - | True | | 151cec8d-7592-48b9-809d-32f9cad9b85c | LuksEncryptor-Template-256 | - | True | | 1bd2cf2a-0a98-454c-abd6-c929fccaf461 | tripleo_hpelefthand | - | True | | 1f5cf5ff-4ded-4283-9837-0fa9f2cc1999 | tripleo_nfs | - | True | | 535c37f4-a1c0-4fd6-a257-d4adf36eff4d | tripleo_ceph | - | True | +--------------------------------------+----------------------------+-------------+-----------+ openstack volume show test-encrypt +--------------------------------+--------------------------------------+ | Field | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2020-01-21T14:08:41.000000 | | description | None | | encrypted | False | | id | 119b36f2-1410-4a42-b04a-7a78769ed2be | | migration_status | error <<-- after retype failed | | multiattach | False | | name | test-encrypt | | os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph | | os-vol-mig-status-attr:migstat | error | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | a578afa9dad2434182d053c07ffba4b8 | | properties | | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | type | tripleo_ceph | | updated_at | 2020-01-21T14:44:03.000000 | | user_id | f9ca5a979d024d58aba8804b29f9c804 | +--------------------------------+--------------------------------------+ Actual results: fails with traceback 2020-01-21 14:26:04.423 58 INFO cinder.volume.manager [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] Remove volume export completed successfully. 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] Failed to copy volume 119b36f2-1410-4a42-b04a-7a78769ed2 be to 4bd9f446-23d0-4f9c-b138-ec9116203902: AttributeError: 'RBDVolumeIOWrapper' object has no attribute 'rfind' 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager Traceback (most recent call last): 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2164, in _migrate_volume_generic 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager self._copy_volume_data(ctxt, volume, new_volume, remote='dest') 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2044, in _copy_volume_data 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager attach_encryptor=attach_encryptor) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1991, in _attach_volume 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager force=True) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager self.force_reraise() 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager six.reraise(self.type_, self.value, self.tb) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1985, in _attach_volume 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager encryption) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 535, in brick_attach_volume_encryptor 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager **encryption) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 527, in brick_get_encryptor 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager *args, **kwargs) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/os_brick/encryptors/__init__.py", line 91, in get_volume_encryptor 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager **kwargs) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager return import_class(import_str)(*args, **kwargs) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 61, in __init__ 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager *args, **kwargs) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/os_brick/encryptors/cryptsetup.py", line 60, in __init__ 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager self.dev_name = 'crypt-%s' % os.path.basename(self.symlink_path) 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager File "/usr/lib64/python2.7/posixpath.py", line 121, in basename 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager i = p.rfind('/') + 1 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager AttributeError: 'RBDVolumeIOWrapper' object has no attribute 'rfind' 2020-01-21 14:26:04.424 58 ERROR cinder.volume.manager 2020-01-21 14:26:04.474 58 DEBUG cinder.coordination [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] Lock "/var/lib/cinder/cinder-4bd9f446-23d0-4f9c-b138-ec9116203902-delete_volume" acquired by "delete_volume" :: waited 0.001s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:150 2020-01-21 14:26:04.584 58 DEBUG cinder.volume.drivers.rbd [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] connecting to ceph (timeout=-1). _do_conn /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:337 2020-01-21 14:26:04.675 58 DEBUG cinder.volume.drivers.rbd [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] volume has no backup snaps _delete_backup_snaps /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:950 2020-01-21 14:26:04.677 58 DEBUG cinder.volume.drivers.rbd [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] Volume volume-4bd9f446-23d0-4f9c-b138-ec9116203902 is not a clone. _get_clone_info /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:973 2020-01-21 14:26:04.682 58 DEBUG cinder.volume.drivers.rbd [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] deleting rbd volume volume-4bd9f446-23d0-4f9c-b138-ec9116203902 delete_volume /usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:1071 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] Exception during message handling: AttributeError: 'RBDVolumeIOWrapper' object has no attribute 'rfind' 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2826, in retype 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server new_reservations, status_update) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2822, in retype 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server new_type_id=new_type_id) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2431, in migrate_volume 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server volume.save() 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2424, in migrate_volume 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server self._migrate_volume_generic(ctxt, volume, host, new_type_id) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2186, in _migrate_volume_generic 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server new_volume) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2164, in _migrate_volume_generic 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server self._copy_volume_data(ctxt, volume, new_volume, remote='dest') 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2044, in _copy_volume_data 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server attach_encryptor=attach_encryptor) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1991, in _attach_volume 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server force=True) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2164, in _migrate_volume_generic 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server self._copy_volume_data(ctxt, volume, new_volume, remote='dest') 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 2044, in _copy_volume_data 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server attach_encryptor=attach_encryptor) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1991, in _attach_volume 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server force=True) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server self.force_reraise() 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1985, in _attach_volume 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server encryption) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 535, in brick_attach_volume_encryptor 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server **encryption) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 527, in brick_get_encryptor 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server *args, **kwargs) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/encryptors/__init__.py", line 91, in get_volume_encryptor 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server **kwargs) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server return import_class(import_str)(*args, **kwargs) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/encryptors/luks.py", line 61, in __init__ 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server *args, **kwargs) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/encryptors/cryptsetup.py", line 60, in __init__ 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server self.dev_name = 'crypt-%s' % os.path.basename(self.symlink_path) 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/posixpath.py", line 121, in basename 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server i = p.rfind('/') + 1 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server AttributeError: 'RBDVolumeIOWrapper' object has no attribute 'rfind' 2020-01-21 14:26:04.738 58 ERROR oslo_messaging.rpc.server 2020-01-21 14:26:04.799 58 INFO cinder.volume.manager [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] Deleted volume successfully. 2020-01-21 14:26:04.800 58 DEBUG cinder.coordination [req-31e8a87c-71a8-4e43-b0fb-c00632db5f02 f9ca5a979d024d58aba8804b29f9c804 a578afa9dad2434182d053c07ffba4b8 - default default] Lock "/var/lib/cinder/cinder-4bd9f446-23d0-4f9c-b138-ec9116203902-delete_volume" released by "delete_volume" :: held 0.326s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:162 force_reraise Expected results: re - type should complete Additional info: