Bug 1795537
| Summary: | [FC-RHOS16] Unable to create cloned volume from encrypted volume | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | bkopilov <bkopilov> | ||||
| Component: | openstack-cinder | Assignee: | Eric Harney <eharney> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> | ||||
| Severity: | urgent | Docs Contact: | Chuck Copello <ccopello> | ||||
| Priority: | urgent | ||||||
| Version: | 16.0 (Train) | CC: | eharney, gcharot, jvisser, ltoscano, pgrist, tshefi | ||||
| Target Milestone: | ga | Keywords: | Regression, Triaged | ||||
| Target Release: | 16.0 (Train on RHEL 8.1) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-cinder-15.0.2-0.20200123220928.900f769.el8ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-02-06 14:44:22 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
bkopilov
2020-01-28 09:19:43 UTC
Created attachment 1655929 [details]
controller logs
Added tar.gz logs and steps from command line.
(overcloud) [stack@puma51 ~]$ cinder show 7ea23b23-8fd8-40e7-8563-9b75c88a9bd8
+--------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------+-------------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2020-01-28T09:00:59.000000 |
| description | None |
| encrypted | True |
| id | 7ea23b23-8fd8-40e7-8563-9b75c88a9bd8 |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | bootEncvol |
| os-vol-host-attr:host | controller-2@3parfc#SSD_r5 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | ac4c0a9f78b44816a870d250de3fb245 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2020-01-28T09:06:03.000000 |
| user_id | d081b16da79340808907fa092fa23bb1 |
| volume_image_metadata | checksum : ba3cd24377dde5dfdd58728894004abb |
| | container_format : bare |
| | disk_format : raw |
| | image_id : 98f483f9-12cf-4fad-8d7d-d398c111ee01 |
| | image_name : cirros |
| | min_disk : 0 |
| | min_ram : 0 |
| | signature_verified : False |
| | size : 46137344 |
| volume_type | LUKS |
+--------------------------------+-------------------------------------------------+
cinder create 1 --source-volid 7ea23b23-8fd8-40e7-8563-9b75c88a9bd8 --volume-type LUKS --name KUKU
hed_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='ac4c0a9f78b44816a870d250de3fb245',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2020-01-28T09:23:15Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=7ea23b23-8fd8-40e7-8563-9b75c88a9bd8,status='creating',terminated_at=None,updated_at=2020-01-28T09:23:15Z,user_id='d081b16da79340808907fa092fa23bb1',volume_attachment=<?>,volume_type=VolumeType(a20f5337-041e-45b2-a5ef-b4d43adb3526),volume_type_id=a20f5337-041e-45b2-a5ef-b4d43adb3526)}
|__Flow 'volume_create_manager': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: cryptsetup luksChangeKey /dev/sdb
Exit code: 2
Stdout: ''
Stderr: '/usr/share/cracklib/pw_dict.pwd.gz: No such file or directory\nPassword quality check failed:\n The password fails the dictionary check - error loading dictionary\n'
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager Traceback (most recent call last):
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager result = task.execute(**arguments)
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1119, in execute
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager context, volume, **volume_spec)
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 635, in _create_from_source_volume
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager rekey_model_update = self._rekey_volume(context, volume)
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 610, in _rekey_volume
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager new_key_id)
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager self.force_reraise()
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager six.reraise(self.type_, self.value, self.tb)
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager raise value
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 564, in _rekey_volume
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager log_errors=processutils.LOG_ALL_ERRORS)
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 126, in execute
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager cmd=sanitized_cmd)
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager Command: cryptsetup luksChangeKey /dev/sdb
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager Exit code: 2
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager Stdout: ''
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager Stderr: '/usr/share/cracklib/pw_dict.pwd.gz: No such file or directory\nPassword quality check failed:\n The password fails the dictionary check - error loading dictionary\n'
2020-01-28 09:23:55.846 50 ERROR cinder.volume.manager
2020-01-28 09:23:55.851 50 DEBUG cinder.volume.manager [req-c5811146-760a-4801-b44
For the record: does the same test work with other backends? This looks like a potential dependency problem where the c-vol container doesn't have cracklib installed but cryptsetup expects it. Answered separately, it seems it work on the Ceph backend. Then the question is: does anyone know why would cryptsetup require cracklib only for that backend? We don't call cryptsetup for the RBD driver. It's used for all iSCSI/FC drivers though. FWIW, cracklib-dicts is in the RHEL8 BaseOS repository Verified on:
openstack-cinder-15.0.2-0.20200123220928.900f769.el8ost.noarch
On a system with Cinder using 3par FC backend:
(overcloud) [stack@puma52 ~]$ cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | controller-1 | nova | enabled | up | 2020-01-31T02:55:27.000000 | - |
| cinder-scheduler | controller-0 | nova | enabled | up | 2020-01-31T02:55:27.000000 | - |
| cinder-scheduler | controller-1 | nova | enabled | up | 2020-01-31T02:55:27.000000 | - |
| cinder-scheduler | controller-2 | nova | enabled | up | 2020-01-31T02:55:28.000000 | - |
| cinder-volume | controller-2@3parfc | nova | enabled | up | 2020-01-31T02:55:31.000000 | - |
Create LUKS type:
(overcloud) [stack@puma52 ~]$ cinder type-create LUKS
+--------------------------------------+------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+------+-------------+-----------+
| 2ff7eab2-5ba5-4c21-b30f-1011acb808dd | LUKS | - | True |
+--------------------------------------+------+-------------+-----------+
(overcloud) [stack@puma52 ~]$ cinder encryption-type-create --cipher aes-xts-plain64 --key_size 256 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
| 2ff7eab2-5ba5-4c21-b30f-1011acb808dd | nova.volume.encryptors.luks.LuksEncryptor | aes-xts-plain64 | 256 | front-end |
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
(overcloud) [stack@puma52 ~]$ cinder type-key LUKS set volume_backend_name=3parfc
Upload Cirros image to Glance
(overcloud) [stack@puma52 ~]$ glance image-create --name cirros2 --disk-format qcow2 --container-format bare --file cirros-0.4.0-x86_64-disk.img
Create an encrypted volume from image as source volume:
(overcloud) [stack@puma52 ~]$ cinder create 3 --volume-type LUKS --name SourceEncVolOn3ParFC --image cirros2
(overcloud) [stack@puma52 ~]$ cinder show SourceEncVolOn3ParFC
+--------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------+-------------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2020-01-31T03:04:58.000000 |
| description | None |
| encrypted | True |
| id | 9b09afde-c401-4194-8352-52a230f3b1ca |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | SourceEncVolOn3ParFC |
| os-vol-host-attr:host | controller-2@3parfc#SSD_r5 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 273cdbd1bd94460fa919d321b70ff251 |
| replication_status | None |
| size | 3 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2020-01-31T03:05:36.000000 |
| user_id | 23fc4ef1ce904a6a86a6883ff0276733 |
| volume_image_metadata | checksum : 443b7623e27ecf03dc9e01ee93f67afe |
| | container_format : bare |
| | disk_format : qcow2 |
| | image_id : bebc3eba-f7a7-4a47-ba8f-88703cf98962 |
| | image_name : cirros2 |
| | min_disk : 0 |
| | min_ram : 0 |
| | signature_verified : False |
| | size : 12716032 |
| volume_type | LUKS |
+--------------------------------+-------------------------------------------------+
Clone said encrypted volume:
(overcloud) [stack@puma52 ~]$ cinder create 3 --source-volid 9b09afde-c401-4194-8352-52a230f3b1ca --name ClonedEncVol --volume-type LUKS
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2020-01-31T03:07:02.000000 |
| description | None |
| encrypted | True |
| id | 62dc7dda-4f70-4d6d-a5be-699553ba8858 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | ClonedEncVol |
| os-vol-host-attr:host | controller-2@3parfc#SSD_r5 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 273cdbd1bd94460fa919d321b70ff251 |
| replication_status | None |
| size | 3 |
| snapshot_id | None |
| source_volid | 9b09afde-c401-4194-8352-52a230f3b1ca |
| status | creating |
| updated_at | 2020-01-31T03:07:03.000000 |
| user_id | 23fc4ef1ce904a6a86a6883ff0276733 |
| volume_type | LUKS |
+--------------------------------+--------------------------------------+
Successfully cloned an encrypted volume backed by Cinder 3par FC:
(overcloud) [stack@puma52 ~]$ cinder show ClonedEncVol
+--------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------+-------------------------------------------------+
| attached_servers | [] |
| attachment_ids | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2020-01-31T03:07:02.000000 |
| description | None |
| encrypted | True |
| id | 62dc7dda-4f70-4d6d-a5be-699553ba8858 |
| metadata | |
| migration_status | None |
| multiattach | False |
| name | ClonedEncVol |
| os-vol-host-attr:host | controller-2@3parfc#SSD_r5 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 273cdbd1bd94460fa919d321b70ff251 |
| replication_status | None |
| size | 3 |
| snapshot_id | None |
| source_volid | 9b09afde-c401-4194-8352-52a230f3b1ca |
| status | available |
| updated_at | 2020-01-31T03:07:55.000000 |
| user_id | 23fc4ef1ce904a6a86a6883ff0276733 |
| volume_image_metadata | checksum : 443b7623e27ecf03dc9e01ee93f67afe |
| | container_format : bare |
| | disk_format : qcow2 |
| | image_id : bebc3eba-f7a7-4a47-ba8f-88703cf98962 |
| | image_name : cirros2 |
| | min_disk : 0 |
| | min_ram : 0 |
| | signature_verified : False |
| | size : 12716032 |
| volume_type | LUKS |
+--------------------------------+-------------------------------------------------+
(overcloud) [stack@puma52 ~]$ cinder list
+--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+
| 62dc7dda-4f70-4d6d-a5be-699553ba8858 | available | ClonedEncVol | 3 | LUKS | true | |
| 9b09afde-c401-4194-8352-52a230f3b1ca | available | SourceEncVolOn3ParFC | 3 | LUKS | true | |
+--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+
Good to verify, successfully cloned an encrypted volume backed 3par FC backend.
Ops forgot to add above also retested on a second deployment this time volumes were backed by LVM (iscsi) it too passed fine. (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+----------------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+----------------------+------+-------------+----------+-------------+ | 0c705680-2ce8-4e27-9383-eb4cb43f1bfb | available | ClonedEncLVMIscsi | 2 | LUKS2 | true | | | aa234d20-9198-4ae9-8504-1af43d2fe5de | available | SourceEcnVol | 2 | LUKS2 | true | Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0283 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |