Bug 1729755 - We don't block retyping of an encrypted volume to multi-attach encrypted volume
Summary: We don't block retyping of an encrypted volume to multi-attach encrypted volume
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Jon Bernard
QA Contact: Tzach Shefi
Chuck Copello
URL:
Whiteboard:
Depends On:
Blocks: 1744112
TreeView+ depends on / blocked
 
Reported: 2019-07-14 09:32 UTC by Tzach Shefi
Modified: 2020-09-14 19:03 UTC (History)
3 users (show)

Fixed In Version: openstack-cinder-14.0.2-0.20190913170442.cd233e5.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1744112 (view as bug list)
Environment:
Last Closed: 2020-03-05 11:54:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 680473 0 'None' MERGED Don't allow retype to encrypted+multiattach type 2020-09-14 18:04:55 UTC

Description Tzach Shefi 2019-07-14 09:32:17 UTC
Description of problem:  We currently rightfully don't allow/support creation of encrypted multi-attach volume:

$ cinder  create 1 --volume-type enc_ma_lvm --name enc_ma_vol
ERROR: Invalid volume: Multiattach cannot be used with encrypted volumes. (HTTP 400) (Request-ID: req-8859dbc6-646f-4cb0-8a0d-419a019ffae1)

However if you create an encrypted (none MA) volume and retype it to a none supported multi-attached encrypted volume it would ideally fail/warn. Yet operation is executed despite the none supported resulting volume, this IMHO should be blocked. 


Yes I know retype doesn't validate/check backends before it accepts what looks like a valid request, which explains why no error is reported and operation is preformed. Call it wishful QE thinking I'd like to see such operations validated and blocked if end result isn't supported.    


Version-Release number of selected component (if applicable):
puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch
openstack-cinder-14.0.1-0.20190607000407.23d1a72.el8ost.noarch
python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch
python3-cinder-14.0.1-0.20190607000407.23d1a72.el8ost.noarch


How reproducible: 
Every time


Steps to Reproduce:
1. Create both backends

| 04c5d0b5-3ed4-4ba1-b74d-7bcd8b2b4540 | enc_lvm    | {'volume_backend_name': 'tripleo_iscsi'}  

cinder encryption-type-create --cipher aes-xts-plain64 --key_size 256 --control_location front-end enc_lvm  nova.volume.encryptors.luks.LuksEncryptor

| 50c3fb05-1c34-45e6-b40c-44296b3b90ad | enc_ma_lvm | {'volume_backend_name': 'tripleo_iscsi', 'multiattach': '<is> True'} |

 cinder encryption-type-create --cipher aes-xts-plain64 --key_size 256 --control_location front-end enc_ma_lvm  nova.volume.encryptors.luks.LuksEncryptor


2. Create an enc lvm volume:

(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --volume-type enc_lvm --name Enc-vol 
+--------------------------------+---------------------------------------+
| Property                       | Value                                 |
+--------------------------------+---------------------------------------+
| attachments                    | []                                    |
| availability_zone              | nova                                  |
| bootable                       | false                                 |
| consistencygroup_id            | None                                  |
| created_at                     | 2019-07-14T09:16:16.000000            |
| description                    | None                                  |
| encrypted                      | True                                  |
| id                             | 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0  |
| metadata                       | {}                                    |
| migration_status               | None                                  |
| multiattach                    | False                                 |
| name                           | Enc-vol                               |
| os-vol-host-attr:host          | hostgroup@tripleo_iscsi#tripleo_iscsi |
| os-vol-mig-status-attr:migstat | None                                  |
| os-vol-mig-status-attr:name_id | None                                  |
| os-vol-tenant-attr:tenant_id   | 5850830f52774e0dab7b7b8e508b6a56      |
| replication_status             | None                                  |
| size                           | 1                                     |
| snapshot_id                    | None                                  |
| source_volid                   | None                                  |
| status                         | available                             |
| updated_at                     | 2019-07-14T09:16:17.000000            |
| user_id                        | 8fe58395864a43158a847d1a9ffd4e9d      |
| volume_type                    | enc_lvm                               |
+--------------------------------+---------------------------------------+

Volume before migration
(overcloud) [stack@undercloud-0 ~]$ cinder list

| 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 | available | Enc-vol          | 1    | enc_lvm     | false    |             |


3. Retype the volume to none supported enc_ma_lvm type:

(overcloud) [stack@undercloud-0 ~]$ cinder retype 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 enc_ma_lvm --migration-policy on-demand    

!error returned :( 
                                                                                                           
(overcloud) [stack@undercloud-0 ~]$ cinder show 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0                                                                                                                                                         
+--------------------------------+---------------------------------------+                                                                                                                                                                   
| Property                       | Value                                 |                                                                                                                                                                   
+--------------------------------+---------------------------------------+                                                                                                                                                                   
| attached_servers               | []                                    |                                                                                                                                                                   
| attachment_ids                 | []                                    |                                                                                                                                                                   
| availability_zone              | nova                                  |                                                                                                                                                                   
| bootable                       | false                                 |                                                                                                                                                                   
| consistencygroup_id            | None                                  |                                                                                                                                                                   
| created_at                     | 2019-07-14T09:16:16.000000            |                                                                                                                                                                   
| description                    | None                                  |                                                                                                                                                                   
| encrypted                      | True                                  |                                                                                                                                                                   
| id                             | 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0  |                                                                                                                                                                   
| metadata                       |                                       |                                                                                                                                                                   
| migration_status               | success                               |                                                                                                                                                                   
| multiattach                    | False                                 |                                                                                                                                                                   
| name                           | Enc-vol                               |                                                                                                                                                                   
| os-vol-host-attr:host          | hostgroup@tripleo_iscsi#tripleo_iscsi |                                                                                                                                                                   
| os-vol-mig-status-attr:migstat | success                               |                                                                                                                                                                   
| os-vol-mig-status-attr:name_id | 558b1fa0-21fb-4d80-afff-f411d5717bcb  |                                                                                                                                                                   
| os-vol-tenant-attr:tenant_id   | 5850830f52774e0dab7b7b8e508b6a56      |                                                                                                                                                                   
| replication_status             | None                                  |
| size                           | 1                                     |
| snapshot_id                    | None                                  |
| source_volid                   | None                                  |
| status                         | available                             |
| updated_at                     | 2019-07-14T09:18:32.000000            |
| user_id                        | 8fe58395864a43158a847d1a9ffd4e9d      |
| volume_type                    | enc_ma_lvm                            |
+--------------------------------+---------------------------------------+

Notice volume is available (bad), don't want to know what state data was left in. Volume type changed from enc_lvm to -> enc_ma_lvm  

Actual results:
Volume is migrated despite not allowing such a resulting type 

Expected results:
Please block such operations, before a customer opens a bug about lost/bad data. 
 

Additional info:
No logs were add as there is no error to be found in logs.

Comment 1 Tzach Shefi 2019-07-14 09:41:19 UTC
FYI might be a case for another bug, when we try to attach the resulting (none supported) Enc MA volume

(overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 auto
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 |
| serverId | a4946e59-08e2-41a9-b00a-d052a6a33d5e |
| tag      | -                                    |
| volumeId | 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 |
+----------+--------------------------------------+


We get stuck with
| 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 | attaching | Enc-vol          | 1    | enc_ma_lvm  | false    |             |

Attaching state till we timeout is reached, then return to available state. 

| 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 | available | Enc-vol          | 1    | enc_ma_lvm  | false    |             |


Would we not allow retype this wouldn't be an issue, if search for attach problems I'm sure I'll find them.
Yet the common user won't understand why attach is failing.

Comment 3 Tzach Shefi 2020-02-24 14:22:01 UTC
John, 
This still looks broken, retested migration worked when it should have been blocked/flagged. 

openstack-cinder-14.0.4-0.20200107100455.a59c01e.el8ost.noarch

Create both luks and enc_ma_lvm volume types:
(overcloud) [stack@undercloud-0 ~]$ cinder extra-specs-list
+--------------------------------------+------------+----------------------------------------------------------------------+
| ID                                   | Name       | extra_specs                                                          |
+--------------------------------------+------------+----------------------------------------------------------------------+
| ae898762-503b-46b0-a2ce-25c14fcccd40 | enc_ma_lvm | {'volume_backend_name': 'tripleo_iscsi', 'multiattach': '<is> True'} |
| d58cbd3f-0b38-415b-b097-7213a18d9e88 | tripleo    | {}                                                                   |
| dc5f07c0-ea57-42bd-9ce8-b964c0f05a0b | luks       | {'volume_backend_name': 'tripleo_iscsi'}                             |
+--------------------------------------+------------+----------------------------------------------------------------------+

Create an encrypted volume:

(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --volume-type luks --name luks1
+--------------------------------+---------------------------------------+
| Property                       | Value                                 |
+--------------------------------+---------------------------------------+
| attachments                    | []                                    |
| availability_zone              | nova                                  |
| bootable                       | false                                 |
| consistencygroup_id            | None                                  |
| created_at                     | 2020-02-24T14:00:14.000000            |
| description                    | None                                  |
| encrypted                      | True                                  |
| id                             | cb668f20-0ed0-47dd-8255-59686cc82e49  |
| metadata                       | {}                                    |
| migration_status               | None                                  |
| multiattach                    | False                                 |-> as expected false.
| name                           | luks1                                 |
| os-vol-host-attr:host          | hostgroup@tripleo_iscsi#tripleo_iscsi |
| os-vol-mig-status-attr:migstat | None                                  |
| os-vol-mig-status-attr:name_id | None                                  |
| os-vol-tenant-attr:tenant_id   | 59e03fc4326145c3ab3182ab0d243bc7      |
| replication_status             | None                                  |
| size                           | 1                                     |
| snapshot_id                    | None                                  |
| source_volid                   | None                                  |
| status                         | available                             |
| updated_at                     | 2020-02-24T14:00:14.000000            |
| user_id                        | 83e5fb87eefb4f08a936e7f9d522e8d2      |
| volume_type                    | luks                                  | -> notice luks
+--------------------------------+---------------------------------------+


Try to migrate luks to enc_ma_lvm 
(overcloud) [stack@undercloud-0 ~]$ cinder retype cb668f20-0ed0-47dd-8255-59686cc82e49 enc_ma_lvm --migration-policy on-demand   
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+   
| ID                                   | Status    | Name  | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+
| 1e3962c5-c18a-40a8-9607-1198b5f72e6d | available | luks1 | 1    | enc_ma_lvm  | false    |             |   During migration
| cb668f20-0ed0-47dd-8255-59686cc82e49 | retyping  | luks1 | 1    | luks        | false    |             |

(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+
| ID                                   | Status    | Name  | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+
| cb668f20-0ed0-47dd-8255-59686cc82e49 | available | luks1 | 1    | enc_ma_lvm  | false    |             |  -> not good not supported should have been blocked/failed. 




(overcloud) [stack@undercloud-0 ~]$ cinder show cb668f20-0ed0-47dd-8255-59686cc82e49
+--------------------------------+---------------------------------------+
| Property                       | Value                                 |
+--------------------------------+---------------------------------------+
| attached_servers               | []                                    |
| attachment_ids                 | []                                    |
| availability_zone              | nova                                  |
| bootable                       | false                                 |
| consistencygroup_id            | None                                  |
| created_at                     | 2020-02-24T14:00:14.000000            |
| description                    | None                                  |
| encrypted                      | True                                  | -> Nice to still enc.
| id                             | cb668f20-0ed0-47dd-8255-59686cc82e49  |
| metadata                       |                                       |
| migration_status               | success                               | -> yet this says success 
| multiattach                    | False                                 |  -> and it ain't MA which is great as we don't support enc_ma volumes. 
| name                           | luks1                                 |
| os-vol-host-attr:host          | hostgroup@tripleo_iscsi#tripleo_iscsi |
| os-vol-mig-status-attr:migstat | success                               |
| os-vol-mig-status-attr:name_id | 1e3962c5-c18a-40a8-9607-1198b5f72e6d  |
| os-vol-tenant-attr:tenant_id   | 59e03fc4326145c3ab3182ab0d243bc7      |
| replication_status             | None                                  |
| size                           | 1                                     |
| snapshot_id                    | None                                  |
| source_volid                   | None                                  |
| status                         | available                             |
| updated_at                     | 2020-02-24T14:08:19.000000            |
| user_id                        | 83e5fb87eefb4f08a936e7f9d522e8d2      |
| volume_type                    | enc_ma_lvm                            | -> however changed from luks to enc_ma_lvm which isn't good.  
+--------------------------------+---------------------------------------+

Maybe this "works" cause both volume types are backed by LVM? 
None the less it should fail or least not switch to enc_ma_lvm if it aint a ma volume.

Comment 7 errata-xmlrpc 2020-03-05 11:54:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0712


Note You need to log in before you can comment on or make changes to this bug.