Bug 1744112

Summary: [OSP13] We don't block retyping of an encrypted volume to multi-attach encrypted volume
Product: Red Hat OpenStack Reporter: Tzach Shefi <tshefi>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED ERRATA QA Contact: Tzach Shefi <tshefi>
Severity: medium Docs Contact: Chuck Copello <ccopello>
Priority: medium    
Version: 13.0 (Queens)CC: abishop, acanan, ccopello, eharney, jobernar, marjones, mgarciac
Target Milestone: z11Keywords: Reopened, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-12.0.10-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1729755 Environment:
Last Closed: 2020-03-13 11:09:35 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1729755    
Bug Blocks:    

Description Tzach Shefi 2019-08-21 11:39:27 UTC
+++ This bug was initially created as a clone of Bug #1729755 +++

We backported MA RBD to OSP13, hit same issue on OSP13. 


Description of problem:  We currently rightfully don't allow/support creation of encrypted multi-attach volume:

$ cinder  create 1 --volume-type enc_ma_lvm --name enc_ma_vol
ERROR: Invalid volume: Multiattach cannot be used with encrypted volumes. (HTTP 400) (Request-ID: req-8859dbc6-646f-4cb0-8a0d-419a019ffae1)

However if you create an encrypted (none MA) volume and retype it to a none supported multi-attached encrypted volume it would ideally fail/warn. Yet operation is executed despite the none supported resulting volume, this IMHO should be blocked. 


Yes I know retype doesn't validate/check backends before it accepts what looks like a valid request, which explains why no error is reported and operation is preformed. Call it wishful QE thinking I'd like to see such operations validated and blocked if end result isn't supported.    


Version-Release number of selected component (if applicable):
puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch
openstack-cinder-14.0.1-0.20190607000407.23d1a72.el8ost.noarch
python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch
python3-cinder-14.0.1-0.20190607000407.23d1a72.el8ost.noarch


How reproducible: 
Every time


Steps to Reproduce:
1. Create both backends

| 04c5d0b5-3ed4-4ba1-b74d-7bcd8b2b4540 | enc_lvm    | {'volume_backend_name': 'tripleo_iscsi'}  

cinder encryption-type-create --cipher aes-xts-plain64 --key_size 256 --control_location front-end enc_lvm  nova.volume.encryptors.luks.LuksEncryptor

| 50c3fb05-1c34-45e6-b40c-44296b3b90ad | enc_ma_lvm | {'volume_backend_name': 'tripleo_iscsi', 'multiattach': '<is> True'} |

 cinder encryption-type-create --cipher aes-xts-plain64 --key_size 256 --control_location front-end enc_ma_lvm  nova.volume.encryptors.luks.LuksEncryptor


2. Create an enc lvm volume:

(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --volume-type enc_lvm --name Enc-vol 
+--------------------------------+---------------------------------------+
| Property                       | Value                                 |
+--------------------------------+---------------------------------------+
| attachments                    | []                                    |
| availability_zone              | nova                                  |
| bootable                       | false                                 |
| consistencygroup_id            | None                                  |
| created_at                     | 2019-07-14T09:16:16.000000            |
| description                    | None                                  |
| encrypted                      | True                                  |
| id                             | 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0  |
| metadata                       | {}                                    |
| migration_status               | None                                  |
| multiattach                    | False                                 |
| name                           | Enc-vol                               |
| os-vol-host-attr:host          | hostgroup@tripleo_iscsi#tripleo_iscsi |
| os-vol-mig-status-attr:migstat | None                                  |
| os-vol-mig-status-attr:name_id | None                                  |
| os-vol-tenant-attr:tenant_id   | 5850830f52774e0dab7b7b8e508b6a56      |
| replication_status             | None                                  |
| size                           | 1                                     |
| snapshot_id                    | None                                  |
| source_volid                   | None                                  |
| status                         | available                             |
| updated_at                     | 2019-07-14T09:16:17.000000            |
| user_id                        | 8fe58395864a43158a847d1a9ffd4e9d      |
| volume_type                    | enc_lvm                               |
+--------------------------------+---------------------------------------+

Volume before migration
(overcloud) [stack@undercloud-0 ~]$ cinder list

| 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 | available | Enc-vol          | 1    | enc_lvm     | false    |             |


3. Retype the volume to none supported enc_ma_lvm type:

(overcloud) [stack@undercloud-0 ~]$ cinder retype 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 enc_ma_lvm --migration-policy on-demand    

!error returned :( 
                                                                                                           
(overcloud) [stack@undercloud-0 ~]$ cinder show 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0                                                                                                                                                         
+--------------------------------+---------------------------------------+                                                                                                                                                                   
| Property                       | Value                                 |                                                                                                                                                                   
+--------------------------------+---------------------------------------+                                                                                                                                                                   
| attached_servers               | []                                    |                                                                                                                                                                   
| attachment_ids                 | []                                    |                                                                                                                                                                   
| availability_zone              | nova                                  |                                                                                                                                                                   
| bootable                       | false                                 |                                                                                                                                                                   
| consistencygroup_id            | None                                  |                                                                                                                                                                   
| created_at                     | 2019-07-14T09:16:16.000000            |                                                                                                                                                                   
| description                    | None                                  |                                                                                                                                                                   
| encrypted                      | True                                  |                                                                                                                                                                   
| id                             | 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0  |                                                                                                                                                                   
| metadata                       |                                       |                                                                                                                                                                   
| migration_status               | success                               |                                                                                                                                                                   
| multiattach                    | False                                 |                                                                                                                                                                   
| name                           | Enc-vol                               |                                                                                                                                                                   
| os-vol-host-attr:host          | hostgroup@tripleo_iscsi#tripleo_iscsi |                                                                                                                                                                   
| os-vol-mig-status-attr:migstat | success                               |                                                                                                                                                                   
| os-vol-mig-status-attr:name_id | 558b1fa0-21fb-4d80-afff-f411d5717bcb  |                                                                                                                                                                   
| os-vol-tenant-attr:tenant_id   | 5850830f52774e0dab7b7b8e508b6a56      |                                                                                                                                                                   
| replication_status             | None                                  |
| size                           | 1                                     |
| snapshot_id                    | None                                  |
| source_volid                   | None                                  |
| status                         | available                             |
| updated_at                     | 2019-07-14T09:18:32.000000            |
| user_id                        | 8fe58395864a43158a847d1a9ffd4e9d      |
| volume_type                    | enc_ma_lvm                            |
+--------------------------------+---------------------------------------+

Notice volume is available (bad), don't want to know what state data was left in. Volume type changed from enc_lvm to -> enc_ma_lvm  

Actual results:
Volume is migrated despite not allowing such a resulting type 

Expected results:
Please block such operations, before a customer opens a bug about lost/bad data. 
 

Additional info:
No logs were add as there is no error to be found in logs.

--- Additional comment from Tzach Shefi on 2019-07-14 09:41:19 UTC ---

FYI might be a case for another bug, when we try to attach the resulting (none supported) Enc MA volume

(overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 auto
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 |
| serverId | a4946e59-08e2-41a9-b00a-d052a6a33d5e |
| tag      | -                                    |
| volumeId | 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 |
+----------+--------------------------------------+


We get stuck with
| 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 | attaching | Enc-vol          | 1    | enc_ma_lvm  | false    |             |

Attaching state till we timeout is reached, then return to available state. 

| 723431a2-38c3-4c9d-b87a-8dc4dd9f9da0 | available | Enc-vol          | 1    | enc_ma_lvm  | false    |             |


Would we not allow retype this wouldn't be an issue, if search for attach problems I'm sure I'll find them.
Yet the common user won't understand why attach is failing.

Comment 1 Alan Bishop 2019-08-27 12:39:55 UTC
@Jon, assigning to you as you have the OSP-15 version of this bz. Feel free to ask for help getting the fix backported to the earlier release.

Comment 4 Tzach Shefi 2020-03-09 10:18:22 UTC
Tested on:
openstack-cinder-12.0.10-2.el7ost.noarch

Created two backends:

(overcloud) [stack@puma51 ~]$ cinder extra-specs-list
+--------------------------------------+---------+----------------------------------------------------------------------+
| ID                                   | Name    | extra_specs                                                          |
+--------------------------------------+---------+----------------------------------------------------------------------+
| 5d5caec0-738b-421e-aa6b-5f9df061d3cb | enc_ma  | {'volume_backend_name': 'tripleo_iscsi', 'multiattach': '<is> True'} |
| 6b187af5-fc0f-4eb6-8f9f-fea97ede359b | enc     | {'volume_backend_name': 'tripleo_iscsi'}                             |
| ab130a7e-bf02-40b0-b551-c84aee7f40dc | tripleo | {}                                                                   |
                                                                                                                                                                                       

Create an encrypted volume from an image 
(overcloud) [stack@puma51 ~]$ cinder create 5 --volume-type enc --name EncVol1
(overcloud) [stack@puma51 ~]$ cinder show EncVol1
+--------------------------------+---------------------------------------+
| Property                       | Value                                 |
+--------------------------------+---------------------------------------+
| attached_servers               | []                                    |
| attachment_ids                 | []                                    |
| availability_zone              | nova                                  |
| bootable                       | false                                 |
| consistencygroup_id            | None                                  |
| created_at                     | 2020-03-09T09:45:39.000000            |
| description                    | None                                  |
| encrypted                      | True                                  |
| id                             | 42b2ba66-6938-4c2d-a464-a35d534160a3  |
| metadata                       |                                       |
| migration_status               | None                                  |
| multiattach                    | False                                 |
| name                           | EncVol1                               |
| os-vol-host-attr:host          | hostgroup@tripleo_iscsi#tripleo_iscsi |
| os-vol-mig-status-attr:migstat | None                                  |
| os-vol-mig-status-attr:name_id | None                                  |
| os-vol-tenant-attr:tenant_id   | c1479695d5cf43b1a854f600226dad39      |
| replication_status             | None                                  |
| size                           | 5                                     |
| snapshot_id                    | None                                  |
| source_volid                   | None                                  |
| status                         | available                             |
| updated_at                     | 2020-03-09T09:45:42.000000            |
| user_id                        | cb29148baa654f739c78ed4f92cb6e2b      |
| volume_type                    | enc                                   |
+--------------------------------+---------------------------------------+


Now lets try to retype this to enc_ma it should fail

(overcloud) [stack@puma51 ~]$ cinder retype 42b2ba66-6938-4c2d-a464-a35d534160a3 enc_ma --migration-policy on-demand  

Retype still happens when  it should be blocked

(overcloud) [stack@puma51 ~]$ cinder list
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| ID                                   | Status    | Name      | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| 42b2ba66-6938-4c2d-a464-a35d534160a3 | retyping  | EncVol1   | 5    | enc         | false    |                                      |
| d97dce16-d2e1-4a00-b1ee-24c2c8dd2073 | available | EncVol1   | 5    | enc_ma      | false    |                                      |
 
(overcloud) [stack@puma51 ~]$ cinder list
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| ID                                   | Status    | Name      | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| 42b2ba66-6938-4c2d-a464-a35d534160a3 | available | EncVol1   | 5    | enc_ma      | false    |                                      |
| d97dce16-d2e1-4a00-b1ee-24c2c8dd2073 | deleting  | EncVol1   | 5    | enc_ma      | false    |                                      |

(overcloud) [stack@puma51 ~]$ cinder list
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| ID                                   | Status    | Name      | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| 42b2ba66-6938-4c2d-a464-a35d534160a3 | available | EncVol1   | 5    | enc_ma      | false    |                                      |


Failed to verify, vol shouldn't migrate from enc to none supported enc_ma
(overcloud) [stack@puma51 ~]$ cinder show 42b2ba66-6938-4c2d-a464-a35d534160a3
+--------------------------------+---------------------------------------+
| Property                       | Value                                 |
+--------------------------------+---------------------------------------+
| attached_servers               | []                                    |
| attachment_ids                 | []                                    |
| availability_zone              | nova                                  |
| bootable                       | false                                 |
| consistencygroup_id            | None                                  |
| created_at                     | 2020-03-09T09:45:39.000000            |
| description                    | None                                  |
| encrypted                      | True                                  |  So volume is encrypted, but look down multipath is false. 
| id                             | 42b2ba66-6938-4c2d-a464-a35d534160a3  |
| metadata                       |                                       |
| migration_status               | success                               | -> migrated "successfully"  when it should have been blocked. 
| multiattach                    | False                                 |  
| name                           | EncVol1                               |
| os-vol-host-attr:host          | hostgroup@tripleo_iscsi#tripleo_iscsi |
| os-vol-mig-status-attr:migstat | success                               |
| os-vol-mig-status-attr:name_id | d97dce16-d2e1-4a00-b1ee-24c2c8dd2073  |
| os-vol-tenant-attr:tenant_id   | c1479695d5cf43b1a854f600226dad39      |
| replication_status             | None                                  |
| size                           | 5                                     |
| snapshot_id                    | None                                  |
| source_volid                   | None                                  |
| status                         | available                             |  -> it might very well be but not sure what's inside if still valid data. 
| updated_at                     | 2020-03-09T10:09:57.000000            |
| user_id                        | cb29148baa654f739c78ed4f92cb6e2b      |
| volume_type                    | enc_ma                                |  -> not good changed from enc to enc_ma which is supported. 
+--------------------------------+---------------------------------------+



Just for reference here I wanted to create/show a enc_ma volume just to check that ma = true. 
Things got interesting, I was blocked from creating such a volume which is great:   


(overcloud) [stack@puma51 ~]$ cinder create 1 --volume-type enc_ma  --name enc_maVol1
ERROR: Invalid volume: Multiattach cannot be used with encrypted volumes. (HTTP 400) (Request-ID: req-ee8a1c1c-4a2d-4890-baaf-bf5f4c0cb995)


However again I wasn't blocked from migrating a supported enc volume to this none supported enc_ma.

Comment 7 errata-xmlrpc 2020-03-10 11:25:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0764