Bug 1987104

Summary:	[OSP16.1] RBD volume created from snapshot on encrypted volume cannot be used
Product:	Red Hat OpenStack	Reporter:	Sofia Enriquez <senrique>
Component:	openstack-cinder	Assignee:	Sofia Enriquez <senrique>
Status:	CLOSED ERRATA	QA Contact:	Tzach Shefi <tshefi>
Severity:	high	Docs Contact:	RHOS Documentation Team <rhos-docs>
Priority:	urgent
Version:	16.1 (Train)	CC:	abishop, gregraka, ltoscano, mburns, ndeevy, pgrist, slinaber, spower
Target Milestone:	z7	Keywords:	Triaged, ZStream
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-cinder-15.4.0-1.20210713144325.el8ost	Doc Type:	Bug Fix
Doc Text:	Before this update, creating a volume from a snapshot of an encrypted volume could result in an unusable volume. When the destination volume is the same size as the source volume, creating an encrypted volume from a snapshot of an encrypted volume truncated the data in the new volume, which caused a size discrepancy. + With this update, the RBD back end accounts for the encryption header and does not truncate the data so that creating a volume from a snapshot of an encrypted volume does not cause the error.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-12-09 20:20:17 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1772531
Bug Blocks:

Description Sofia Enriquez 2021-07-28 21:41:58 UTC

This bug was initially created as a copy of Bug #1772531

I am copying this bug because:

Upstream bug:
https://bugs.launchpad.net/cinder/+bug/1852168

Creating a volume from a snapshot of an encrypted volume may result in an unusable volume.
Detectable only by looking at behaviour inside the instance upon attach.

It's a RBD bug.

When creating an encrypted volume from a snapshot of an encrypted volume, if the amount
of data in the original volume at the time the snapshot was created is very close to the
gibibyte boundary given by the volume's size, it is possible for the data in the new
volume to be silently truncated.

Usually the source volume would be the same size or smaller than the destination volume
and they must share the same volume-type.

In particular RBD workflow would be something like this:
A source luks volume would be 1026M, we write some data and create a snap from it. We
like to create a new luks volume from a snapshot so the create_volume_from_snapshot()
method performs a RBD clone first and then a resize if needed.

If the desired size of the destination luks volume is 1G the
create_volume_from_snapshot() won't perform any resize and will be 1026M as the parent.
This solves bug https://bugs.launchpad.net/cinder/+bug/1922408 because we don't force
it to resize and because of that we don't truncate the data anymore.

The second case scenario is when we would like to increase the size of the destination
volume. As far as I test it this won't face the encryption header problem but we still
need to calculate the difference size to provide the size that the user is expecting.

Comment 12 Tzach Shefi 2021-09-19 11:34:24 UTC

Verified on:
openstack-cinder-15.4.0-1.20210713144325.el8ost.noarch

On a Ceph backed deployment, I duplicated same verification steps from 16.2 clone. 


1. configure an encrypted volume type:

(overcloud) [stack@undercloud-0 ~]$ cinder type-create LUKS
+--------------------------------------+------+-------------+-----------+
| ID                                   | Name | Description | Is_Public |
+--------------------------------------+------+-------------+-----------+
| 364d0a80-13a0-4f0c-9f3a-ca8bb8698211 | LUKS | -           | True      |
+--------------------------------------+------+-------------+-----------+

(overcloud) [stack@undercloud-0 ~]$ cinder encryption-type-create --cipher aes-xts-plain64 --key_size 256   --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
| Volume Type ID                       | Provider                                  | Cipher          | Key Size | Control Location |
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
| 364d0a80-13a0-4f0c-9f3a-ca8bb8698211 | nova.volume.encryptors.luks.LuksEncryptor | aes-xts-plain64 | 256      | front-end        |
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+

(overcloud) [stack@undercloud-0 ~]$ cinder type-key LUKS set volume_backend_name=tripleo_ceph



2. Create an empty encrypted volume:
(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --volume-type LUKS --name enc_vol1
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2021-09-19T11:07:28.000000           |
| description                    | None                                 |
| encrypted                      | True                                 |
| id                             | 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | enc_vol1                             |
| os-vol-host-attr:host          | hostgroup@tripleo_ceph#tripleo_ceph  |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | a347a2f81d444e9980d84d0682f30c22     |
| replication_status             | None                                 |
| size                           | 2                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | 2021-09-19T11:07:28.000000           |
| user_id                        | 534e728b258948e7a01b55461f057aed     |
| volume_type                    | LUKS                                 |
+--------------------------------+--------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+----------+------+-------------+----------+-------------+
| ID                                   | Status    | Name     | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+----------+------+-------------+----------+-------------+
| 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295 | available | enc_vol1 | 2    | LUKS        | false    |             |
+--------------------------------------+-----------+----------+------+-------------+----------+-------------+


3. Boot an instance, attach enc volume to it:
\(overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295
+-----------------------+--------------------------------------+
| Property              | Value                                |
+-----------------------+--------------------------------------+
| delete_on_termination | False                                |
| device                | /dev/vdb                             |
| id                    | 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295 |
| serverId              | 2c68cd06-98ad-4065-80b0-0e2df74bf5b2 |
| tag                   | -                                    |
| volumeId              | 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295 |
+-----------------------+--------------------------------------+


(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+--------+----------+------+-------------+----------+--------------------------------------+
| ID                                   | Status | Name     | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+--------+----------+------+-------------+----------+--------------------------------------+
| 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295 | in-use | enc_vol1 | 2    | LUKS        | false    | 2c68cd06-98ad-4065-80b0-0e2df74bf5b2 |
+--------------------------------------+--------+----------+------+-------------+----------+--------------------------------------+


4. ssh into instance, mount enc volume fill it with data:
(overcloud) [stack@undercloud-0 ~]$ ssh cirros.0.245
Warning: Permanently added '10.0.0.245' (ECDSA) to the list of known hosts.
$ sudo -i
# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda     253:0    0    1G  0 disk 
|-vda1  253:1    0 1015M  0 part /
`-vda15 253:15   0    8M  0 part 
vdb     253:16   0    2G  0 disk 
# mkdir mnt
# mkfs.ext4 /dev/vdb
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 524288 4k blocks and 131072 inodes
Filesystem UUID: 51b7800a-c492-4366-9b60-d798b374251d
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

# mount /dev/vdb mnt/
# cd mnt/
# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda     253:0    0    1G  0 disk 
|-vda1  253:1    0 1015M  0 part /
`-vda15 253:15   0    8M  0 part 
vdb     253:16   0    2G  0 disk /root/mnt
# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev                    240.1M         0    240.1M   0% /dev
/dev/vda1               978.9M     23.9M    914.2M   3% /
tmpfs                   244.2M         0    244.2M   0% /dev/shm
tmpfs                   244.2M     92.0K    244.1M   0% /run
/dev/vdb                  1.9G      3.0M      1.8G   0% /root/mnt
# dd if=/dev/urandom of=/root/mnt/data_file.bin  bs=10M count=184
184+0 records in
184+0 records out

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev                    240.1M         0    240.1M   0% /dev
/dev/vda1               978.9M     23.9M    914.2M   3% /
tmpfs                   244.2M         0    244.2M   0% /dev/shm
tmpfs                   244.2M     92.0K    244.1M   0% /run
/dev/vdb                  1.9G      1.8G         0 100% /root/mnt  -> filled to the max. 


4. Lets create a snapshot of the volume:
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295 --force --name EncVol1Snap
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2021-09-19T11:24:06.129263           |
| description | None                                 |
| id          | ec20f907-5db3-4180-ae85-621f4ae1942b |
| metadata    | {}                                   |
| name        | EncVol1Snap                          |
| size        | 2                                    |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295 |
+-------------+--------------------------------------+


(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-list
+--------------------------------------+--------------------------------------+-----------+-------------+------+
| ID                                   | Volume ID                            | Status    | Name        | Size |
+--------------------------------------+--------------------------------------+-----------+-------------+------+
| ec20f907-5db3-4180-ae85-621f4ae1942b | 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295 | available | EncVol1Snap | 2    |
+--------------------------------------+--------------------------------------+-----------+-------------+------+

-5.  Create a new encrypted volume from snap:

(overcloud) [stack@undercloud-0 ~]$ cinder create 2 --snapshot-id=ec20f907-5db3-4180-ae85-621f4ae1942b --name enc_vol2_from_snap
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2021-09-19T11:28:17.000000           |
| description                    | None                                 |
| encrypted                      | True                                 |
| id                             | 14b7a6b5-fbb9-4cb2-9236-c128425acdd2 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | enc_vol2_from_snap                   |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | a347a2f81d444e9980d84d0682f30c22     |
| replication_status             | None                                 |
| size                           | 2                                    |
| snapshot_id                    | ec20f907-5db3-4180-ae85-621f4ae1942b |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | 534e728b258948e7a01b55461f057aed     |
| volume_type                    | LUKS                                 |
+--------------------------------+--------------------------------------+



(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+
| ID                                   | Status    | Name               | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+
| 0574d4ca-f9f1-4c9f-9a7d-ac36d8388295 | in-use    | enc_vol1           | 2    | LUKS        | false    | 2c68cd06-98ad-4065-80b0-0e2df74bf5b2 |
| 14b7a6b5-fbb9-4cb2-9236-c128425acdd2 | available | enc_vol2_from_snap | 2    | LUKS        | false    |                                      |
+--------------------------------------+-----------+--------------------+------+-------------+----------+--------------------------------------+


6. Attach volume to instance:

(overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 14b7a6b5-fbb9-4cb2-9236-c128425acdd2
+-----------------------+--------------------------------------+
| Property              | Value                                |
+-----------------------+--------------------------------------+
| delete_on_termination | False                                |
| device                | /dev/vdc                             |
| id                    | 14b7a6b5-fbb9-4cb2-9236-c128425acdd2 |
| serverId              | 2c68cd06-98ad-4065-80b0-0e2df74bf5b2 |
| tag                   | -                                    |
| volumeId              | 14b7a6b5-fbb9-4cb2-9236-c128425acdd2 |
+-----------------------+--------------------------------------+


7. Mount cloned vol compare content:
## lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda     253:0    0    1G  0 disk 
|-vda1  253:1    0 1015M  0 part /
`-vda15 253:15   0    8M  0 part 
vdb     253:16   0    2G  0 disk /root/mnt
vdc     253:32   0    2G  0 disk 

## mkdir mnt2
# mount /dev/vdc mnt2/
# ls mnt2/
data_file.bin  lost+found
# df -h
FFilesystem           1K-blocks      Used Available Use% Mounted on
/dev                    245908         0    245908   0% /dev
/dev/vda1              1002422     24513    936092   3% /
tmpfs                   250076         0    250076   0% /dev/shm
tmpfs                   250076        92    249984   0% /run
/dev/vdb               1998672   1887240         0 100% /root/mnt
/dev/vdc               1998672   1887236         0 100% /root/mnt/mnt2


# diff mnt/data_file.bin mnt/data_file.bin   -> both files are  the same.
As we were able to create an encrypted cloned volume from an encrypted source volume,  good to verify.

Comment 27 errata-xmlrpc 2021-12-09 20:20:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.7 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3762