Bug 1956762
| Summary: | [OSP13][cinder] Issue with volume created from the snapshot of an encrypted volume where cinder backend is ceph | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Rohini Diwakar <rdiwakar> |
| Component: | openstack-cinder | Assignee: | Sofia Enriquez <senrique> |
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> |
| Severity: | high | Docs Contact: | RHOS Documentation Team <rhos-docs> |
| Priority: | high | ||
| Version: | 13.0 (Queens) | CC: | astillma, astupnik, dmaley, eharney, gcharot, jvisser, pgrist, senrique, slinaber, spower, udesale |
| Target Milestone: | async | Keywords: | Triaged, ZStream |
| Target Release: | 13.0 (Queens) | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-cinder-12.0.10-25.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: |
Usually the source volume would be the same size or smaller than the destination volume and they must share the same volume-type. In particular, when the destination volume is the same size as the source volume, creating an encrypted volume from a snapshot of an encrypted volume truncates the data in the new volume. In order to fix this the RBD workflow would be something like this: A source luks volume would be 1026M, we write some data and create a snap from it. We like to create a new luks volume from a snapshot so the create_volume_from_snapshot() method performs a RBD clone first and then a resize if needed. In addition the _clone() method creates a clone (copy-on-write child) of the parent snapshot. Object size will be identical to that of the parent image unless specified (we don't in cinder) so size will be the same as the parent snapshot. If the desired size of the destination luks volume is 1G the create_volume_from_snapshot() won't perform any resize and will be 1026M as the parent. This solves the bug because we don't force it to resize and because of that we don't truncate the data any more. The second case scenario is when we would like to increase the size of the destination volume. As far as I can tell this won't face the encryption header problem but we still need to calculate the difference size to provide the size that the user is expecting. That's why the fix proposed calculate the new_size based on: size difference = desired size - size of source volume new size = current size + size difference.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-12-15 15:57:04 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1772531 | ||
| Bug Blocks: | |||
|
Description
Rohini Diwakar
2021-05-04 11:28:23 UTC
Hello,
37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb: The source volume
bcb95a0a-a0f3-4634-b84c-c02253269c40: The final volume which is created from snapshot of source volume.
[root@svl-ceph-110 ~]# rbd info -p cloud-svldev-2-cinder volume-37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb
rbd image 'volume-37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb':
size 1.00GiB in 257 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.eba40f6b8b4567
format: 2
features: layering, exclusive-lock, object-map
flags:
create_timestamp: Tue May 4 04:49:01 2021
[root@svl-ceph-110 ~]# rbd info -p cloud-svldev-2-cinder volume-bcb95a0a-a0f3-4634-b84c-c02253269c40
rbd image 'volume-bcb95a0a-a0f3-4634-b84c-c02253269c40':
size 1GiB in 256 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.ec05e94113f7ca
format: 2
features: layering, exclusive-lock, object-map
flags:
create_timestamp: Tue May 4 04:56:43 2021
parent: cloud-svldev-2-cinder/volume-37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb@snapshot-9d20be7a-bb48-4e78-be55-8c4e7320ba6d
overlap: 1GiB
After exporting the rbd volumes we can see the difference in their size
[murg@svldev2-cfc-a-mgmt-001 ~]$ qemu-img info ./source.img
image: ./source.img
file format: luks
virtual size: 1.0G (1073741824 bytes)
disk size: 1.0G
encrypted: yes
Format specific information:
ivgen alg: plain64
hash alg: sha256
cipher alg: aes-256
uuid: 9f4cbf11-435f-4247-bc05-76090edd9872
cipher mode: xts
slots:
[0]:
active: true
iters: 123186
key offset: 4096
stripes: 4000
[1]:
active: false
key offset: 262144
[2]:
active: false
key offset: 520192
[3]:
active: false
key offset: 778240
[4]:
active: false
key offset: 1036288
[5]:
active: false
key offset: 1294336
[6]:
active: false
key offset: 1552384
[7]:
active: false
key offset: 1810432
payload offset: 2068480
master key iters: 31107
[murg@svldev2-cfc-a-mgmt-001 ~]$ qemu-img info ./final.img
image: ./final.img
file format: luks
virtual size: 1.0G (1071673344 bytes)
disk size: 1.0G
encrypted: yes
Format specific information:
ivgen alg: plain64
hash alg: sha256
cipher alg: aes-256
uuid: 9f4cbf11-435f-4247-bc05-76090edd9872
cipher mode: xts
slots:
[0]:
active: true
iters: 123186
key offset: 4096
stripes: 4000
[1]:
active: false
key offset: 262144
[2]:
active: false
key offset: 520192
[3]:
active: false
key offset: 778240
[4]:
active: false
key offset: 1036288
[5]:
active: false
key offset: 1294336
[6]:
active: false
key offset: 1552384
[7]:
active: false
key offset: 1810432
payload offset: 2068480
master key iters: 31107
For the sake of completeness I'm adding a summary of the problem and an explanation of the fix proposed upstream. Usually the source volume would be the same size or smaller than the destination volume and they must share the same volume-type. Current RBD workflow: A source luks volume would be 1026M, we write some data and create a snap from it. In order to create a new luks volume from that snapshot the create_volume_from_snapshot() method performs a RBD clone first and then a resize. In particular, the _clone() method creates a clone (copy-on-write child) of the parent snapshot. The size will be identical to that of the parent unless specified (we don't in cinder) so size will be the same as the parent snapshot. The first scenario is when destination and source volume are the same size. This fix proposed that create_volume_from_snapshot() won't perform any resize. In this way the data won't truncate. The second case scenario is when we would like to increase the size of the destination volume. As far as I can tell this won't face the encryption header problem but we still need to calculate the difference size to provide the size that the user is expecting. That's why the fix proposed calculate the new_size based on : size difference = desired size - size of source volume new size = current size + size difference Verified on:
openstack-cinder-12.0.10-25.el7ost
On a ceph backed deployment, lets follow the reproduce steps:
1. Create an encrypted vol type and a volume.
(overcloud) [stack@undercloud-0 ~]$ cinder type-create LUKS
+--------------------------------------+------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+------+-------------+-----------+
| 9dc1ea58-97d8-434d-972a-fa2f20bfd3df | LUKS | - | True |
+--------------------------------------+------+-------------+-----------+
(overcloud) [stack@undercloud-0 ~]$ cinder encryption-type-create --cipher aes-xts-plain64 --key_size 256 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
| Volume Type ID | Provider | Cipher | Key Size | Control Location |
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
| 9dc1ea58-97d8-434d-972a-fa2f20bfd3df | nova.volume.encryptors.luks.LuksEncryptor | aes-xts-plain64 | 256 | front-end |
+--------------------------------------+-------------------------------------------+-----------------+----------+------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder type-key LUKS set volume_backend_name=tripleo_ceph
(overcloud) [stack@undercloud-0 ~]$ cinder extra-specs-list
+--------------------------------------+---------+-----------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+---------+-----------------------------------------+
| 84078b67-659c-47fb-9ea0-b5310f259b55 | tripleo | {} |
| 9dc1ea58-97d8-434d-972a-fa2f20bfd3df | LUKS | {'volume_backend_name': 'tripleo_ceph'} |
+--------------------------------------+---------+-----------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --volume-type LUKS --name EncVolA
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-10-17T06:57:49.000000 |
| description | None |
| encrypted | True |
| id | 8d50602c-e772-401d-a8fb-d1ceef78c260 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | EncVolA |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 9e006c53c1434c65b22c5b22c184c560 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2021-10-17T06:57:49.000000 |
| user_id | e5436c0c3b6d4c8d9b8fcf1f82d68559 |
| volume_type | LUKS |
+--------------------------------+--------------------------------------+
2. Boot an instance attach said volume, create filesystem on this volume, mount and write data on volume:
(overcloud) [stack@undercloud-0 ~]$ cinder list
nova list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 8d50602c-e772-401d-a8fb-d1ceef78c260 | available | EncVolA | 1 | LUKS | false | |
(overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 8d50602c-e772-401d-a8fb-d1ceef78c260
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | 8d50602c-e772-401d-a8fb-d1ceef78c260 |
| serverId | 2454b0b1-ab35-401d-aff7-cb362d3f65f0 |
| volumeId | 8d50602c-e772-401d-a8fb-d1ceef78c260 |
+----------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ ssh cirros.0.219
Warning: Permanently added '10.0.0.219' (ECDSA) to the list of known hosts.
$ sudo -i
# mkfs.ext4 /dev/vd
vda vda1 vda15 vdb
# mkfs.ext4 /dev/vdb
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 82901116-6673-43fa-ba3c-b63c2090263b
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
# mkdir mnt
# mount /dev/vdb mnt/
# echo HelloData > mnt/data.txt
# cat mnt/data.txt
HelloData
3. Create a snapshot from this encrypted volume:
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-create EncVolA --force --name snapOfEncVolA
+-------------+--------------------------------------+
| Property | Value |
+-------------+--------------------------------------+
| created_at | 2021-10-17T07:35:45.644295 |
| description | None |
| id | 6d54954d-53d6-4d1a-b3e9-4ab58c845688 |
| metadata | {} |
| name | snapOfEncVolA |
| size | 1 |
| status | creating |
| updated_at | None |
| volume_id | 8d50602c-e772-401d-a8fb-d1ceef78c260 |
+-------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder snapshot-list
+--------------------------------------+--------------------------------------+-----------+---------------+------+
| ID | Volume ID | Status | Name | Size |
+--------------------------------------+--------------------------------------+-----------+---------------+------+
| 6d54954d-53d6-4d1a-b3e9-4ab58c845688 | 8d50602c-e772-401d-a8fb-d1ceef78c260 | available | snapOfEncVolA | 1 |
+--------------------------------------+--------------------------------------+-----------+---------------+------+
4. Create a cinder volume using the snapshot:
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --snapshot-id 6d54954d-53d6-4d1a-b3e9-4ab58c845688 --name VolBFromSnapshot
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-10-17T07:38:20.000000 |
| description | None |
| encrypted | True |
| id | 3f2d5a34-388b-46b4-b5f0-1a007c457d9b |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | VolBFromSnapshot |
| os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 9e006c53c1434c65b22c5b22c184c560 |
| replication_status | None |
| size | 1 |
| snapshot_id | 6d54954d-53d6-4d1a-b3e9-4ab58c845688 |
| source_volid | None |
| status | creating |
| updated_at | 2021-10-17T07:38:20.000000 |
| user_id | e5436c0c3b6d4c8d9b8fcf1f82d68559 |
| volume_type | LUKS |
+--------------------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
| 3f2d5a34-388b-46b4-b5f0-1a007c457d9b | available | VolBFromSnapshot | 1 | LUKS | false | |
| 8d50602c-e772-401d-a8fb-d1ceef78c260 | in-use | EncVolA | 1 | LUKS | false | 2454b0b1-ab35-401d-aff7-cb362d3f65f0 |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
5. Attach the volume to an instance:
(overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 3f2d5a34-388b-46b4-b5f0-1a007c457d9b
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdc |
| id | 3f2d5a34-388b-46b4-b5f0-1a007c457d9b |
| serverId | 2454b0b1-ab35-401d-aff7-cb362d3f65f0 |
| volumeId | 3f2d5a34-388b-46b4-b5f0-1a007c457d9b |
+----------+--------------------------------------+
6. Mount the volume:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 1G 0 disk
|-vda1 253:1 0 1015M 0 part /
`-vda15 253:15 0 8M 0 part
vdb 253:16 0 1G 0 disk /root/mnt
vdc 253:32 0 1G 0 disk
# mkdir mnt2
# mount /dev/vdc mnt2/
# cat mnt2/data.txt
HelloData
Good to verify, a second mounted volume created from the first volume's snapshot is mountable, data is present.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 13 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:5156 |