Bug 1956762 - Issue with volume created from the snapshot of an encrypted volume where cinder backend is ceph
Summary: Issue with volume created from the snapshot of an encrypted volume where cind...
Keywords:
Status: NEW
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 13.0 (Queens)
Hardware: All
OS: All
high
high
Target Milestone: ---
: ---
Assignee: Sofia Enriquez
QA Contact: Tzach Shefi
Andy Stillman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-04 11:28 UTC by Rohini Diwakar
Modified: 2021-05-06 13:45 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1936564 1 high ASSIGNED Creating an encrypted bootable volume from an unencrypted glance image causes the resulting instance to fail to boot. 2021-05-06 13:45:42 UTC

Description Rohini Diwakar 2021-05-04 11:28:23 UTC
Description of problem:
The issue is similar to the upstream bug https://bugs.launchpad.net/cinder/+bug/1852168

Step 1:

[murg@svldev2-cfc-a-mgmt-001 ~]$ cinder create --display-name encrypted-vol-murg-1 --volume-type LUKS-CBS 1
+------------------------------+------------------------------------------------------------------+
| Property                     | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attachments                  | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2021-05-04T04:48:55.000000                                       |
| description                  | None                                                             |
| encrypted                    | True                                                             |
| id                           | 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb                             |
| metadata                     | {}                                                               |
| multiattach                  | False                                                            |
| name                         | encrypted-vol-murg-1                                             |
| os-vol-tenant-attr:tenant_id | 879af372515f4c7288d4f5818fe962b2                                 |
| replication_status           | None                                                             |
| size                         | 1                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | creating                                                         |
| updated_at                   | 2021-05-04T04:48:55.000000                                       |
| user_id                      | 3fc690999329c3fd7d9aba6d83c0b0ffd3a16129155a3ef0eeb9dd154ede489d |
| volume_type                  | LUKS-CBS                                                         |
+------------------------------+------------------------------------------------------------------+

Step 2:
[murg@svldev2-cfc-a-mgmt-001 ~]$ nova volume-attach murg-test-vm-1 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/sdb                             |
| id       | 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb |
| serverId | 1590bda5-4eac-4f11-89f7-468ec393e405 |
| volumeId | 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb |
+----------+--------------------------------------+

[murg@svldev2-cfc-a-mgmt-001 ~]$ cinder show 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb
+------------------------------+------------------------------------------------------------------+
| Property                     | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attached_servers             | ['1590bda5-4eac-4f11-89f7-468ec393e405']                         |
| attachment_ids               | ['79601c20-fcdd-4d18-8587-f30462ec2943']                         |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2021-05-04T04:48:55.000000                                       |
| description                  | None                                                             |
| encrypted                    | True                                                             |
| id                           | 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb                             |
| metadata                     | attached_mode : rw                                               |
| multiattach                  | False                                                            |
| name                         | encrypted-vol-murg-1                                             |
| os-vol-tenant-attr:tenant_id | 879af372515f4c7288d4f5818fe962b2                                 |
| replication_status           | None                                                             |
| size                         | 1                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | in-use                                                           |
| updated_at                   | 2021-05-04T04:49:47.000000                                       |
| user_id                      | 3fc690999329c3fd7d9aba6d83c0b0ffd3a16129155a3ef0eeb9dd154ede489d |
| volume_type                  | LUKS-CBS                                                         |
+------------------------------+------------------------------------------------------------------+

Step 3:
Mount the volume,create FS, Mount and copy data.

[root@murg-test-vm-1 cloud-user]# fdisk -l|grep ^Disk|grep /dev
Disk /dev/sda: 50 GiB, 53687091200 bytes, 104857600 sectors
Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors

[root@murg-test-vm-1 cloud-user]# mkfs.xfs /dev/sdb
meta-data=/dev/sdb               isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.

[root@murg-test-vm-1 cloud-user]# mount -t xfs /dev/sdb /mnt
[root@murg-test-vm-1 cloud-user]# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb       1014M   40M  975M   4% /mnt

[root@murg-test-vm-1 cloud-user]# cp /var/log/messages /mnt
[root@murg-test-vm-1 cloud-user]# ls /mnt/*
/mnt/messages

Step 4:
Umount the FS and detach the cinder volume from the VM instance
[murg@svldev2-cfc-a-mgmt-001 ~]$ nova volume-detach murg-test-vm-1 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb
[murg@svldev2-cfc-a-mgmt-001 ~]$ cinder show 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb
+------------------------------+------------------------------------------------------------------+
| Property                     | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attached_servers             | []                                                               |
| attachment_ids               | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2021-05-04T04:48:55.000000                                       |
| description                  | None                                                             |
| encrypted                    | True                                                             |
| id                           | 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb                             |
| metadata                     |                                                                  |
| multiattach                  | False                                                            |
| name                         | encrypted-vol-murg-1                                             |
| os-vol-tenant-attr:tenant_id | 879af372515f4c7288d4f5818fe962b2                                 |
| replication_status           | None                                                             |
| size                         | 1                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| updated_at                   | 2021-05-04T04:54:41.000000                                       |
| user_id                      | 3fc690999329c3fd7d9aba6d83c0b0ffd3a16129155a3ef0eeb9dd154ede489d |
| volume_type                  | LUKS-CBS                                                         |
+------------------------------+------------------------------------------------------------------+

Step 5:
Create snapshot from this volume.
[murg@svldev2-cfc-a-mgmt-001 ~]$ cinder snapshot-create --name encrypted-vol-murg-1-snap encrypted-vol-murg-1
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| created_at  | 2021-05-04T04:55:38.942664           |
| description | None                                 |
| id          | 9d20be7a-bb48-4e78-be55-8c4e7320ba6d |
| metadata    | {}                                   |
| name        | encrypted-vol-murg-1-snap            |
| size        | 1                                    |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb |
+-------------+--------------------------------------+
[murg@svldev2-cfc-a-mgmt-001 ~]$ cinder snapshot-show 9d20be7a-bb48-4e78-be55-8c4e7320ba6d
+--------------------------------------------+--------------------------------------+
| Property                                   | Value                                |
+--------------------------------------------+--------------------------------------+
| created_at                                 | 2021-05-04T04:55:38.000000           |
| description                                | None                                 |
| id                                         | 9d20be7a-bb48-4e78-be55-8c4e7320ba6d |
| metadata                                   | {}                                   |
| name                                       | encrypted-vol-murg-1-snap            |
| os-extended-snapshot-attributes:progress   | 100%                                 |
| os-extended-snapshot-attributes:project_id | 879af372515f4c7288d4f5818fe962b2     |
| size                                       | 1                                    |
| status                                     | available                            |
| updated_at                                 | 2021-05-04T04:55:41.000000           |
| volume_id                                  | 37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb |
+--------------------------------------------+--------------------------------------+

Step 6:
Create a cinder volume using the snapshot.
[murg@svldev2-cfc-a-mgmt-001 ~]$ openstack volume create --size 1 --snapshot 9d20be7a-bb48-4e78-be55-8c4e7320ba6d encrypted-vol-murg-1-clone
+---------------------+------------------------------------------------------------------+
| Field               | Value                                                            |
+---------------------+------------------------------------------------------------------+
| attachments         | []                                                               |
| availability_zone   | nova                                                             |
| bootable            | false                                                            |
| consistencygroup_id | None                                                             |
| created_at          | 2021-05-04T04:56:42.000000                                       |
| description         | None                                                             |
| encrypted           | True                                                             |
| id                  | bcb95a0a-a0f3-4634-b84c-c02253269c40                             |
| multiattach         | False                                                            |
| name                | encrypted-vol-murg-1-clone                                       |
| properties          |                                                                  |
| replication_status  | None                                                             |
| size                | 1                                                                |
| snapshot_id         | 9d20be7a-bb48-4e78-be55-8c4e7320ba6d                             |
| source_volid        | None                                                             |
| status              | creating                                                         |
| type                | LUKS-CBS                                                         |
| updated_at          | None                                                             |
| user_id             | 3fc690999329c3fd7d9aba6d83c0b0ffd3a16129155a3ef0eeb9dd154ede489d |
+---------------------+------------------------------------------------------------------+

[murg@svldev2-cfc-a-mgmt-001 ~]$ cinder show bcb95a0a-a0f3-4634-b84c-c02253269c40
+------------------------------+------------------------------------------------------------------+
| Property                     | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attached_servers             | []                                                               |
| attachment_ids               | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2021-05-04T04:56:42.000000                                       |
| description                  | None                                                             |
| encrypted                    | True                                                             |
| id                           | bcb95a0a-a0f3-4634-b84c-c02253269c40                             |
| metadata                     |                                                                  |
| multiattach                  | False                                                            |
| name                         | encrypted-vol-murg-1-clone                                       |
| os-vol-tenant-attr:tenant_id | 879af372515f4c7288d4f5818fe962b2                                 |
| replication_status           | None                                                             |
| size                         | 1                                                                |
| snapshot_id                  | 9d20be7a-bb48-4e78-be55-8c4e7320ba6d                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| updated_at                   | 2021-05-04T04:56:44.000000                                       |
| user_id                      | 3fc690999329c3fd7d9aba6d83c0b0ffd3a16129155a3ef0eeb9dd154ede489d |
| volume_type                  | LUKS-CBS                                                         |
+------------------------------+------------------------------------------------------------------+

Step 7:
Attach the volume "encrypted-vol-murg-1-clone" to openstack instance and mount.

[murg@svldev2-cfc-a-mgmt-001 ~]$ nova volume-attach murg-test-vm-1 bcb95a0a-a0f3-4634-b84c-c02253269c40
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/sdb                             |
| id       | bcb95a0a-a0f3-4634-b84c-c02253269c40 |
| serverId | 1590bda5-4eac-4f11-89f7-468ec393e405 |
| volumeId | bcb95a0a-a0f3-4634-b84c-c02253269c40 |
+----------+--------------------------------------+

[root@murg-test-vm-1 cloud-user]# fdisk -l|grep ^Disk|grep /dev
Disk /dev/sda: 50 GiB, 53687091200 bytes, 104857600 sectors
Disk /dev/sdb: 1022 MiB, 1071673344 bytes, 2093112 sectors

Dmesg:
[412440.315332] sd 0:0:0:1: [sdb] Synchronizing SCSI cache
[412440.758658] sd 0:0:0:1: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[412440.763957] sd 0:0:0:1: [sdb] Sense Key : Illegal Request [current]
[412440.767631] sd 0:0:0:1: [sdb] Add. Sense: Logical unit not supported
[412650.987667] scsi 0:0:0:1: Direct-Access     QEMU     QEMU HARDDISK    2.5+ PQ: 0 ANSI: 5
[412651.000266] sd 0:0:0:1: Attached scsi generic sg1 type 0
[412651.002809] sd 0:0:0:1: Power-on or device reset occurred
[412651.007657] sd 0:0:0:1: [sdb] 2093112 512-byte logical blocks: (1.07 GB/1022 MiB)
[412651.011395] sd 0:0:0:1: [sdb] Write Protect is off
[412651.013654] sd 0:0:0:1: [sdb] Mode Sense: 63 00 00 08
[412651.014653] sd 0:0:0:1: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[412651.028648] sd 0:0:0:1: [sdb] Attached SCSI disk

[root@murg-test-vm-1 cloud-user]#  mount -t xfs /dev/sdb /mnt
mount: /mnt: can't read superblock on /dev/sdb.

we can see there is some size difference of 2 MB on the volume created from the snapshot, the filesystem is not mounting and it is unsuable.

Version-Release number of selected component (if applicable):
RHOSP13

How reproducible:


Steps to Reproduce:
1. Create an encrypted cinder volume.
2. Attach this cinder volume to an instance and create filesystem on this volume, mount the filesystem and add some data.
3. Create a snapshot from this encrypted volume (  encrypted-vol-murg-1  )
4. Create a cinder volume using the snapshot.
5. Attach the volume to an instance.
6. Mount the volume.

Actual results:


Expected results:


Additional info:
$ cat installed-rpms | grep cinder
openstack-cinder-12.0.10-11.el7ost.noarch                   Fri Feb 12 06:44:09 2021
python2-cinderclient-3.5.0-2.el7ost.noarch                  Fri Feb 12 06:43:54 2021
python-cinder-12.0.10-11.el7ost.noarch                      Fri Feb 12 06:44:06 2021

$ cat installed-rpms | grep ceph
ceph-common-12.2.12-124.el7cp.x86_64                        Fri Feb  5 08:32:51 2021
libcephfs2-12.2.12-124.el7cp.x86_64                         Fri Feb  5 08:32:49 2021
python-cephfs-12.2.12-124.el7cp.x86_64                      Fri Feb  5 08:32:49 2021

Comment 1 Rohini Diwakar 2021-05-04 11:57:42 UTC
Hello,

37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb: The source volume
bcb95a0a-a0f3-4634-b84c-c02253269c40: The final volume which is created from snapshot of source volume.  

[root@svl-ceph-110 ~]# rbd info -p cloud-svldev-2-cinder volume-37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb
rbd image 'volume-37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb':
        size 1.00GiB in 257 objects
        order 22 (4MiB objects)
        block_name_prefix: rbd_data.eba40f6b8b4567
        format: 2
        features: layering, exclusive-lock, object-map
        flags:
        create_timestamp: Tue May  4 04:49:01 2021

[root@svl-ceph-110 ~]# rbd info -p cloud-svldev-2-cinder  volume-bcb95a0a-a0f3-4634-b84c-c02253269c40
rbd image 'volume-bcb95a0a-a0f3-4634-b84c-c02253269c40':
        size 1GiB in 256 objects
        order 22 (4MiB objects)
        block_name_prefix: rbd_data.ec05e94113f7ca
        format: 2
        features: layering, exclusive-lock, object-map
        flags:
        create_timestamp: Tue May  4 04:56:43 2021
        parent: cloud-svldev-2-cinder/volume-37bbc0d1-3bfc-4624-a77a-f8fb89db1ccb@snapshot-9d20be7a-bb48-4e78-be55-8c4e7320ba6d
        overlap: 1GiB


After exporting the rbd volumes we can see the difference in their size 

[murg@svldev2-cfc-a-mgmt-001 ~]$ qemu-img info ./source.img
image: ./source.img
file format: luks
virtual size: 1.0G (1073741824 bytes)
disk size: 1.0G
encrypted: yes
Format specific information:
    ivgen alg: plain64
    hash alg: sha256
    cipher alg: aes-256
    uuid: 9f4cbf11-435f-4247-bc05-76090edd9872
    cipher mode: xts
    slots:
        [0]:
            active: true
            iters: 123186
            key offset: 4096
            stripes: 4000
        [1]:
            active: false
            key offset: 262144
        [2]:
            active: false
            key offset: 520192
        [3]:
            active: false
            key offset: 778240
        [4]:
            active: false
            key offset: 1036288
        [5]:
            active: false
            key offset: 1294336
        [6]:
            active: false
            key offset: 1552384
        [7]:
            active: false
            key offset: 1810432
    payload offset: 2068480
    master key iters: 31107

[murg@svldev2-cfc-a-mgmt-001 ~]$ qemu-img info ./final.img
image: ./final.img
file format: luks
virtual size: 1.0G (1071673344 bytes)
disk size: 1.0G
encrypted: yes
Format specific information:
    ivgen alg: plain64
    hash alg: sha256
    cipher alg: aes-256
    uuid: 9f4cbf11-435f-4247-bc05-76090edd9872
    cipher mode: xts
    slots:
        [0]:
            active: true
            iters: 123186
            key offset: 4096
            stripes: 4000
        [1]:
            active: false
            key offset: 262144
        [2]:
            active: false
            key offset: 520192
        [3]:
            active: false
            key offset: 778240
        [4]:
            active: false
            key offset: 1036288
        [5]:
            active: false
            key offset: 1294336
        [6]:
            active: false
            key offset: 1552384
        [7]:
            active: false
            key offset: 1810432
    payload offset: 2068480
    master key iters: 31107


Note You need to log in before you can comment on or make changes to this bug.