Description of problem: It seems like applying the errata RHBA-2016-1618 introduce a regression. Without it it is possible to create an encrypted volume, mount it, touch a dummy file, transfer it to glance image as encrypted, create a volume from this encrypted image without any problem. With errata RHBA-2016-1618, creating a volume from an encrypted image results in a re-encryption to happen an already encrypted image. Version-Release number of selected component (if applicable): RHOSP 6 How reproducible: My tests on UNPATCHED RHOSP6 + LVM backend: After a RHOSP6 install: Be carefull to use a block storage backend (I'm using lvm for now). add fixed_key in nova.conf & cinder.conf: fixed_key=123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234 cinder type-create LUKS cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 \ --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor Create two volumes: cinder create --display-name "Unenc_Cinder" 1 cinder create --display-name "Enc_Cinder" --volume-type LUKS 1 Now we will attach those volume to a vm: [root@dell-m520-4 ~(keystone_admin)]# neutron net-list +--------------------------------------+---------+------------------------------------------------------+ | id | name | subnets | +--------------------------------------+---------+------------------------------------------------------+ | 70c9eb24-196c-4b88-8355-e2f2d2cf877e | private | 4c50ceb1-7afb-4ebf-965c-577ed059bcbf 10.0.0.0/24 | | aa16de11-212d-4d29-83f7-7d59e3a28baa | public | fc70b58f-7aa4-4a90-a77e-fbbb70faed89 172.24.4.224/28 | +--------------------------------------+---------+------------------------------------------------------+ nova boot --flavor m1.tiny --image cirros vm-test --key-name def --nic net-id=70c9eb24-196c-4b88-8355-e2f2d2cf877e [root@dell-m520-4 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 45b4e02f-b9c9-4fef-adba-78c75ea892ab | available | Unenc_Cinder | 1 | None | false | | | c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 | available | Enc_Cinder | 1 | LUKS | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [root@dell-m520-4 ~(keystone_admin)]# nova volume-attach vm-test 45b4e02f-b9c9-4fef-adba-78c75ea892ab +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | 45b4e02f-b9c9-4fef-adba-78c75ea892ab | | serverId | e2436205-63d3-4cbc-a6e2-e74d870bfb71 | | volumeId | 45b4e02f-b9c9-4fef-adba-78c75ea892ab | +----------+--------------------------------------+ [root@dell-m520-4 ~(keystone_admin)]# nova volume-attach vm-test c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdc | | id | c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 | | serverId | e2436205-63d3-4cbc-a6e2-e74d870bfb71 | | volumeId | c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 | +----------+--------------------------------------+ in vm: mkfs.ext4 /dev/vdb mkfs.ext4 /dev/vdc mount /dev/vdb /media/unenc mount /dev/vdc /media/enc echo "unenc `hostname` `date`" > /media/unenc/test_unenc echo "enc `hostname` `date`" > /media/enc/test_enc umount /media/enc umount /media/unenc Detach from instance: [root@dell-m520-4 ~(keystone_admin)]# nova volume-detach vm-test 45b4e02f-b9c9-4fef-adba-78c75ea892ab [root@dell-m520-4 ~(keystone_admin)]# nova volume-detach vm-test c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 [root@dell-m520-4 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 45b4e02f-b9c9-4fef-adba-78c75ea892ab | available | Unenc_Cinder | 1 | None | false | | | c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 | available | Enc_Cinder | 1 | LUKS | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [root@dell-m520-4 ~(keystone_admin)]# cinder upload-to-image 45b4e02f-b9c9-4fef-adba-78c75ea892ab Unenc_From_Cinder +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | container_format | bare | | disk_format | raw | | display_description | None | | id | 45b4e02f-b9c9-4fef-adba-78c75ea892ab | | image_id | 456fc48e-627f-476f-a96b-05e0c50ddb80 | | image_name | Unenc_From_Cinder | | size | 1 | | status | uploading | | updated_at | 2016-10-25T08:11:29.000000 | | volume_type | None | +---------------------+--------------------------------------+ [root@dell-m520-4 ~(keystone_admin)]# cinder upload-to-image c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 Enc_From_Cinder +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | container_format | bare | | disk_format | raw | | display_description | None | | id | c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 | | image_id | 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 | | image_name | Enc_From_Cinder | | size | 1 | | status | uploading | | updated_at | 2016-10-25T08:11:42.000000 | | volume_type | {u'name': u'LUKS', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2016-10-25T00:17:04.000000', u'updated_at': None, u'deleted_at': None, u'id': u'228a651a-83bd-44e7-a3eb-8f0645284b4c'} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [root@dell-m520-4 ~(keystone_admin)]# glance image-list +--------------------------------------+-------------------+-------------+------------------+------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+-------------------+-------------+------------------+------------+--------+ | 5eb1be97-d1b8-45e8-b4e9-929b14e57e7e | cirros | qcow2 | bare | 13200896 | active | | 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 | Enc_From_Cinder | raw | bare | 1073741824 | active | | 456fc48e-627f-476f-a96b-05e0c50ddb80 | Unenc_From_Cinder | raw | bare | 1073741824 | active | +--------------------------------------+-------------------+-------------+------------------+------------+--------+ Now, we download those images and verify their content: [root@dell-m520-4 ~(keystone_admin)]# glance image-list +--------------------------------------+-------------------+-------------+------------------+------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+-------------------+-------------+------------------+------------+--------+ | 5eb1be97-d1b8-45e8-b4e9-929b14e57e7e | cirros | qcow2 | bare | 13200896 | active | | 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 | Enc_From_Cinder | raw | bare | 1073741824 | active | | 456fc48e-627f-476f-a96b-05e0c50ddb80 | Unenc_From_Cinder | raw | bare | 1073741824 | active | +--------------------------------------+-------------------+-------------+------------------+------------+--------+ [root@dell-m520-4 ~(keystone_admin)]# glance image-download --file Enc_From_Cinder.raw 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 [root@dell-m520-4 ~(keystone_admin)]# glance image-download --file Unenc_From_Cinder.raw 456fc48e-627f-476f-a96b-05e0c50ddb80 [root@dell-m520-4 ~(keystone_admin)]# file Enc_From_Cinder.raw Enc_From_Cinder.raw: LUKS encrypted file, ver 1 [aes, xts-plain64, sha1] UUID: 658fcd06-5b44-41c8-bcbd-89f088612482 [root@dell-m520-4 ~(keystone_admin)]# file Unenc_From_Cinder.raw Unenc_From_Cinder.raw: Linux rev 1.0 ext4 filesystem data, UUID=92ffebeb-3387-4933-b31d-32e7ce727f82 (extents) (large files) (huge files) Trying to open LUKS file: [root@dell-m520-4 ~(keystone_admin)]# losetup /dev/loop2 Enc_From_Cinder.raw [root@dell-m520-4 ~(keystone_admin)]# echo "123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234" | cryptsetup luksOpen /dev/loop2 Verify [root@dell-m520-4 ~(keystone_admin)]# mkdir -p /media/test [root@dell-m520-4 ~(keystone_admin)]# mount /dev/mapper/Verify /media/test [root@dell-m520-4 ~(keystone_admin)]# cat /media//test/ lost+found/ test_enc [root@dell-m520-4 ~(keystone_admin)]# cat /media//test/test_enc enc vm-test Tue Oct 25 02:10:27 MDT 2016 Now we upload this as an encrypted image to glance: [root@dell-m520-4 ~(keystone_admin)]# glance image-create --file Enc_From_Cinder.raw --property encrypted=true --name Enc_Upload_2_Glance --disk-format raw --container-format bare +----------------------+--------------------------------------+ | Property | Value | +----------------------+--------------------------------------+ | Property 'encrypted' | true | | checksum | 61059b887a324f1ae03183ec650e8db1 | | container_format | bare | | created_at | 2016-10-25T08:23:11 | | deleted | False | | deleted_at | None | | disk_format | raw | | id | ed120b77-6a6d-43f3-b003-8308618d360b | | is_public | False | | min_disk | 0 | | min_ram | 0 | | name | Enc_Upload_2_Glance | | owner | bbba5876435442f8a737cc562462a023 | | protected | False | | size | 1073741824 | | status | active | | updated_at | 2016-10-25T08:23:16 | | virtual_size | None | +----------------------+--------------------------------------+ [root@dell-m520-4 ~(keystone_admin)]# glance image-list +--------------------------------------+---------------------+-------------+------------------+------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+---------------------+-------------+------------------+------------+--------+ | 5eb1be97-d1b8-45e8-b4e9-929b14e57e7e | cirros | qcow2 | bare | 13200896 | active | | 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 | Enc_From_Cinder | raw | bare | 1073741824 | active | | ed120b77-6a6d-43f3-b003-8308618d360b | Enc_Upload_2_Glance | raw | bare | 1073741824 | active | | 456fc48e-627f-476f-a96b-05e0c50ddb80 | Unenc_From_Cinder | raw | bare | 1073741824 | active | +--------------------------------------+---------------------+-------------+------------------+------------+--------+ Now we create volume from image: [root@dell-m520-4 ~(keystone_admin)]# glance image-list +--------------------------------------+---------------------+-------------+------------------+------------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+---------------------+-------------+------------------+------------+--------+ | 5eb1be97-d1b8-45e8-b4e9-929b14e57e7e | cirros | qcow2 | bare | 13200896 | active | | 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 | Enc_From_Cinder | raw | bare | 1073741824 | active | | ed120b77-6a6d-43f3-b003-8308618d360b | Enc_Upload_2_Glance | raw | bare | 1073741824 | active | | 456fc48e-627f-476f-a96b-05e0c50ddb80 | Unenc_From_Cinder | raw | bare | 1073741824 | active | +--------------------------------------+---------------------+-------------+------------------+------------+--------+ [root@dell-m520-4 ~(keystone_admin)]# cinder create --image-id ed120b77-6a6d-43f3-b003-8308618d360b --display-name From_Glance_Enc_Image --volume-typ LUKS 1 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2016-10-25T08:27:56.656624 | | display_description | None | | display_name | From_Glance_Enc_Image | | encrypted | True | | id | f4211437-81c2-40a0-a81a-1b6f606dcf7f | | image_id | ed120b77-6a6d-43f3-b003-8308618d360b | | metadata | {} | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | LUKS | +---------------------+--------------------------------------+ [root@dell-m520-4 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ | 45b4e02f-b9c9-4fef-adba-78c75ea892ab | available | Unenc_Cinder | 1 | None | false | | | c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 | available | Enc_Cinder | 1 | LUKS | false | | | f4211437-81c2-40a0-a81a-1b6f606dcf7f | available | From_Glance_Enc_Image | 1 | LUKS | true | | +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ Attaching volume to vm-test: [root@dell-m520-4 ~(keystone_admin)]# nova volume-attach vm-test f4211437-81c2-40a0-a81a-1b6f606dcf7f +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | f4211437-81c2-40a0-a81a-1b6f606dcf7f | | serverId | e2436205-63d3-4cbc-a6e2-e74d870bfb71 | | volumeId | f4211437-81c2-40a0-a81a-1b6f606dcf7f | +----------+---------------------------------- On Vm: # mount /dev/vdb /media/re-enc/ # cat /media/re-enc/ lost+found/ test_enc # cat /media/re-enc/test_enc enc vm-test Tue Oct 25 02:10:27 MDT 2016 So this is working. It was also working on Cisco + iSCSI NetApp (provided that was the real setup). It failed on SFR platform with The errata https://rhn.redhat.com/errata/RHBA-2016-1618.html Steps to Reproduce: 1.Apply errata 2.Follow previous steps 3.Mounting the cinder from encrypted glance image into vm. Actual results: mount: unknown filesystem type 'crypto_LUKS' Expected results: Mounting without any trouble Additional info:
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
I believe the change in behavior was introduced here: https://review.openstack.org/#/c/216567/ This patch enables the encryptor layer for the volume when downloading an image to an encrypted volume. But, this is not the right thing to do if the image was already encrypted. (Hence the doubly-encrypted volume.) I am prototyping a fix to resolve this and generally clean up the interaction between encrypted volumes and images. Doing this correctly, I think, involves storing additional metadata when uploading Cinder volumes to Glance. I have initial pieces of this working, but it will take a little while to get it in shape to submit.
Created attachment 1327342 [details] Cinder logs
I don't have a complete understanding, but here's some more information. First, I found the actual error in volume.log-20170920.gz (attached): 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last): 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 547, in _copy_image_to_volume 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id) 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 820, in copy_image_to_encrypted_volume 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume context, volume, image_service, image_id, encrypted=True) 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 846, in _copy_image_data_to_volume 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume size=volume['size']) 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 337, in fetch_to_raw 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume run_as_root=run_as_root) 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 420, in fetch_to_volume_format 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume run_as_root=run_as_root) 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 204, in convert_image 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume run_as_root=run_as_root) 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 165, in _convert_image 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume utils.execute(*cmd, run_as_root=run_as_root) 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume return processutils.execute(*cmd, **kwargs) 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 394, in execute 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume cmd=sanitized_cmd) 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume ProcessExecutionError: Unexpected error while running command. 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -t none -f raw -O raw /var/lib/cinder/conversion/tmpad0jkU /dev/disk/by-id/scsi-3600140576c0520d8dfe4f81a63bd6a84 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Exit code: 1 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Stdout: u'' 2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Stderr: u'qemu-img: /dev/disk/by-id/scsi-3600140576c0520d8dfe4f81a63bd6a84: error while converting raw: Device is too small\n' I found some Internet search results indicating it might be due to insufficient space in the LVM backend, so I logged onto Tzach's system, deleted a few images and volumes, but quickly reproduced the problem and got the same error message (qemu-img: "error while converting raw: Device is too small"). On a lark, I tried increasing the size of the cinder image (from 1G to 2G), and it worked: $ cinder create --image-id e1051519-cb08-4663-ad3b-89b2fdc13b1f --display-name abishop_test --volume-typ LUKS 2 $ cinder list +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ | bfe9e4f3-0e02-4cd3-861a-d2740117c355 | available | Enc_Cinder | 1 | LUKS | false | | | d15e756b-9d59-4ee4-88d9-a01f4086b2f7 | error | From_Glance_Enc_Image | 1 | LUKS | false | | | ea50e6ee-e404-4340-bb9b-bb59f8959403 | available | abishop_test | 2 | LUKS | true | | +--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+ I wonder if something wants to reserve space for a LUKS header, not knowing that the glance image already includes the header. I looked over the recent code changes that handles this, but can't find anything amiss.
Great debug work Alan. If it consistently passes with a reasonable sized volume we can add the the doc needed release note. Let's see what else we can find tomorrow.
Ashamed I didn't spot that error my self, thanks Alan. Hit another snag still need help, volume attach of Alan's successfully created enc volume from glance image worked fine. #nova volume-attach 81cbe07a-965f-45af-b62f-ca9c0e866180 ea50e6ee-e404-4340-bb9b-bb59f8959403 Cinder list shows ^ vol as attached | ea50e6ee-e404-4340-bb9b-bb59f8959403 | in-use | abishop_test | 2 | LUKS | true | 81cbe07a-965f-45af-b62f-ca9c0e866180 | But mounting that encrypted volume inside the instance failed. Which leads me to believe we may have more issues lying in the dark other than just Alan's size tip #36. # mkdir /media/re-enc # mount /dev/vdb /media/re-enc/ mount: mounting /dev/vdb on /media/re-enc/ failed: Invalid argument # mount rootfs on / type rootfs (rw) /dev on /dev type devtmpfs (rw,relatime,size=251788k,nr_inodes=62947,mode=755) /dev/vda1 on / type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered) /proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /dev/shm type tmpfs (rw,relatime,mode=777) tmpfs on /run type tmpfs (rw,nosuid,relatime,size=200k,mode=755) ^ Expected as mount failed we don't see our mount. # cat /media/re-enc/ cat: read error: Is a directory ^ nor can we access encrypted data.
Verified on: openstack-cinder-9.1.4-9.el7ost Reproduction steps returned expected result, see step 16. My only concern on step .14 had to set a larger volume 2G size source vol is 1G Alan's comment #36, do we open a new bug about it or document this? Steps below listed below. 1. Set fixed_key in nova.conf & cinder.conf: fixed_key=123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234 cinder_encryption_key_id 123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234 2. Restart Cinder/Nova 3. Configure LUKS type cinder type-create LUKS cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor 4. Create two volumes: cinder create --display-name unenc_Cinder 1 cinder create --display-name Enc_Cinder --volume-type LUKS 1 5. Cinder list $ cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 0a76cc89-2891-45de-a728-3e65aaf1066c | available | unenc_Cinder | 1 | - | false | | | 1dac35d1-2384-4682-beda-0ae159e30466 | available | - | 1 | LUKS | true | | | 5573016a-77ee-4029-9ac1-9b30cbdcfe42 | available | Enc_Cinder | 1 | LUKS | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+ 5. Attach both volumes to a inst1: $ nova volume-attach inst1 0a76cc89-2891-45de-a728-3e65aaf1066c +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | 0a76cc89-2891-45de-a728-3e65aaf1066c | | serverId | 81cbe07a-965f-45af-b62f-ca9c0e866180 | | volumeId | 0a76cc89-2891-45de-a728-3e65aaf1066c | +----------+--------------------------------------+ $ nova volume-attach inst1 5573016a-77ee-4029-9ac1-9b30cbdcfe42 +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdc | | id | 5573016a-77ee-4029-9ac1-9b30cbdcfe42 | | serverId | 81cbe07a-965f-45af-b62f-ca9c0e866180 | | volumeId | 5573016a-77ee-4029-9ac1-9b30cbdcfe42 | +----------+--------------------------------------+ 6. Login to instance ssh cirros.0.210 sudo -i mkfs.ext4 /dev/vdb mkfs.ext4 /dev/vdc mkdir /media/unenc && mkdir /media/enc mount /dev/vdb /media/unenc && mount /dev/vdc /media/enc echo "unenc `hostname` `date`" > /media/unenc/test_unenc mount .. /dev/vdb on /media/unenc type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered) /dev/vdc on /media/enc type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered) echo "enc `hostname` `date`" > /media/enc/test_enc umount /media/enc && umount /media/unenc ls /media/enc/ lost+found test_enc Exit from instance 7. Detach from instance: $ nova volume-detach inst1 0a76cc89-2891-45de-a728-3e65aaf1066c $ nova volume-detach inst1 5573016a-77ee-4029-9ac1-9b30cbdcfe42 8. cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 0a76cc89-2891-45de-a728-3e65aaf1066c | available | unenc_Cinder | 1 | - | false | | | 1dac35d1-2384-4682-beda-0ae159e30466 | available | - | 1 | LUKS | true | | | 5573016a-77ee-4029-9ac1-9b30cbdcfe42 | available | Enc_Cinder | 1 | LUKS | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ 9. Cinder upload to glance both volumes. cinder upload-to-image 0a76cc89-2891-45de-a728-3e65aaf1066c Unenc_From_Cinder ... cinder upload-to-image 5573016a-77ee-4029-9ac1-9b30cbdcfe42 Enc_From_Cinder ... 10. Check ^ images glance image-show 7fd5d05d-bae7-4f6d-9bdf-53b3c16d00b2 (Unenc_From_Cinder) +------------------+-----------------------------------------------------------------+ | Property | Value | +------------------+-----------------------------------------------------------------+ | checksum | c1d318ca58ec40526d01343531d74069 | | container_format | bare | | created_at | 2017-09-27T08:16:35Z | | direct_url | swift+config://ref1/glance/7fd5d05d-bae7-4f6d-9bdf-53b3c16d00b2 | | disk_format | raw | | id | 7fd5d05d-bae7-4f6d-9bdf-53b3c16d00b2 | | min_disk | 0 | | min_ram | 0 | | name | Unenc_From_Cinder | | owner | 6a6b9ff12c8f4d908c441bd3ffb77b2a | | protected | False | | size | 1073741824 | | status | active | | tags | [] | | updated_at | 2017-09-27T08:16:49Z | | virtual_size | None | | visibility | private | +------------------+-----------------------------------------------------------------+ glance image-show fd002261-196d-4cc6-ac2a-5b19f5058f62 (Enc_From_Cinder) +--------------------------+-----------------------------------------------------------------+ | Property | Value | +--------------------------+-----------------------------------------------------------------+ | checksum | d1a4fda6a5974f5f4684aa60d8337a72 | | cinder_encryption_key_id | 00000000-0000-0000-0000-000000000000 | | container_format | bare | | created_at | 2017-09-27T08:16:57Z | | direct_url | swift+config://ref1/glance/fd002261-196d-4cc6-ac2a-5b19f5058f62 | | disk_format | raw | | id | fd002261-196d-4cc6-ac2a-5b19f5058f62 | | min_disk | 0 | | min_ram | 0 | | name | Enc_From_Cinder | | owner | 6a6b9ff12c8f4d908c441bd3ffb77b2a | | protected | False | | size | 1073741824 | | status | active | | tags | [] | | updated_at | 2017-09-27T08:17:12Z | | virtual_size | None | | visibility | private | +--------------------------+-----------------------------------------------------------------+ 11. Download those images and verify their content: # glance image-download --file Unenc_From_Cinder.raw 7fd5d05d-bae7-4f6d-9bdf-53b3c16d00b2 # glance image-download --file Enc_From_Cinder.raw fd002261-196d-4cc6-ac2a-5b19f5058f62 12. Verify files $ file Unenc_From_Cinder.raw Unenc_From_Cinder.raw: Linux rev 1.0 ext4 filesystem data, UUID=579faba5-4f3f-4444-82a8-9be7d34876b2 (extents) (large files) (huge files) $ file file Enc_From_Cinder.raw file: cannot open (No such file or directory) Enc_From_Cinder.raw: LUKS encrypted file, ver 1 [aes, xts-plain64, sha256] UUID: 193d5fcc-58af-4657-af11-e76994459aa4 Trying to open LUKS file: # losetup /dev/loop2 Enc_From_Cinder.raw # echo "123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234" | cryptsetup luksOpen /dev/loop2 Verify # mkdir -p /media/test # mount /dev/mapper/Verify /media/test # ll /media/test/ total 20 drwx------. 2 root root 16384 Sep 18 06:00 lost+found -rw-------. 1 root root 39 Sep 18 06:01 test_enc # cat /media/test/test_enc enc inst1 Mon Sep 18 11:01:17 UTC 2017 13. upload encrypted image to glance: #glance image-create --file Enc_From_Cinder.raw --property encrypted=true --name Enc_Upload_3_Glance --disk-format raw --container-format bare --property cinder_encryption_key_id=123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234 +------------------+-----------------------------------------------------------------+ | Property | Value | +------------------+-----------------------------------------------------------------+ | checksum | d1a4fda6a5974f5f4684aa60d8337a72 | | container_format | bare | | created_at | 2017-09-27T09:33:25Z | | direct_url | swift+config://ref1/glance/7c813c95-6b86-400b-a368-82f7f28e6ffd | | disk_format | raw | | encrypted | true | | id | 7c813c95-6b86-400b-a368-82f7f28e6ffd | | min_disk | 0 | | min_ram | 0 | | name | Enc_Upload_3_Glance | | owner | 6a6b9ff12c8f4d908c441bd3ffb77b2a | | protected | False | | size | 1073741824 | | status | active | | tags | [] | | updated_at | 2017-09-27T09:33:39Z | | virtual_size | None | | visibility | private | +------------------+-----------------------------------------------------------------+ Take care if you fail to set cinder_encryption_key the uploaded image won't be encrtupted with correct key. Volume will later fail to mount inside instance. 14. Create new enc volume from new image [stack@undercloud-0 ~]$ cinder create --image-id 7c813c95-6b86-400b-a368-82f7f28e6ffd --display-name From_Glance_Enc_Image2 --volume-type LUKS 2 -> notice size must be bigger 2 (1 will fail) +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2017-09-27T09:37:22.000000 | | description | None | | encrypted | True | | id | e4344324-b800-4e3b-a64f-f600aab85f0d | | metadata | {} | | migration_status | None | | multiattach | False | | name | From_Glance_Enc_Image2 | | os-vol-host-attr:host | None | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 6a6b9ff12c8f4d908c441bd3ffb77b2a | | replication_status | disabled | | size | 2 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | None | | user_id | 260b3a8c738e4ba9ac4f78f357c00652 | | volume_type | LUKS | +--------------------------------+--------------------------------------+ [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+------------------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------------+------+-------------+----------+-------------+ | 0a76cc89-2891-45de-a728-3e65aaf1066c | available | unenc_Cinder | 1 | - | false | | | 5573016a-77ee-4029-9ac1-9b30cbdcfe42 | available | Enc_Cinder | 1 | LUKS | false | | | e4344324-b800-4e3b-a64f-f600aab85f0d | available | From_Glance_Enc_Image2 | 2 | LUKS | true | | 15. Attach to isnt1: #nova volume-attach inst1 e4344324-b800-4e3b-a64f-f600aab85f0d +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | e4344324-b800-4e3b-a64f-f600aab85f0d | | serverId | 81cbe07a-965f-45af-b62f-ca9c0e866180 | | volumeId | e4344324-b800-4e3b-a64f-f600aab85f0d | +----------+--------------------------------------+ 16. ssh into instance ssh cirros.0.210 On Vm: # mount /dev/vdb /media/re-enc/ -> again failed. # ls /media/re-enc/ lost+found test_enc # cat /media/re-enc/test_enc enc inst1 Wed Sep 27 09:08:38 UTC 2017 As execpted we got original data, there for verified.
(In reply to Tzach Shefi from comment #44) > 13. upload encrypted image to glance: > > #glance image-create --file Enc_From_Cinder.raw --property encrypted=true > --name Enc_Upload_3_Glance --disk-format raw --container-format bare > --property > cinder_encryption_key_id=123456789abcdef123456789abcdef123456789abcdef1234567 > 89abcdef1234 Just for posterity / documentation folks: cinder_encryption_key_id here needs to be a UUID referring to an encryption key -- for the conf key manager this is all 0s and for Barbican deployments it's the UUID of the key. I believe it worked in this test anyway because the conf key manager path in Cinder is not strictly checking that the key id is all 0s.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2821