Bug 1388559 - RHBA-2016-1618 Regression, re-encryption of encrypted image
Summary: RHBA-2016-1618 Regression, re-encryption of encrypted image
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 6.0 (Juno)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: z5
: 10.0 (Newton)
Assignee: Eric Harney
QA Contact: Avi Avraham
URL:
Whiteboard:
Depends On: 1454850
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-25 16:14 UTC by Pierre-Andre MOREY
Modified: 2020-12-14 07:49 UTC (History)
15 users (show)

Fixed In Version: openstack-cinder-9.1.4-9.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-09-28 16:31:38 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Cinder logs (336.43 KB, application/x-gzip)
2017-09-18 12:00 UTC, Tzach Shefi
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1285089 0 high CLOSED [RFE] Boot instance from encrypted volume [iSCSI] 2022-08-03 15:36:24 UTC
Red Hat Bugzilla 1454850 0 high CLOSED Handle upload-to-image and volume-from-image for encrypted volumes 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2017:2821 0 normal SHIPPED_LIVE openstack-cinder bug fix advisory 2017-09-28 20:30:49 UTC

Internal Links: 1285089 1454850

Description Pierre-Andre MOREY 2016-10-25 16:14:19 UTC
Description of problem:
It seems like applying the errata RHBA-2016-1618 introduce a regression.

Without it it is possible to create an encrypted volume, mount it, touch a dummy file, transfer it to glance image as encrypted, create a volume from this encrypted image without any problem.

With errata RHBA-2016-1618, creating a volume from an encrypted image results in a re-encryption to happen an already encrypted image.

Version-Release number of selected component (if applicable):
RHOSP 6

How reproducible:

My tests on UNPATCHED RHOSP6 + LVM backend:

After a RHOSP6 install:

Be carefull to use a block storage backend (I'm using lvm for now).

add fixed_key in nova.conf & cinder.conf:
fixed_key=123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234

cinder type-create LUKS
cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 \
  --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor

Create two volumes:
cinder create --display-name "Unenc_Cinder" 1
cinder create --display-name "Enc_Cinder" --volume-type LUKS 1

Now we will attach those volume to a vm:
[root@dell-m520-4 ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+------------------------------------------------------+
| id                                   | name    | subnets                                              |
+--------------------------------------+---------+------------------------------------------------------+
| 70c9eb24-196c-4b88-8355-e2f2d2cf877e | private | 4c50ceb1-7afb-4ebf-965c-577ed059bcbf 10.0.0.0/24     |
| aa16de11-212d-4d29-83f7-7d59e3a28baa | public  | fc70b58f-7aa4-4a90-a77e-fbbb70faed89 172.24.4.224/28 |
+--------------------------------------+---------+------------------------------------------------------+

nova boot --flavor m1.tiny --image cirros vm-test --key-name def --nic net-id=70c9eb24-196c-4b88-8355-e2f2d2cf877e

[root@dell-m520-4 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 45b4e02f-b9c9-4fef-adba-78c75ea892ab | available | Unenc_Cinder |  1   |     None    |  false   |             |
| c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 | available |  Enc_Cinder  |  1   |     LUKS    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

[root@dell-m520-4 ~(keystone_admin)]# nova volume-attach vm-test 45b4e02f-b9c9-4fef-adba-78c75ea892ab
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 45b4e02f-b9c9-4fef-adba-78c75ea892ab |
| serverId | e2436205-63d3-4cbc-a6e2-e74d870bfb71 |
| volumeId | 45b4e02f-b9c9-4fef-adba-78c75ea892ab |
+----------+--------------------------------------+

[root@dell-m520-4 ~(keystone_admin)]# nova volume-attach vm-test c7dc9dd8-5cc0-4925-9c14-5c18b75343c5
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdc                             |
| id       | c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 |
| serverId | e2436205-63d3-4cbc-a6e2-e74d870bfb71 |
| volumeId | c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 |
+----------+--------------------------------------+

in vm:
mkfs.ext4 /dev/vdb
mkfs.ext4 /dev/vdc
mount /dev/vdb /media/unenc
mount /dev/vdc /media/enc
echo "unenc `hostname` `date`" > /media/unenc/test_unenc
echo "enc `hostname` `date`" > /media/enc/test_enc
umount /media/enc
umount /media/unenc

Detach from instance:
[root@dell-m520-4 ~(keystone_admin)]# nova volume-detach vm-test 45b4e02f-b9c9-4fef-adba-78c75ea892ab
[root@dell-m520-4 ~(keystone_admin)]# nova volume-detach vm-test c7dc9dd8-5cc0-4925-9c14-5c18b75343c5
[root@dell-m520-4 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 45b4e02f-b9c9-4fef-adba-78c75ea892ab | available | Unenc_Cinder |  1   |     None    |  false   |             |
| c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 | available |  Enc_Cinder  |  1   |     LUKS    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
[root@dell-m520-4 ~(keystone_admin)]# cinder upload-to-image 45b4e02f-b9c9-4fef-adba-78c75ea892ab Unenc_From_Cinder
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|   container_format  |                 bare                 |
|     disk_format     |                 raw                  |
| display_description |                 None                 |
|          id         | 45b4e02f-b9c9-4fef-adba-78c75ea892ab |
|       image_id      | 456fc48e-627f-476f-a96b-05e0c50ddb80 |
|      image_name     |          Unenc_From_Cinder           |
|         size        |                  1                   |
|        status       |              uploading               |
|      updated_at     |      2016-10-25T08:11:29.000000      |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[root@dell-m520-4 ~(keystone_admin)]# cinder upload-to-image c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 Enc_From_Cinder
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|       Property      |                                                                                                Value                                                                                                 |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|   container_format  |                                                                                                 bare                                                                                                 |
|     disk_format     |                                                                                                 raw                                                                                                  |
| display_description |                                                                                                 None                                                                                                 |
|          id         |                                                                                 c7dc9dd8-5cc0-4925-9c14-5c18b75343c5                                                                                 |
|       image_id      |                                                                                 76134b7c-c9e7-4a45-99cd-fbe22f37ad46                                                                                 |
|      image_name     |                                                                                           Enc_From_Cinder                                                                                            |
|         size        |                                                                                                  1                                                                                                   |
|        status       |                                                                                              uploading                                                                                               |
|      updated_at     |                                                                                      2016-10-25T08:11:42.000000                                                                                      |
|     volume_type     | {u'name': u'LUKS', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2016-10-25T00:17:04.000000', u'updated_at': None, u'deleted_at': None, u'id': u'228a651a-83bd-44e7-a3eb-8f0645284b4c'} |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@dell-m520-4 ~(keystone_admin)]# glance image-list
+--------------------------------------+-------------------+-------------+------------------+------------+--------+
| ID                                   | Name              | Disk Format | Container Format | Size       | Status |
+--------------------------------------+-------------------+-------------+------------------+------------+--------+
| 5eb1be97-d1b8-45e8-b4e9-929b14e57e7e | cirros            | qcow2       | bare             | 13200896   | active |
| 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 | Enc_From_Cinder   | raw         | bare             | 1073741824 | active |
| 456fc48e-627f-476f-a96b-05e0c50ddb80 | Unenc_From_Cinder | raw         | bare             | 1073741824 | active |
+--------------------------------------+-------------------+-------------+------------------+------------+--------+


Now, we download those images and verify their content:
[root@dell-m520-4 ~(keystone_admin)]# glance image-list
+--------------------------------------+-------------------+-------------+------------------+------------+--------+
| ID                                   | Name              | Disk Format | Container Format | Size       | Status |
+--------------------------------------+-------------------+-------------+------------------+------------+--------+
| 5eb1be97-d1b8-45e8-b4e9-929b14e57e7e | cirros            | qcow2       | bare             | 13200896   | active |
| 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 | Enc_From_Cinder   | raw         | bare             | 1073741824 | active |
| 456fc48e-627f-476f-a96b-05e0c50ddb80 | Unenc_From_Cinder | raw         | bare             | 1073741824 | active |
+--------------------------------------+-------------------+-------------+------------------+------------+--------+
[root@dell-m520-4 ~(keystone_admin)]# glance image-download --file Enc_From_Cinder.raw 76134b7c-c9e7-4a45-99cd-fbe22f37ad46
[root@dell-m520-4 ~(keystone_admin)]# glance image-download --file Unenc_From_Cinder.raw 456fc48e-627f-476f-a96b-05e0c50ddb80
[root@dell-m520-4 ~(keystone_admin)]# file Enc_From_Cinder.raw 
Enc_From_Cinder.raw: LUKS encrypted file, ver 1 [aes, xts-plain64, sha1] UUID: 658fcd06-5b44-41c8-bcbd-89f088612482
[root@dell-m520-4 ~(keystone_admin)]# file Unenc_From_Cinder.raw 
Unenc_From_Cinder.raw: Linux rev 1.0 ext4 filesystem data, UUID=92ffebeb-3387-4933-b31d-32e7ce727f82 (extents) (large files) (huge files)

Trying to open LUKS file:
[root@dell-m520-4 ~(keystone_admin)]# losetup /dev/loop2 Enc_From_Cinder.raw 
[root@dell-m520-4 ~(keystone_admin)]# echo "123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234" | cryptsetup luksOpen /dev/loop2 Verify
[root@dell-m520-4 ~(keystone_admin)]# mkdir -p /media/test
[root@dell-m520-4 ~(keystone_admin)]# mount /dev/mapper/Verify /media/test
[root@dell-m520-4 ~(keystone_admin)]# cat /media//test/
lost+found/ test_enc    
[root@dell-m520-4 ~(keystone_admin)]# cat /media//test/test_enc 
enc vm-test Tue Oct 25 02:10:27 MDT 2016

Now we upload this as an encrypted image to glance:
[root@dell-m520-4 ~(keystone_admin)]# glance image-create --file Enc_From_Cinder.raw --property encrypted=true --name Enc_Upload_2_Glance --disk-format raw --container-format bare
+----------------------+--------------------------------------+
| Property             | Value                                |
+----------------------+--------------------------------------+
| Property 'encrypted' | true                                 |
| checksum             | 61059b887a324f1ae03183ec650e8db1     |
| container_format     | bare                                 |
| created_at           | 2016-10-25T08:23:11                  |
| deleted              | False                                |
| deleted_at           | None                                 |
| disk_format          | raw                                  |
| id                   | ed120b77-6a6d-43f3-b003-8308618d360b |
| is_public            | False                                |
| min_disk             | 0                                    |
| min_ram              | 0                                    |
| name                 | Enc_Upload_2_Glance                  |
| owner                | bbba5876435442f8a737cc562462a023     |
| protected            | False                                |
| size                 | 1073741824                           |
| status               | active                               |
| updated_at           | 2016-10-25T08:23:16                  |
| virtual_size         | None                                 |
+----------------------+--------------------------------------+

[root@dell-m520-4 ~(keystone_admin)]# glance image-list
+--------------------------------------+---------------------+-------------+------------------+------------+--------+
| ID                                   | Name                | Disk Format | Container Format | Size       | Status |
+--------------------------------------+---------------------+-------------+------------------+------------+--------+
| 5eb1be97-d1b8-45e8-b4e9-929b14e57e7e | cirros              | qcow2       | bare             | 13200896   | active |
| 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 | Enc_From_Cinder     | raw         | bare             | 1073741824 | active |
| ed120b77-6a6d-43f3-b003-8308618d360b | Enc_Upload_2_Glance | raw         | bare             | 1073741824 | active |
| 456fc48e-627f-476f-a96b-05e0c50ddb80 | Unenc_From_Cinder   | raw         | bare             | 1073741824 | active |
+--------------------------------------+---------------------+-------------+------------------+------------+--------+

Now we create volume from image:
[root@dell-m520-4 ~(keystone_admin)]# glance image-list
+--------------------------------------+---------------------+-------------+------------------+------------+--------+
| ID                                   | Name                | Disk Format | Container Format | Size       | Status |
+--------------------------------------+---------------------+-------------+------------------+------------+--------+
| 5eb1be97-d1b8-45e8-b4e9-929b14e57e7e | cirros              | qcow2       | bare             | 13200896   | active |
| 76134b7c-c9e7-4a45-99cd-fbe22f37ad46 | Enc_From_Cinder     | raw         | bare             | 1073741824 | active |
| ed120b77-6a6d-43f3-b003-8308618d360b | Enc_Upload_2_Glance | raw         | bare             | 1073741824 | active |
| 456fc48e-627f-476f-a96b-05e0c50ddb80 | Unenc_From_Cinder   | raw         | bare             | 1073741824 | active |
+--------------------------------------+---------------------+-------------+------------------+------------+--------+
[root@dell-m520-4 ~(keystone_admin)]# cinder create --image-id ed120b77-6a6d-43f3-b003-8308618d360b --display-name From_Glance_Enc_Image --volume-typ LUKS 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2016-10-25T08:27:56.656624      |
| display_description |                 None                 |
|     display_name    |        From_Glance_Enc_Image         |
|      encrypted      |                 True                 |
|          id         | f4211437-81c2-40a0-a81a-1b6f606dcf7f |
|       image_id      | ed120b77-6a6d-43f3-b003-8308618d360b |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 LUKS                 |
+---------------------+--------------------------------------+
[root@dell-m520-4 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+
|                  ID                  |   Status  |      Display Name     | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+
| 45b4e02f-b9c9-4fef-adba-78c75ea892ab | available |      Unenc_Cinder     |  1   |     None    |  false   |             |
| c7dc9dd8-5cc0-4925-9c14-5c18b75343c5 | available |       Enc_Cinder      |  1   |     LUKS    |  false   |             |
| f4211437-81c2-40a0-a81a-1b6f606dcf7f | available | From_Glance_Enc_Image |  1   |     LUKS    |   true   |             |
+--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+

Attaching volume to vm-test:
[root@dell-m520-4 ~(keystone_admin)]# nova volume-attach vm-test f4211437-81c2-40a0-a81a-1b6f606dcf7f
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | f4211437-81c2-40a0-a81a-1b6f606dcf7f |
| serverId | e2436205-63d3-4cbc-a6e2-e74d870bfb71 |
| volumeId | f4211437-81c2-40a0-a81a-1b6f606dcf7f |
+----------+----------------------------------

On Vm:
# mount /dev/vdb /media/re-enc/
# cat /media/re-enc/
lost+found/  test_enc
# cat /media/re-enc/test_enc 
enc vm-test Tue Oct 25 02:10:27 MDT 2016



So this is working. It was also working on Cisco + iSCSI NetApp (provided that was the real setup). It failed on SFR platform with The errata https://rhn.redhat.com/errata/RHBA-2016-1618.html




Steps to Reproduce:
1.Apply errata
2.Follow previous steps
3.Mounting the cinder from encrypted glance image into vm.

Actual results:
mount: unknown filesystem type 'crypto_LUKS'

Expected results:
Mounting without any trouble

Additional info:

Comment 14 Red Hat Bugzilla Rules Engine 2017-02-03 14:09:46 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 15 Eric Harney 2017-03-31 18:38:45 UTC
I believe the change in behavior was introduced here:

https://review.openstack.org/#/c/216567/

This patch enables the encryptor layer for the volume when downloading an image to an encrypted volume.  But, this is not the right thing to do if the image was already encrypted.  (Hence the doubly-encrypted volume.)

I am prototyping a fix to resolve this and generally clean up the interaction between encrypted volumes and images.  Doing this correctly, I think, involves storing additional metadata when uploading Cinder volumes to Glance.

I have initial pieces of this working, but it will take a little while to get it in shape to submit.

Comment 29 Tzach Shefi 2017-09-18 12:00:36 UTC
Created attachment 1327342 [details]
Cinder logs

Comment 36 Alan Bishop 2017-09-25 17:24:16 UTC
I don't have a complete understanding, but here's some more information. 
First, I found the actual error in volume.log-20170920.gz (attached):

2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last):
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 547, in _copy_image_to_volume
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume     context, volume, image_service, image_id)
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 820, in copy_image_to_encrypted_volume
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume     context, volume, image_service, image_id, encrypted=True)
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 846, in _copy_image_data_to_volume
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume     size=volume['size'])
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 337, in fetch_to_raw
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume     run_as_root=run_as_root)
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 420, in fetch_to_volume_format
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume     run_as_root=run_as_root)
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 204, in convert_image
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume     run_as_root=run_as_root)
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 165, in _convert_image
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume     utils.execute(*cmd, run_as_root=run_as_root)
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 123, in execute
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume     return processutils.execute(*cmd, **kwargs)
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 394, in execute
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume     cmd=sanitized_cmd)
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume ProcessExecutionError: Unexpected error while running command.
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -t none -f raw -O raw /var/lib/cinder/conversion/tmpad0jkU /dev/disk/by-id/scsi-3600140576c0520d8dfe4f81a63bd6a84
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Exit code: 1
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Stdout: u''
2017-09-18 11:35:12.038 133573 ERROR cinder.volume.flows.manager.create_volume Stderr: u'qemu-img: /dev/disk/by-id/scsi-3600140576c0520d8dfe4f81a63bd6a84: error while converting raw: Device is too small\n'

I found some Internet search results indicating it might be due to
insufficient space in the LVM backend, so I logged onto Tzach's system,
deleted a few images and volumes, but quickly reproduced the problem and got
the same error message (qemu-img: "error while converting raw: Device is too
small").

On a lark, I tried increasing the size of the cinder image (from 1G to 2G),
and it worked:

$ cinder create --image-id e1051519-cb08-4663-ad3b-89b2fdc13b1f --display-name
abishop_test --volume-typ LUKS 2

$ cinder list
+--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name                  | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+
| bfe9e4f3-0e02-4cd3-861a-d2740117c355 | available | Enc_Cinder            | 1    | LUKS        | false    |             |
| d15e756b-9d59-4ee4-88d9-a01f4086b2f7 | error     | From_Glance_Enc_Image | 1    | LUKS        | false    |             |
| ea50e6ee-e404-4340-bb9b-bb59f8959403 | available | abishop_test          | 2    | LUKS        | true     |             |
+--------------------------------------+-----------+-----------------------+------+-------------+----------+-------------+

I wonder if something wants to reserve space for a LUKS header, not knowing
that the glance image already includes the header. I looked over the recent
code changes that handles this, but can't find anything amiss.

Comment 37 Paul Grist 2017-09-26 00:56:45 UTC
Great debug work Alan.  If it consistently passes with a reasonable sized volume we can add the the doc needed release note. Let's see what else we can find tomorrow.

Comment 40 Tzach Shefi 2017-09-26 08:32:58 UTC
Ashamed I didn't spot that error my self, thanks Alan. 

Hit another snag still need help, volume attach of Alan's successfully created enc volume from glance image worked fine. 

#nova volume-attach 81cbe07a-965f-45af-b62f-ca9c0e866180 ea50e6ee-e404-4340-bb9b-bb59f8959403

Cinder list shows ^ vol as attached
| ea50e6ee-e404-4340-bb9b-bb59f8959403 | in-use    | abishop_test          | 2    | LUKS        | true     | 81cbe07a-965f-45af-b62f-ca9c0e866180 |


But mounting that encrypted volume inside the instance failed. 
Which leads me to believe we may have more issues lying in the dark other than just Alan's size tip #36.  

# mkdir /media/re-enc
# mount /dev/vdb /media/re-enc/
mount: mounting /dev/vdb on /media/re-enc/ failed: Invalid argument

# mount
rootfs on / type rootfs (rw)
/dev on /dev type devtmpfs (rw,relatime,size=251788k,nr_inodes=62947,mode=755)
/dev/vda1 on / type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered)
/proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime,mode=777)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=200k,mode=755)

^ Expected as mount failed we don't see our mount. 

# cat /media/re-enc/
cat: read error: Is a directory

^ nor can we access encrypted data.

Comment 44 Tzach Shefi 2017-09-27 10:51:52 UTC
Verified on: 
openstack-cinder-9.1.4-9.el7ost

Reproduction steps returned expected result, see step 16. 
My only concern on step .14 had to set a larger volume 2G size source vol is 1G
Alan's comment #36, do we open a new bug about it or document this?  

Steps below listed below.

1. Set fixed_key in nova.conf & cinder.conf:
fixed_key=123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234


 cinder_encryption_key_id 123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234   

2. Restart Cinder/Nova

3. Configure LUKS type
cinder type-create LUKS
cinder encryption-type-create --cipher aes-xts-plain64 --key_size 512 --control_location front-end LUKS nova.volume.encryptors.luks.LuksEncryptor

4. Create two volumes:
cinder create --display-name unenc_Cinder 1
cinder create --display-name Enc_Cinder --volume-type LUKS 1

5. Cinder list
$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name         | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 0a76cc89-2891-45de-a728-3e65aaf1066c | available | unenc_Cinder | 1    | -           | false    |             |
| 1dac35d1-2384-4682-beda-0ae159e30466 | available | -            | 1    | LUKS        | true     |             |
| 5573016a-77ee-4029-9ac1-9b30cbdcfe42 | available | Enc_Cinder   | 1    | LUKS        | false    |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+

5. Attach both volumes to a inst1:

$ nova volume-attach inst1 0a76cc89-2891-45de-a728-3e65aaf1066c
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 0a76cc89-2891-45de-a728-3e65aaf1066c |
| serverId | 81cbe07a-965f-45af-b62f-ca9c0e866180 |
| volumeId | 0a76cc89-2891-45de-a728-3e65aaf1066c |
+----------+--------------------------------------+
$ nova volume-attach inst1 5573016a-77ee-4029-9ac1-9b30cbdcfe42
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdc                             |
| id       | 5573016a-77ee-4029-9ac1-9b30cbdcfe42 |
| serverId | 81cbe07a-965f-45af-b62f-ca9c0e866180 |
| volumeId | 5573016a-77ee-4029-9ac1-9b30cbdcfe42 |
+----------+--------------------------------------+

6. Login to instance 
ssh cirros.0.210
sudo -i 
  mkfs.ext4 /dev/vdb
  mkfs.ext4 /dev/vdc
  mkdir /media/unenc && mkdir /media/enc
  mount /dev/vdb /media/unenc && mount /dev/vdc /media/enc
  echo "unenc `hostname` `date`" > /media/unenc/test_unenc
  mount ..
     /dev/vdb on /media/unenc type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
     /dev/vdc on /media/enc type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)

  echo "enc `hostname` `date`" > /media/enc/test_enc
  umount /media/enc && umount /media/unenc
  ls /media/enc/
     lost+found  test_enc
Exit from instance

7. Detach from instance:
$ nova volume-detach inst1 0a76cc89-2891-45de-a728-3e65aaf1066c
$ nova volume-detach inst1 5573016a-77ee-4029-9ac1-9b30cbdcfe42

8. cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name         | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 0a76cc89-2891-45de-a728-3e65aaf1066c | available | unenc_Cinder | 1    | -           | false    |             |
| 1dac35d1-2384-4682-beda-0ae159e30466 | available | -            | 1    | LUKS        | true     |             |
| 5573016a-77ee-4029-9ac1-9b30cbdcfe42 | available | Enc_Cinder   | 1    | LUKS        | false    |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

9. Cinder upload to glance both volumes.
cinder upload-to-image 0a76cc89-2891-45de-a728-3e65aaf1066c Unenc_From_Cinder
...
cinder upload-to-image 5573016a-77ee-4029-9ac1-9b30cbdcfe42 Enc_From_Cinder
...

10. Check ^ images

glance image-show 7fd5d05d-bae7-4f6d-9bdf-53b3c16d00b2    (Unenc_From_Cinder)
+------------------+-----------------------------------------------------------------+
| Property         | Value                                                           |
+------------------+-----------------------------------------------------------------+
| checksum         | c1d318ca58ec40526d01343531d74069                                |
| container_format | bare                                                            |
| created_at       | 2017-09-27T08:16:35Z                                            |
| direct_url       | swift+config://ref1/glance/7fd5d05d-bae7-4f6d-9bdf-53b3c16d00b2 |
| disk_format      | raw                                                             |
| id               | 7fd5d05d-bae7-4f6d-9bdf-53b3c16d00b2                            |
| min_disk         | 0                                                               |
| min_ram          | 0                                                               |
| name             | Unenc_From_Cinder                                               |
| owner            | 6a6b9ff12c8f4d908c441bd3ffb77b2a                                |
| protected        | False                                                           |
| size             | 1073741824                                                      |
| status           | active                                                          |
| tags             | []                                                              |
| updated_at       | 2017-09-27T08:16:49Z                                            |
| virtual_size     | None                                                            |
| visibility       | private                                                         |
+------------------+-----------------------------------------------------------------+

glance image-show fd002261-196d-4cc6-ac2a-5b19f5058f62   (Enc_From_Cinder) 
+--------------------------+-----------------------------------------------------------------+
| Property                 | Value                                                           |
+--------------------------+-----------------------------------------------------------------+
| checksum                 | d1a4fda6a5974f5f4684aa60d8337a72                                |
| cinder_encryption_key_id | 00000000-0000-0000-0000-000000000000                            |
| container_format         | bare                                                            |
| created_at               | 2017-09-27T08:16:57Z                                            |
| direct_url               | swift+config://ref1/glance/fd002261-196d-4cc6-ac2a-5b19f5058f62 |
| disk_format              | raw                                                             |
| id                       | fd002261-196d-4cc6-ac2a-5b19f5058f62                            |
| min_disk                 | 0                                                               |
| min_ram                  | 0                                                               |
| name                     | Enc_From_Cinder                                                 |
| owner                    | 6a6b9ff12c8f4d908c441bd3ffb77b2a                                |
| protected                | False                                                           |
| size                     | 1073741824                                                      |
| status                   | active                                                          |
| tags                     | []                                                              |
| updated_at               | 2017-09-27T08:17:12Z                                            |
| virtual_size             | None                                                            |
| visibility               | private                                                         |
+--------------------------+-----------------------------------------------------------------+

11. Download those images and verify their content:
# glance image-download --file Unenc_From_Cinder.raw  7fd5d05d-bae7-4f6d-9bdf-53b3c16d00b2
# glance image-download --file Enc_From_Cinder.raw fd002261-196d-4cc6-ac2a-5b19f5058f62


12. Verify files 

$ file Unenc_From_Cinder.raw
Unenc_From_Cinder.raw: Linux rev 1.0 ext4 filesystem data, UUID=579faba5-4f3f-4444-82a8-9be7d34876b2 (extents) (large files) (huge files)

$ file file Enc_From_Cinder.raw
file:                cannot open (No such file or directory)
Enc_From_Cinder.raw: LUKS encrypted file, ver 1 [aes, xts-plain64, sha256] UUID: 193d5fcc-58af-4657-af11-e76994459aa4


Trying to open LUKS file:
# losetup /dev/loop2 Enc_From_Cinder.raw 
# echo "123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234" | cryptsetup luksOpen /dev/loop2 Verify
# mkdir -p /media/test
# mount /dev/mapper/Verify /media/test
# ll /media/test/
total 20
drwx------. 2 root root 16384 Sep 18 06:00 lost+found
-rw-------. 1 root root    39 Sep 18 06:01 test_enc

# cat /media/test/test_enc 
enc inst1 Mon Sep 18 11:01:17 UTC 2017

13.  upload encrypted image to glance:

#glance image-create --file Enc_From_Cinder.raw --property encrypted=true --name Enc_Upload_3_Glance --disk-format raw --container-format bare --property  cinder_encryption_key_id=123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234
+------------------+-----------------------------------------------------------------+
| Property         | Value                                                           |
+------------------+-----------------------------------------------------------------+
| checksum         | d1a4fda6a5974f5f4684aa60d8337a72                                |
| container_format | bare                                                            |
| created_at       | 2017-09-27T09:33:25Z                                            |
| direct_url       | swift+config://ref1/glance/7c813c95-6b86-400b-a368-82f7f28e6ffd |
| disk_format      | raw                                                             |
| encrypted        | true                                                            |
| id               | 7c813c95-6b86-400b-a368-82f7f28e6ffd                            |
| min_disk         | 0                                                               |
| min_ram          | 0                                                               |
| name             | Enc_Upload_3_Glance                                             |
| owner            | 6a6b9ff12c8f4d908c441bd3ffb77b2a                                |
| protected        | False                                                           |
| size             | 1073741824                                                      |
| status           | active                                                          |
| tags             | []                                                              |
| updated_at       | 2017-09-27T09:33:39Z                                            |
| virtual_size     | None                                                            |
| visibility       | private                                                         |
+------------------+-----------------------------------------------------------------+

Take care if you fail to set cinder_encryption_key the uploaded image won't be encrtupted with correct key.
Volume will later fail to mount inside instance. 



14. Create new enc volume from new image
 
[stack@undercloud-0 ~]$ cinder create --image-id 7c813c95-6b86-400b-a368-82f7f28e6ffd --display-name From_Glance_Enc_Image2 --volume-type LUKS 2  -> notice size must be bigger 2 (1 will fail)
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2017-09-27T09:37:22.000000           |
| description                    | None                                 |
| encrypted                      | True                                 |
| id                             | e4344324-b800-4e3b-a64f-f600aab85f0d |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | From_Glance_Enc_Image2               |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 6a6b9ff12c8f4d908c441bd3ffb77b2a     |
| replication_status             | disabled                             |
| size                           | 2                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | 260b3a8c738e4ba9ac4f78f357c00652     |
| volume_type                    | LUKS                                 |
+--------------------------------+--------------------------------------+

[stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+------------------------+------+-------------+----------+-------------+
| ID                                   | Status    | Name                   | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------------------------+------+-------------+----------+-------------+
| 0a76cc89-2891-45de-a728-3e65aaf1066c | available | unenc_Cinder           | 1    | -           | false    |             |
| 5573016a-77ee-4029-9ac1-9b30cbdcfe42 | available | Enc_Cinder             | 1    | LUKS        | false    |             |
| e4344324-b800-4e3b-a64f-f600aab85f0d | available | From_Glance_Enc_Image2 | 2    | LUKS        | true     |             |


15. Attach to isnt1:
#nova volume-attach inst1 e4344324-b800-4e3b-a64f-f600aab85f0d
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | e4344324-b800-4e3b-a64f-f600aab85f0d |
| serverId | 81cbe07a-965f-45af-b62f-ca9c0e866180 |
| volumeId | e4344324-b800-4e3b-a64f-f600aab85f0d |
+----------+--------------------------------------+

16. ssh into instance
ssh cirros.0.210

On Vm:
# mount /dev/vdb /media/re-enc/   -> again failed.

# ls /media/re-enc/
lost+found  test_enc
# cat /media/re-enc/test_enc 
enc inst1 Wed Sep 27 09:08:38 UTC 2017

As execpted we got original data, there for verified.

Comment 47 Eric Harney 2017-09-27 13:07:49 UTC
(In reply to Tzach Shefi from comment #44)

> 13.  upload encrypted image to glance:
> 
> #glance image-create --file Enc_From_Cinder.raw --property encrypted=true
> --name Enc_Upload_3_Glance --disk-format raw --container-format bare
> --property 
> cinder_encryption_key_id=123456789abcdef123456789abcdef123456789abcdef1234567
> 89abcdef1234

Just for posterity / documentation folks: cinder_encryption_key_id here needs to be a UUID referring to an encryption key -- for the conf key manager this is all 0s and for Barbican deployments it's the UUID of the key.  I believe it worked in this test anyway because the conf key manager path in Cinder is not strictly checking that the key id is all 0s.

Comment 48 errata-xmlrpc 2017-09-28 16:31:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2821


Note You need to log in before you can comment on or make changes to this bug.