The cinder RBD driver appears to incorrectly handle a volume encrypt request and returns connection info missing the necessary metatdata for nova to make use of it. See the upstream LP bug for current info.
See bug 1230405 for the Nova side
(In reply to Sean Cohen from comment #3) > See bug 1230405 for the Nova side BZ#1230402
Matt Reidmann has been working on this issue in upstream He has root caused this and it seems this needs fixes across Nova, devstack-gate, tempest & Cinder. In short, Nova encrytion drivers need to understand/support RBD volumes to be able to get the tempest encrypted volumes test pass, since nova doesn't support it yet, a new flag to bypass encrypted test in tempest was introduced, ceph job made to pass tempest by using this flag to bypass enc vols test. In details: * list email that explains the issue * http://lists.openstack.org/pipermail/openstack-dev/2015-June/068117.html * Cinder patch to enable 'encryption' for all volume drivers (if specified by admin/user) * https://review.openstack.org/#/c/193673/ * NOTE: GlusterFS CI job fails here and is expected as nova fails to encrypt hence attach the volume * See http://logs.openstack.org/73/193673/3/check/check-tempest-dsvm-full-glusterfs-nv/1d497c8/console.html#_2015-06-29_08_25_14_612 * Nova fix to handle the case where cinder volume driver asks for encryption but Nova doesn't have support for it * https://review.openstack.org/#/c/193830/ * devstack-gate change to ensure Ceph job disables 'encrypted' volume tempest tests * https://review.openstack.org/#/c/193835/3 * Tempest new config option to skip encrypted volume tests * https://review.openstack.org/#/c/193831/ thanx, deepak
FYI, the nova spec wasn't approved for Mitaka so this will push to N.
I don't see any evidence upstream that this is functional in Newton.
Can we have more info on this one? What's the status? It's not clear to me if we are missing anything and what we are missing... Thanks!
I cleared the dependency on #b1302261 which has basically tracked some fixes that are all upstream now. This bug tracks the the need for RBD luks-based encryption which does have a dependency on libvirt change - we need to find that bz and get the dependency tracked here. Working on closing out bz1302261 which doesn't seem to be needed anymore.
Could get and update please
Just adding my voice so that demand in the outside world is known: I am working on an OSP deployment design with a customer today and the capability to encrypt volumes living on RBD using Cinder and Nova has been requested as a key feature - Data-at-Rest security, etc.
I now have a second, larger customer, who is making an issue of not having the ability to do Data-at-Rest encryption via this driver, so this is definitely becoming bigger in my region.
*** Bug 1419710 has been marked as a duplicate of this bug. ***
*** Bug 1419712 has been marked as a duplicate of this bug. ***
Above mentioned patches for cinder and nova are in following builds. openstack-cinder-12.0.0-0.20180227162609.7d27804.el7ost openstack-nova-17.0.0-0.20180223162252.a4a53bf.el7ost
Eric, One more question/issue with one more test case related to this RFE. 1. Create encrypted volume from cirros.raw image - works fine. # cinder create --display-name 'encryptedVolFromCirros' --volume-type LUKS 2 --image 7b27430e-e692-40a2-bba8-11d141256e0e 2. Upload volume to Glance image - works fine. #cinder --debug upload-to-image 67a779fe-4b39-4361-a6c0-4f5e0ed4925a EncCirrosImage --container-format bare --disk-format raw 3. Boot instance from image - well Nova says instance is up but I can't reach Cirros's console fear something didn't work in the process. # nova flavor-create tiny2 auto 512 3 1 # nova boot inst2 --image 613fff0f-e121-45b9-b41d-5e07c3703ecd --flavor tiny2 --nic net-id=75d9bf66-1e14-4422-beba-4f1600dc5ec6 Instance is active, yet virsh console on compute node with instance-0000000a doesn't get cirros user/password prompt. nova list | 9588592e-d28f-4bf5-8386-4c6b58b8bf26 | inst2 | ACTIVE | - | Running | internal=192.168.0.21 | +--------------------------------------+-------+-------- Any ideas did I miss anything? Another instances booted from a none encrypted Cirros image works fine.
Eric, The above probably failed due to (overcloud) [stack@undercloud-0 ~]$ glance image-show 613fff0f-e121-45b9-b41d-5e07c3703ecd +--------------------------+----------------------------------------------------------------------------------+ | Property | Value | +--------------------------+----------------------------------------------------------------------------------+ | checksum | 9db1957f0fb9a50b3d15c165ec9f5601 | | cinder_encryption_key_id | 00000000-0000-0000-0000-000000000000 I recall messing about with this a while back. From another bz -> #glance image-create --file Enc_From_Cinder.raw --property encrypted=true --name Enc_Upload_3_Glance --disk-format raw --container-format bare --property cinder_encryption_key_id=123456789abcdef123456789abcdef123456789abcdef123456789ab Setting these two: --property encrypted=true --property cinder_encryption_key_id=123456789abcdef123456789abcdef123456789abcdef123456789ab glance image-update 613fff0f-e121-45b9-b41d-5e07c3703ecd --property encrypted=true --property cinder_encryption_key_id=04d6b077d60e323711b37813b3a68a71 Didn't help, glance image-show looks better +--------------------------+----------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ glance image-update 613fff0f-e121-45b9-b41d-5e07c3703ecd --property encrypted=true --property cinder_encryption_key_id=04d6b077d60e323711b37813b3a68a71 .. | cinder_encryption_key_id | 04d6b077d60e323711b37813b3a68a71 | .. | | encrypted | true Yet again booted instance from this image says it's up but not cirros prompt.
Hey Eric, Yep it just re-hit me cinder_encryption_key_id isn't the key but the key's ID and in fixed_key's case -> 000000.. I've updated my image and it now looks like this: (overcloud) [stack@undercloud-0 ~]$ glance image-show 613fff0f-e121-45b9-b41d-5e07c3703ecd +--------------------------+----------------------------------------------------------------------------------+ | Property | Value | +--------------------------+----------------------------------------------------------------------------------+ | checksum | 9db1957f0fb9a50b3d15c165ec9f5601 | | cinder_encryption_key_id | 00000000-0000-0000-0000-000000000000 | | container_format | bare | | created_at | 2018-03-29T03:39:44Z | | direct_url | rbd://91b780b2-309b-11e8-906f-525400bb0f8f/images/613fff0f-e121-45b9-b41d- | | | 5e07c3703ecd/snap | | disk_format | raw | | encrypted | true Yet same as before Cirros fails to boot/prompt despite Nova's showing status as up. Still messing around trying to get this to work. Out of ideas on what to check for next, if this is even supported at all. Cinder encrypted volume from raw Cirros image, then upload volume to Glance and boot instance from it.
BTW creating an encrypted volume form image on rbd works but on LVM fails Avi's bug 1560244.
Eric can you take a look at Benny's new bug might be possible blocking bug. Maybe you have some insight/input. https://bugzilla.redhat.com/show_bug.cgi?id=1565039
Tzach, Booting an instance from a Glance image that was itself created from an encrypted Cinder volume isn't a supported use case and never has been. Nova is presenting the RAW LUKS encrypted disk to the instance thus the lack of any console output. Let me know if there are any other use cases that you have concerns about. Regards, Lee
Great thanks for letting me know. Is there any use case for an encrypted image at all? I'll raise this during today's meeting. The only two use cases I could think of are: 1. Encrypt the image in case you wish to export/import a secure version. 2. Create an encrypted volume from an encrypted image, but then again during volume create you can set encrypted vol type. Argo no need for image to be encrypted to being with. As a Nova expert, how would I go about creating "secure" instance where all of it's disks would be encrypted (boot vol and ephemeral)? Or can this only be done if you boot an instance from an encrypted Cinder volume, is this supported?
(In reply to Tzach Shefi from comment #52) > Great thanks for letting me know. > > Is there any use case for an encrypted image at all? > I'll raise this during today's meeting. I guess, if the tenant user doesn't fully trust the image service and/or store but that's a little tenuous if I'm honest. I'm sure the Barbican and Glance folks will have more of an insight here. > The only two use cases I could think of are: > > 1. Encrypt the image in case you wish to export/import a secure version. > > 2. Create an encrypted volume from an encrypted image, but then again during > volume create you can set encrypted vol type. Argo no need for image to be > encrypted to being with. > > As a Nova expert, how would I go about creating "secure" instance where all > of it's disks would be encrypted (boot vol and ephemeral)? We can encrypt ephemeral storage but we can't seed that with data from Glance AFAIK in the same way that Cinder is able to, it's a limitation and valid RFE IMHO : https://docs.openstack.org/security-guide/tenant-data/data-encryption.html#ephemeral-disk-encryption I'd really like to clean up this area once QEMU/Libvirt land support for both RAW and qcow2 based native LUKS decryption, currently planned for 7.6. > Or can this only be done if you boot an instance from an encrypted Cinder > volume, is this supported? Yes booting from an encrypted Cinder volume is supported and if I'm honest is the way that upstream is pushing over refactoring the non-local ephemeral storage code such as RBD etc.
FYI another bug found while verifying this RFE. Backup of an encrypted volume fails same for attached vs none attached volumes. None encrypted volume backup works fine. Looking at the logs I suspect again related to some Barbican/keystone issue. https://bugzilla.redhat.com/show_bug.cgi?id=1567607
Another issue nova this time, re-attach of an encrypted volume to an instance fails. https://bugzilla.redhat.com/show_bug.cgi?id=1567614
We still have some generic Cinder Barbican issues -> BZ1412823 has been moved to modified state. This RFE is this still stuck as well added blocked by that BZ. However comments #54-55 seam to be fluke cases. Was unable to reproduce them hence removed depends on 1412823.
Verified on: openstack-cinder-12.0.1-0.20180418194613.c476898.el7ost.noarch Most of the test cases passed, one problem found (probably not a bug) where an encrypted volume was uploaded to Glance, then from that image a new encrypted volume was created this new volume failed to attach to an instance. I'm guessing by doing so I probably encrypted the volume twice meaning I won't ever get to original data any way, so not a valid case to begin with. Opened bug about it here: https://bugzilla.redhat.com/show_bug.cgi?id=1573870
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086