Bug 1230405 - [RFE] Cinder RBD driver encryption support
Summary: [RFE] Cinder RBD driver encryption support
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 8.0 (Liberty)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: Upstream M3
: 13.0 (Queens)
Assignee: Eric Harney
QA Contact: Avi Avraham
URL:
Whiteboard:
: 1419710 1419712 (view as bug list)
Depends On: 1301026 1821539 1301019 1301021 1305024 1406796 1406803 1518998 1567614 1631239
Blocks: 1411525 1419948 1442136 1230402 1262120 1273812
TreeView+ depends on / blocked
 
Reported: 2015-06-10 20:10 UTC by Jon Bernard
Modified: 2020-04-07 03:08 UTC (History)
25 users (show)

Fixed In Version: openstack-cinder-12.0.0-0.20180227162609.7d27804.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:26:22 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1463525 None None None Never
OpenStack gerrit 523958 'None' MERGED libvirt: QEMU native LUKS decryption for encrypted volumes 2020-07-28 09:09:25 UTC
OpenStack gerrit 534811 'None' MERGED RBD: Support encrypted volumes 2020-07-28 09:09:25 UTC
Red Hat Knowledge Base (Solution) 2137751 None None None 2016-02-05 11:43:00 UTC
Red Hat Product Errata RHEA-2018:2086 normal SHIPPED_LIVE Red Hat OpenStack Platform 13.0 Enhancement Advisory 2018-06-28 19:51:39 UTC

Internal Links: 1560624

Description Jon Bernard 2015-06-10 20:10:49 UTC
The cinder RBD driver appears to incorrectly handle a volume encrypt request and returns connection info missing the necessary metatdata for nova to make use of it.  See the upstream LP bug for current info.

Comment 3 Sean Cohen 2015-06-10 20:13:36 UTC
See bug 1230405 for the Nova side

Comment 4 Sergey Gotliv 2015-06-20 11:40:52 UTC
(In reply to Sean Cohen from comment #3)
> See bug 1230405 for the Nova side

BZ#1230402

Comment 5 Deepak C Shetty 2015-07-02 15:09:45 UTC
Matt Reidmann has been working on this issue in upstream

He has root caused this and it seems this needs fixes across Nova, devstack-gate, tempest & Cinder.

In short, Nova encrytion drivers need to understand/support RBD volumes to be able to get the tempest encrypted volumes test pass, since nova doesn't support it yet, a new flag to bypass encrypted test in tempest was introduced, ceph job made to pass tempest by using this flag to bypass enc vols test.

In details:

* list email that explains the issue
  * http://lists.openstack.org/pipermail/openstack-dev/2015-June/068117.html

* Cinder patch to enable 'encryption' for all volume drivers (if specified by admin/user)

  * https://review.openstack.org/#/c/193673/

  * NOTE: GlusterFS CI job fails here and is expected as nova fails to encrypt hence attach the volume

  * See http://logs.openstack.org/73/193673/3/check/check-tempest-dsvm-full-glusterfs-nv/1d497c8/console.html#_2015-06-29_08_25_14_612

* Nova fix to handle the case where cinder volume driver asks for encryption but Nova doesn't have support for it

  * https://review.openstack.org/#/c/193830/

* devstack-gate change to ensure Ceph job disables 'encrypted' volume tempest tests

  * https://review.openstack.org/#/c/193835/3

* Tempest new config option to skip encrypted volume tests

  * https://review.openstack.org/#/c/193831/

thanx,
deepak

Comment 7 Jon Bernard 2016-01-06 22:35:46 UTC
FYI, the nova spec wasn't approved for Mitaka so this will push to N.

Comment 9 Eric Harney 2016-07-21 16:36:25 UTC
I don't see any evidence upstream that this is functional in Newton.

Comment 10 seb 2016-07-28 13:06:35 UTC
Can we have more info on this one? What's the status? It's not clear to me if we are missing anything and what we are missing...

Thanks!

Comment 11 Paul Grist 2016-12-13 01:19:40 UTC
I cleared the dependency on #b1302261 which has basically tracked some fixes that are all upstream now. This bug tracks the the need for RBD luks-based encryption which does have a dependency on libvirt change - we need to find that bz and get the dependency tracked here.  Working on closing out bz1302261 which doesn't seem to be needed anymore.

Comment 16 Edu Alcaniz 2017-05-08 08:22:45 UTC
Could get and update please

Comment 21 John Apple II 2017-08-04 02:25:40 UTC
Just adding my voice so that demand in the outside world is known: I am working on an OSP deployment design with a customer today and the capability to encrypt volumes living on RBD using Cinder and Nova has been requested as a key feature - Data-at-Rest security, etc.

Comment 22 John Apple II 2017-09-12 03:15:54 UTC
I now have a second, larger customer, who is making an issue of not having the ability to do Data-at-Rest encryption via this driver, so this is definitely becoming bigger in my region.

Comment 28 Sean Cohen 2017-11-17 13:13:48 UTC
*** Bug 1419710 has been marked as a duplicate of this bug. ***

Comment 29 Sean Cohen 2017-11-17 13:25:44 UTC
*** Bug 1419712 has been marked as a duplicate of this bug. ***

Comment 35 Jon Schlueter 2018-03-01 17:26:52 UTC
Above mentioned patches for cinder and nova are in following builds.

openstack-cinder-12.0.0-0.20180227162609.7d27804.el7ost
openstack-nova-17.0.0-0.20180223162252.a4a53bf.el7ost

Comment 43 Tzach Shefi 2018-03-29 06:05:54 UTC
Eric, 
One more question/issue with one more test case related to this RFE. 
  
1. Create encrypted volume from cirros.raw image - works fine. 
# cinder create --display-name 'encryptedVolFromCirros' --volume-type LUKS 2 --image 7b27430e-e692-40a2-bba8-11d141256e0e


2. Upload volume to Glance image - works fine.
#cinder --debug upload-to-image 67a779fe-4b39-4361-a6c0-4f5e0ed4925a EncCirrosImage --container-format bare --disk-format raw

3. Boot instance from image - well Nova says instance is up but I can't reach Cirros's console fear something didn't work in the process.
# nova flavor-create tiny2 auto 512 3 1  
# nova boot inst2 --image 613fff0f-e121-45b9-b41d-5e07c3703ecd --flavor tiny2 --nic net-id=75d9bf66-1e14-4422-beba-4f1600dc5ec6

Instance is active,  yet virsh console on compute node with instance-0000000a  doesn't get cirros user/password prompt. 

nova list

| 9588592e-d28f-4bf5-8386-4c6b58b8bf26 | inst2 | ACTIVE | -          | Running     | internal=192.168.0.21             |
+--------------------------------------+-------+--------

Any ideas did I miss anything?

Another instances booted from a none encrypted Cirros image works fine.

Comment 44 Tzach Shefi 2018-03-29 08:33:37 UTC
Eric, 

The above probably failed due to 

(overcloud) [stack@undercloud-0 ~]$ glance image-show 613fff0f-e121-45b9-b41d-5e07c3703ecd
+--------------------------+----------------------------------------------------------------------------------+
| Property                 | Value                                                                            |
+--------------------------+----------------------------------------------------------------------------------+
| checksum                 | 9db1957f0fb9a50b3d15c165ec9f5601                                                 |
| cinder_encryption_key_id | 00000000-0000-0000-0000-000000000000  

I recall messing about with this a while back. 
From another bz -> 
#glance image-create --file Enc_From_Cinder.raw --property encrypted=true --name Enc_Upload_3_Glance --disk-format raw --container-format bare --property  cinder_encryption_key_id=123456789abcdef123456789abcdef123456789abcdef123456789ab

Setting these two: 
--property encrypted=true
--property  cinder_encryption_key_id=123456789abcdef123456789abcdef123456789abcdef123456789ab

glance image-update 613fff0f-e121-45b9-b41d-5e07c3703ecd --property encrypted=true --property  cinder_encryption_key_id=04d6b077d60e323711b37813b3a68a71

Didn't help, glance image-show looks better 
+--------------------------+----------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ glance image-update 613fff0f-e121-45b9-b41d-5e07c3703ecd --property encrypted=true --property  cinder_encryption_key_id=04d6b077d60e323711b37813b3a68a71
..

| cinder_encryption_key_id | 04d6b077d60e323711b37813b3a68a71                                                 |
..                                                                            |
| encrypted                | true 

Yet again booted instance from this image says it's up but not cirros prompt.

Comment 47 Tzach Shefi 2018-04-01 05:18:30 UTC
Hey Eric, 

Yep it just re-hit me cinder_encryption_key_id isn't the key but the key's ID and in fixed_key's case -> 000000.. 

I've updated my image and it now looks like this:

(overcloud) [stack@undercloud-0 ~]$ glance image-show 613fff0f-e121-45b9-b41d-5e07c3703ecd
+--------------------------+----------------------------------------------------------------------------------+
| Property                 | Value                                                                            |
+--------------------------+----------------------------------------------------------------------------------+
| checksum                 | 9db1957f0fb9a50b3d15c165ec9f5601                                                 |
| cinder_encryption_key_id | 00000000-0000-0000-0000-000000000000                                             |
| container_format         | bare                                                                             |
| created_at               | 2018-03-29T03:39:44Z                                                             |
| direct_url               | rbd://91b780b2-309b-11e8-906f-525400bb0f8f/images/613fff0f-e121-45b9-b41d-       |
|                          | 5e07c3703ecd/snap                                                                |
| disk_format              | raw                                                                              |
| encrypted                | true 

Yet same as before Cirros fails to boot/prompt despite Nova's showing status as up. 

Still messing around trying to get this to work. 
Out of ideas on what to check for next, if this is even supported at all.
Cinder encrypted volume from raw Cirros image, then upload volume to Glance and boot instance from it.

Comment 49 Tzach Shefi 2018-04-01 10:13:45 UTC
BTW creating an encrypted volume form image on rbd works but on LVM fails Avi's bug 1560244.

Comment 50 Tzach Shefi 2018-04-09 12:32:29 UTC
Eric can you take a look at Benny's new bug might be possible blocking bug. 
Maybe you have some insight/input.  
https://bugzilla.redhat.com/show_bug.cgi?id=1565039

Comment 51 Lee Yarwood 2018-04-11 05:51:31 UTC
Tzach, 

Booting an instance from a Glance image that was itself created from an encrypted Cinder volume isn't a supported use case and never has been. Nova is presenting the RAW LUKS encrypted disk to the instance thus the lack of any console output.

Let me know if there are any other use cases that you have concerns about.

Regards,

Lee

Comment 52 Tzach Shefi 2018-04-12 07:40:51 UTC
Great thanks for letting me know. 

Is there any use case for an encrypted image at all?
I'll raise this during today's meeting. 

The only two use cases I could think of are:

1. Encrypt the image in case you wish to export/import a secure version.  

2. Create an encrypted volume from an encrypted image, but then again during volume create you can set encrypted vol type. Argo no need for image to be encrypted to being with. 

As a Nova expert, how would I go about creating "secure" instance where all of it's disks would be encrypted (boot vol and ephemeral)? 

Or can this only be done if you boot an instance from an encrypted Cinder volume, is this supported?

Comment 53 Lee Yarwood 2018-04-12 14:24:51 UTC
(In reply to Tzach Shefi from comment #52)
> Great thanks for letting me know. 
> 
> Is there any use case for an encrypted image at all?
> I'll raise this during today's meeting. 

I guess, if the tenant user doesn't fully trust the image service and/or store but that's a little tenuous if I'm honest. I'm sure the Barbican and Glance folks will have more of an insight here.

> The only two use cases I could think of are:
> 
> 1. Encrypt the image in case you wish to export/import a secure version.  
> 
> 2. Create an encrypted volume from an encrypted image, but then again during
> volume create you can set encrypted vol type. Argo no need for image to be
> encrypted to being with. 
> 
> As a Nova expert, how would I go about creating "secure" instance where all
> of it's disks would be encrypted (boot vol and ephemeral)? 

We can encrypt ephemeral storage but we can't seed that with data from Glance AFAIK in the same way that Cinder is able to, it's a limitation and valid RFE IMHO :

https://docs.openstack.org/security-guide/tenant-data/data-encryption.html#ephemeral-disk-encryption

I'd really like to clean up this area once QEMU/Libvirt land support for both RAW and qcow2 based native LUKS decryption, currently planned for 7.6.

> Or can this only be done if you boot an instance from an encrypted Cinder
> volume, is this supported?

Yes booting from an encrypted Cinder volume is supported and if I'm honest is the way that upstream is pushing over refactoring the non-local ephemeral storage code such as RBD etc.

Comment 54 Tzach Shefi 2018-04-15 12:16:51 UTC
FYI another bug found while verifying this RFE. 

Backup of an encrypted volume fails same for attached vs none attached volumes. None encrypted volume backup works fine. 

Looking at the logs I suspect again related to some Barbican/keystone issue. 

https://bugzilla.redhat.com/show_bug.cgi?id=1567607

Comment 55 Tzach Shefi 2018-04-15 13:29:44 UTC
Another issue nova this time,  re-attach of an encrypted volume to an instance fails. 

https://bugzilla.redhat.com/show_bug.cgi?id=1567614

Comment 56 Tzach Shefi 2018-04-26 12:00:03 UTC
We still have some generic Cinder Barbican issues -> BZ1412823 has been moved to modified state. This RFE is this still stuck as well added blocked by that BZ. 

However comments #54-55 seam to be fluke cases. 
Was unable to reproduce them hence removed depends on 1412823.

Comment 57 Tzach Shefi 2018-05-02 12:58:57 UTC
Verified on:
openstack-cinder-12.0.1-0.20180418194613.c476898.el7ost.noarch

Most of the test cases passed, one problem found (probably not a bug) where an encrypted volume was uploaded to Glance, then from that image a new encrypted volume was created this new volume failed to attach to an instance. I'm guessing by doing so I probably encrypted the volume twice meaning I won't ever get to original data any way, so not a valid case to begin with. 

Opened bug about it here:
https://bugzilla.redhat.com/show_bug.cgi?id=1573870

Comment 60 errata-xmlrpc 2018-06-27 13:26:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.