Bug 1804848

Summary: [OSP 16.0.1] Volume encryption keys deleted when snapshotting instances created from images with cinder_encryption_key_id set
Product: Red Hat OpenStack Reporter: Brian Rosmaita <brian.rosmaita>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED DUPLICATE QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 16.0 (Train)CC: amcleod, dasmith, eglynn, jhakimra, kchamart, lyarwood, sbauza, sgordon, vromanso
Target Milestone: z1   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
There is a known issue when all of the following conditions exist: (0) You are using the OpenStack Train release (or code from master (Ussuri development)) (1) cinder_encryption_key_id and cinder_encryption_key_deletion_policy are not included in the non_inheritable_image_properties setting in nova.conf. These properties are not included by default. (2) A user has created a volume of an encrypted volume-type in the Block Storage service (cinder). For example, Volume-1. (3) Using the Block Storage service, the user has uploaded the encrypted volume as an image to the Image service (glance). For example, Image-1. (4) Using the Compute service (nova), the user has attempted to boot a server from the image directly. Note: this is an unsupported action, the supported workflow is to use the image to boot-from-volume. (5) Although an unsupported action, if a user does (4), it currently results in a server in status ACTIVE but which is unusable because the operating system cannot be found. (6) Using the Compute service, the user requests the createImage action on the unusable server, resulting in the creation of Image-2. (7) Using the Image service, the user deletes Image-2 which has inherited the cinder_encryption_key_* properties from Image-1 and the encryption key is deleted. As a result, Image-1 is rendered non-decryptable so that it can no longer be used in the normal boot-from-volume workflow. The workaround for this issue is to add the cinder_encryption_key_id,cinder_encryption_key_deletion_policy properties to the non_inheritable_image_properties option in the [DEFAULT] section of nova.conf. Image-2 can be deleted and the encryption key used by Image-1 remains available.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-21 10:17:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Brian Rosmaita 2020-02-19 18:19:30 UTC 
This bug was initially created as a copy of Bug #1801255

I am copying this bug because:

The fix is scheduled for 16.0.2, but we want to get a "known issues" about it into the release notes for 16.0.1.



We have to take this seriously because of possible data loss, but the end user would have to boot an instance that is 'active' but unusable, and then do the image-create action on that instance.

Brief backstory: when a Cinder encrypted volume is uploaded as an image to Glance, a secret in barbican specific to that image is created.  Beginning with Train, the image metadata "cinder_encryption_key_deletion_policy": "on_image_deletion" is also put on such an image.  On image deletion, if Glance finds such metadata on an image, it will try to delete the barbican secret for the image.

The problem: direct boot of an instance from an image I-1 created from an encrypted volume is *not supported* by Nova.  (The workflow is that you use the image to boot an instance from a volume.)  However, if you do try a direct boot of I-1, Nova creates an instance S-1 in 'active' status (though it is unusable).  If a user does the Nova image-create action on S-1, a new image I-2 will be created *and* it will inherit the image properties from I-1, including the cinder_encryption_key_id and cinder_encryption_key_deletion_policy metadata.  Since I-2 has a reference to the barbican secret for I-1, when I-2 is deleted, the secret for I-1 will be deleted, thereby making it impossible to decrypt I-1.


Version-Release number of selected component (if applicable): 16 (Train)
This does not occur in earlier versions.


How reproducible: always


Steps to Reproduce:
1. in cinder: create volume V-1 of an encrypted volume-type
2. in cinder: use the upload-volume-to-image action to create I-1 in glance
3. in nova: boot an instance S-1 from I-1
4. in nova: do the image-create action on S-1, yielding image I-2 in glance
5. in glance: delete image I-2


Actual results:
The barbican secret whose uuid is contained in the cinder_encryption_key_id image property of image I-1 is deleted.


Expected results:
The barbican secret for image I-1 should *not* be deleted.


Additional info:
Nova has a non_inheritable_image_properties configuration option.  The 'cinder_encryption_key_*' properties should be added to this list, and then this situation will not occur.
Comment 2 Brian Rosmaita 2020-02-19 18:30:21 UTC 
See http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012641.html
for the complete write-up sent to the openstack mailing list.
Comment 3 Lee Yarwood 2020-02-21 10:17:46 UTC 

*** This bug has been marked as a duplicate of bug 1801255 ***