Bug 1801255

Summary: [OSP 16.0.2] Volume encryption keys deleted when snapshotting instances created from images with cinder_encryption_key_id set
Product: Red Hat OpenStack Reporter: Brian Rosmaita <brian.rosmaita>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.0 (Train)CC: abishop, dasmith, eglynn, gcharot, jhakimra, kchamart, mgarciac, sbauza, sgordon, vromanso
Target Milestone: z2Keywords: Patch, Triaged
Target Release: 16.0 (Train on RHEL 8.1)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-nova-20.1.1-0.20200211145524.8363905.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1804657 (view as bug list) Environment:
Last Closed: 2020-05-14 12:10:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1804657, 1804659    

Description Brian Rosmaita 2020-02-10 14:20:30 UTC
We have to take this seriously because of possible data loss, but the end user would have to boot an instance that is 'active' but unusable, and then do the image-create action on that instance.

Brief backstory: when a Cinder encrypted volume is uploaded as an image to Glance, a secret in barbican specific to that image is created.  Beginning with Train, the image metadata "cinder_encryption_key_deletion_policy": "on_image_deletion" is also put on such an image.  On image deletion, if Glance finds such metadata on an image, it will try to delete the barbican secret for the image.

The problem: direct boot of an instance from an image I-1 created from an encrypted volume is *not supported* by Nova.  (The workflow is that you use the image to boot an instance from a volume.)  However, if you do try a direct boot of I-1, Nova creates an instance S-1 in 'active' status (though it is unusable).  If a user does the Nova image-create action on S-1, a new image I-2 will be created *and* it will inherit the image properties from I-1, including the cinder_encryption_key_id and cinder_encryption_key_deletion_policy metadata.  Since I-2 has a reference to the barbican secret for I-1, when I-2 is deleted, the secret for I-1 will be deleted, thereby making it impossible to decrypt I-1.


Version-Release number of selected component (if applicable): 16 (Train)
This does not occur in earlier versions.


How reproducible: always


Steps to Reproduce:
1. in cinder: create volume V-1 of an encrypted volume-type
2. in cinder: use the upload-volume-to-image action to create I-1 in glance
3. in nova: boot an instance S-1 from I-1
4. in nova: do the image-create action on S-1, yielding image I-2 in glance
5. in glance: delete image I-2


Actual results:
The barbican secret whose uuid is contained in the cinder_encryption_key_id image property of image I-1 is deleted.


Expected results:
The barbican secret for image I-1 should *not* be deleted.


Additional info:
Nova has a non_inheritable_image_properties configuration option.  The 'cinder_encryption_key_*' properties should be added to this list, and then this situation will not occur.

Comment 1 Brian Rosmaita 2020-02-10 14:28:27 UTC
Alan, do you know if the nova non_inheritable_image_properties config option is set by Director?

Comment 2 Alan Bishop 2020-02-10 14:43:03 UTC
@Brian,

No, there's no sign of tripleo setting that config option. I checked THT, puppet-tripleo and puppet-nova.

Comment 3 Brian Rosmaita 2020-02-10 14:47:09 UTC
Since RHOS is using the default value for non_inheritable_image_properties, the upstream patch changing this default will be effective.  If operators are changing this on their own, the default will be ignored in favor of the configured value.  May need to add some documentation about this.

Comment 4 Brian Rosmaita 2020-02-13 19:38:37 UTC
Feedback upstream is that https://review.opendev.org/706298 won't be accepted.  Instead, this will be covered by the fix to BZ #1801282.

I think we still need to get the info out to customers running RHOSP 16, because setting a value for the non_inheritable_image_properties config option in nova can avoid some possible unpleasantness in the short term.

Comment 6 Lee Yarwood 2020-02-14 11:37:21 UTC
*** Bug 1801282 has been marked as a duplicate of this bug. ***

Comment 7 Lee Yarwood 2020-02-21 10:17:46 UTC
*** Bug 1804848 has been marked as a duplicate of this bug. ***

Comment 15 errata-xmlrpc 2020-05-14 12:10:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2154