Bug 1801255 - [OSP 16.0.2] Volume encryption keys deleted when snapshotting instances created from images with cinder_encryption_key_id set
Summary: [OSP 16.0.2] Volume encryption keys deleted when snapshotting instances creat...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 16.0 (Train)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z2
: 16.0 (Train on RHEL 8.1)
Assignee: Lee Yarwood
QA Contact: OSP DFG:Compute
URL:
Whiteboard:
: 1801282 1804848 (view as bug list)
Depends On:
Blocks: 1804657 1804659
TreeView+ depends on / blocked
 
Reported: 2020-02-10 14:20 UTC by Brian Rosmaita
Modified: 2023-03-21 19:30 UTC (History)
10 users (show)

Fixed In Version: openstack-nova-20.1.1-0.20200211145524.8363905.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1804657 (view as bug list)
Environment:
Last Closed: 2020-05-14 12:10:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1852106 0 None None None 2020-02-10 14:22:37 UTC
OpenStack gerrit 707738 0 None MERGED Reject boot request for unsupported images 2020-10-07 15:44:20 UTC
OpenStack gerrit 708126 0 None MERGED Absolutely-non-inheritable image properties 2020-10-07 15:44:20 UTC
OpenStack gerrit 708577 0 None MERGED Reject boot request for unsupported images 2020-10-07 15:44:28 UTC
Red Hat Issue Tracker OSP-23516 0 None None None 2023-03-21 19:30:49 UTC
Red Hat Product Errata RHBA-2020:2154 0 None None None 2020-05-14 12:10:45 UTC

Internal Links: 1801282

Description Brian Rosmaita 2020-02-10 14:20:30 UTC
We have to take this seriously because of possible data loss, but the end user would have to boot an instance that is 'active' but unusable, and then do the image-create action on that instance.

Brief backstory: when a Cinder encrypted volume is uploaded as an image to Glance, a secret in barbican specific to that image is created.  Beginning with Train, the image metadata "cinder_encryption_key_deletion_policy": "on_image_deletion" is also put on such an image.  On image deletion, if Glance finds such metadata on an image, it will try to delete the barbican secret for the image.

The problem: direct boot of an instance from an image I-1 created from an encrypted volume is *not supported* by Nova.  (The workflow is that you use the image to boot an instance from a volume.)  However, if you do try a direct boot of I-1, Nova creates an instance S-1 in 'active' status (though it is unusable).  If a user does the Nova image-create action on S-1, a new image I-2 will be created *and* it will inherit the image properties from I-1, including the cinder_encryption_key_id and cinder_encryption_key_deletion_policy metadata.  Since I-2 has a reference to the barbican secret for I-1, when I-2 is deleted, the secret for I-1 will be deleted, thereby making it impossible to decrypt I-1.


Version-Release number of selected component (if applicable): 16 (Train)
This does not occur in earlier versions.


How reproducible: always


Steps to Reproduce:
1. in cinder: create volume V-1 of an encrypted volume-type
2. in cinder: use the upload-volume-to-image action to create I-1 in glance
3. in nova: boot an instance S-1 from I-1
4. in nova: do the image-create action on S-1, yielding image I-2 in glance
5. in glance: delete image I-2


Actual results:
The barbican secret whose uuid is contained in the cinder_encryption_key_id image property of image I-1 is deleted.


Expected results:
The barbican secret for image I-1 should *not* be deleted.


Additional info:
Nova has a non_inheritable_image_properties configuration option.  The 'cinder_encryption_key_*' properties should be added to this list, and then this situation will not occur.

Comment 1 Brian Rosmaita 2020-02-10 14:28:27 UTC
Alan, do you know if the nova non_inheritable_image_properties config option is set by Director?

Comment 2 Alan Bishop 2020-02-10 14:43:03 UTC
@Brian,

No, there's no sign of tripleo setting that config option. I checked THT, puppet-tripleo and puppet-nova.

Comment 3 Brian Rosmaita 2020-02-10 14:47:09 UTC
Since RHOS is using the default value for non_inheritable_image_properties, the upstream patch changing this default will be effective.  If operators are changing this on their own, the default will be ignored in favor of the configured value.  May need to add some documentation about this.

Comment 4 Brian Rosmaita 2020-02-13 19:38:37 UTC
Feedback upstream is that https://review.opendev.org/706298 won't be accepted.  Instead, this will be covered by the fix to BZ #1801282.

I think we still need to get the info out to customers running RHOSP 16, because setting a value for the non_inheritable_image_properties config option in nova can avoid some possible unpleasantness in the short term.

Comment 6 Lee Yarwood 2020-02-14 11:37:21 UTC
*** Bug 1801282 has been marked as a duplicate of this bug. ***

Comment 7 Lee Yarwood 2020-02-21 10:17:46 UTC
*** Bug 1804848 has been marked as a duplicate of this bug. ***

Comment 15 errata-xmlrpc 2020-05-14 12:10:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2154


Note You need to log in before you can comment on or make changes to this bug.