Bug 1889443

Summary: [RFE] Support for rotating and migrating Fernet keys used to encrypt and decrypt keystone credentials
Product: Red Hat OpenStack Reporter: Lance Bragstad <lbragsta>
Component: tripleo-ansibleAssignee: OSP Team <rhos-maint>
Status: NEW --- QA Contact: Jeremy Agee <jagee>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 17.0 (Wallaby)CC: alee, hrybacki
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Lance Bragstad 2020-10-19 16:55:04 UTC
Description of problem:

Currently, it's possible for deployers to unintentionally lock all users out of their credential data.

Keystone supports a credentials backend, which is primarily used for storing MFA secrets and ec2 credentials (not passwords). In the Newton release, these credentials were encrypted so they weren't stored in plaintext. The upstream community re-used the existing token enryption
mechanism to encrypt credential data. As a result, credential encryption uses an entirely separate key repository from token encryption, but both repositories are managed using similar rotation flows.

The credential backend stores a hash of the key used to encrypt it. The `keystone-manage credential_rotate` command relies on this information to ensure it doesn't overrotate the keys. If keys are overrotated, the encrypted data is unrecoverable. Deployers must use the following flow to rotate credential fernet keys safely:

1.) setup the credential repository (`keystone-manage credential_setup`)
2.) rotate the credential keys (`keystone-manage credential_rotate`)
3.) re-enrypt credentials using the newest fernet key (`keystone-manage credential_migrate`)

Steps #2 and #3 can be done as many times as needed, so long as they're done in lock-step.

Currently, we don't have anything in TripleO that enforces this workflow. It's possible for deployers to set KeystoneCredential0 and KeystoneCredential1 to keys of their own choosing, manually rotating the keys. This can result in the deployer unintentionally locking all credential data since the symmetric keys used to encrypt them are no longer available.

Comment 2 Lance Bragstad 2020-10-19 16:59:18 UTC
More information on credential encryption, including the rotation process, is documented upstream [0].

[0] https://docs.openstack.org/keystone/latest/admin/credential-encryption.html