Bug 1979244
| Summary: | [KMS] Keys are still listed in vault after deleting encrypted PVCs while using kv-v2 secret engine | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Rachael <rgeorge> |
| Component: | csi-driver | Assignee: | Niels de Vos <ndevos> |
| Status: | CLOSED ERRATA | QA Contact: | Rachael <rgeorge> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.8 | CC: | edonnell, etamir, kbg, kramdoss, madam, muagarwa, nberry, ndevos, ocs-bugs, odf-bz-bot, ratamir |
| Target Milestone: | --- | ||
| Target Release: | ODF 4.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
.Keys are completely destroyed in Vault after deleting encrypted persistent volume claims (PVCs) while using the `kv-v2` secret engine
HashiCorp Vault added a feature for the key-value store v2 where deletion of the stored keys makes it possible to recover the contents in case the metadata of the deleted key is not removed in a separate step. When using key-value v2 storage for secrets in HashiCorp Vault, deletion of volumes did not remove the metadata of the encryption passphrase from the KMS.
With this update, the keys in HashiCorp Vault is completely destroyed by default when a PVC is deleted. You can set the new configuration option `VAULT_DESTROY_KEYS` to `false` to enable the previous behavior. In that case, the metadata of the keys will be kept in HashiCorp Vault so that recovery of the encryption passphrase of the removed PVC is possible.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-12-13 17:44:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1966894, 2011326 | ||
The metadata with v2 behaviour and same RCA here too https://bugzilla.redhat.com/show_bug.cgi?id=1975323#c7 We could get the doc text ready for the same. @Niels, adding you here so please feel free to add if you have any extra details Niels, Pls fill the doc text I wonder if this is a duplicate of bug 1941836? Being able to undelete/recover data from Hashicorp Vault is a feature from the service. We may need to check with Hashicorp if there is an option to automatically purge metadata of deleted keys after a period of time. I was not able to find something like that in the documentation at https://www.vaultproject.io/docs/secrets/kv/kv-v2#deleting-and-destroying-data As Hashicorp Vault added this feature, their customers may want to use it, or even rely on it. Working around the feature to prevent recovering of deleted data might not be acceptable for all customers. We would need to know what customers of Hashicorp Vault expect, and what configuration options we need to add to meet those expectations. Niels, please fill the doc text FWIW, in Ceph-CSI upstream we'll make this behaviour configurable, and the current proposal is to destroy the contents and metadata of the keys when they are deleted. Is this something you would like to see included in an upcoming release of OCS/ODF? We will need Eran's input on this Doc text needs to be changed, earlier it was a known issue and now a Bug Fix. Olive, the doc-text has been updated to match the current bug fix state. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.9.0 enhancement, security, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:5086 |
Description of problem (please be detailed as possible and provide log snippets): In OCS 4.8, when KV-v2 secret engine is used to store the keys for encrypted RBD PVC, on deleting the PVC, the keys are still listed in Vault. As seen from the output below the data part of the key is deleted, however the metadata is still present and needs to be deleted separately for the entire key to be removed. $ oc get cm csi-kms-connection-details -n openshift-storage -o yaml apiVersion: v1 data: 1-vault: '{"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault-cluster.vault.aws.hashicorp.cloud:8200","VAULT_BACKEND_PATH":"rbd","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"admin","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":"","VAULT_SKIP_VERIFY":"true"}' $ vault secrets list -detailed|grep rbd rbd/ kv kv_245f4a86 system system false replicated false false map[version:2] n/a $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-rbd-block-1 Bound pvc-3cefaee7-667f-4b55-af9a-f7b7fb4cd0b0 5Gi RWX test-pv-encryption 14m $ oc describe pv pvc-3cefaee7-667f-4b55-af9a-f7b7fb4cd0b0 |grep VolumeHandle VolumeHandle: 0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e Before deleting the PVC: ======================== $ vault kv get rbd/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e ====== Metadata ====== Key Value --- ----- created_time 2021-07-05T11:05:48.648644411Z deletion_time n/a destroyed false version 1 ==== Data ==== Key Value --- ----- data map[passphrase:Sf89f9DXEdc7Oh8Nf3av-5f56lo=] After deleting the PVC: ======================= $ oc get pvc No resources found in test namespace. $ oc describe pv pvc-3cefaee7-667f-4b55-af9a-f7b7fb4cd0b0 Error from server (NotFound): persistentvolumes "pvc-3cefaee7-667f-4b55-af9a-f7b7fb4cd0b0" not found $ vault kv get rbd/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e ====== Metadata ====== Key Value --- ----- created_time 2021-07-05T11:05:48.648644411Z deletion_time 2021-07-05T11:24:10.920777426Z destroyed false version 1 Version of all relevant components (if applicable): OCP: 4.8.0-0.nightly-2021-06-25-182927 OCS: ocs-operator.v4.8.0-443.ci Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? No Is there any workaround available to the best of your knowledge? Yes, deleting the metadata from vault, after the PV deletion deletes the keys. $ vault kv metadata delete rbd/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e Success! Data deleted (if it existed) at: rbd/metadata/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e $ vault kv get rbd/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e No value found at rbd/data/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 2 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: No Steps to Reproduce: 1. Create an encryption enabled RBD storageclass with kv-v2 2. Create a PVC using the SC above and verify if the key is available in vault 3. Delete the PVC 4. Check if the key is still listed in the backend path in Vault Actual results: The keys in vault are not deleted Expected results: The keys should be deleted. Additional info: Similar issue is seen for cluster wide encryption as well: https://bugzilla.redhat.com/show_bug.cgi?id=1975323