Description of problem (please be detailed as possible and provide log snippets): In OCS 4.8, when the cluster deployed with cluster-wide encryption using KMS and kv-v2 as the secret engine is uninstalled, the keys in vault created for the OSDs are not deleted. Before uninstall: ================= $ oc get pvc |grep deviceset ocs-deviceset-thin-0-data-0m27q8 Bound pvc-8422bf20-677b-43d3-905d-10468f87ac40 512Gi RWO thin 95m ocs-deviceset-thin-1-data-0sq227 Bound pvc-ac1ecb3f-cfb6-42f0-bc3f-5d1052af0987 512Gi RWO thin 95m ocs-deviceset-thin-2-data-0xzszb Bound pvc-755ea6d9-0fc8-4409-b210-a6a2962c0389 512Gi RWO thin 95m $ vault kv list kv-v2 Keys ---- NOOBAA_ROOT_SECRET_PATH/ rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8 rook-ceph-osd-encryption-key-ocs-deviceset-thin-1-data-0sq227 rook-ceph-osd-encryption-key-ocs-deviceset-thin-2-data-0xzszb After uninstall: ================ $ oc get pvc No resources found in openshift-storage namespace. $ oc get pv No resources found $ vault kv list kv-v2 Keys ---- NOOBAA_ROOT_SECRET_PATH/ rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8 rook-ceph-osd-encryption-key-ocs-deviceset-thin-1-data-0sq227 rook-ceph-osd-encryption-key-ocs-deviceset-thin-2-data-0xzszb Version of all relevant components (if applicable): OCP: 4.8.0-0.nightly-2021-06-19-005119 OCS: ocs-operator.v4.8.0-424.ci Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? No Is there any workaround available to the best of your knowledge? Yes, deleting the metadata from vault, after the uninstall deletes the keys. $ vault kv get kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8 ====== Metadata ====== Key Value --- ----- created_time 2021-06-23T10:06:30.650103555Z deletion_time 2021-06-23T11:46:35.045328495Z destroyed false version 1 $ vault kv metadata delete kv-v2/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8 Success! Data deleted (if it existed) at: kv-v2/metadata/rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0m27q8 $ vault kv list kv-v2 Keys ---- NOOBAA_ROOT_SECRET_PATH/ rook-ceph-osd-encryption-key-ocs-deviceset-thin-1-data-0sq227 rook-ceph-osd-encryption-key-ocs-deviceset-thin-2-data-0xzszb Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 2 Can this issue reproducible? Yes Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: No Steps to Reproduce: Follow the steps here: https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.8/html-single/deploying_openshift_container_storage_on_vmware_vsphere/index?lb_target=preview#assembly_uninstalling-openshift-container-storage_rhocs Actual results: The keys in vault are not deleted after uninstall Expected results: The keys should be deleted.
Requires doc text as a known issue
Rachael, can we get a mustgather or simply the rook-ceph-operator logs to understand a bit more what going on here? Thanks!
The bug is not only affecting kv v2 but also v1. The bug is not experienced when the PVC reclaim policy is Retain.
(In reply to Sébastien Han from comment #5) > The bug is not only affecting kv v2 but also v1. > The bug is not experienced when the PVC reclaim policy is Retain. **Disregard this comment**. I mixed things up, I'm still looking.
Ok after some local debugging there is no bug per se. What we see is the remaining metadata but if you inspect the Secret, you won't see any data, for instance: BEFORE DELETION: runner@fv-az72-598:~/work/rook/rook$ kubectl exec -ti vault-0 -- vault kv get -ca-cert /vault/userconfig/vault-server-tls/vault.crt rook/ver2/mybucketkey ====== Metadata ====== Key Value --- ----- created_time 2021-06-24T10:20:52.07073694Z deletion_time n/a destroyed false version 1 === Data === Key Value --- ----- key IdCRBZ+sAA7D87dqUO6F+Hn7MQ24SDhb4lnTqB7QNf4= AFTER DELETION: runner@fv-az72-598:~/work/rook/rook$ kubectl exec -ti vault-0 -- vault kv get -ca-cert /vault/userconfig/vault-server-tls/vault.crt rook/ver2/mybucketkey ====== Metadata ====== Key Value --- ----- created_time 2021-06-24T12:33:02.669280958Z deletion_time 2021-06-24T12:40:46.483734597Z destroyed false version 1 Do not get confused "version 1" is the versioned number of the key, the revision if you will. Updated the doc text too. I'm moving this to MODIFIED since I believe the doc text is sufficient.
Moving back to ASSIGNED for an eng fix, we also need a clone for the Doc I suppose?
Ok final info on this one, the key is not deleted, it's a soft delete that can be undone.
The docs are verified but I don't think it need to be in known issues but in some release notes. It's no so bad :) @olakra @muagarwa @
Agreed, that sounds good as a release note item. It's by-design so not really a known issue, depending on how you define "issue" though :)
Known issues are put into release notes, so I am little confused here. Do you want to include this in documentation or just release notes?
But we understand that it is not an issue but by design so isn't there any other place to put this information? Maybe add it as a note in uninstall instructions? @muagarwa
Duplicated this BZ for documentation and closing it as not a bug.