Description of problem (please be detailed as possible and provide log snippets): When cluster wide encryption is enabled using KMS and the same backend path on the vault server is shared among two or more OCS clusters, only one NOOBAA_ROOT_SECRET_PATH key is created. Before OCS storagecluster creation, the following keys were present in the backend path, which were created as part of a previous OCS deployment. $ vault kv list ocs Keys ---- NOOBAA_ROOT_SECRET_PATH rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0-8j44m rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-1-b9qsg rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-2-8k6l5 [fedora@vaulttest ~]$ vault kv get ocs/NOOBAA_ROOT_SECRET_PATH ======= Data ======= Key Value --- ----- rootkeyb64 uInK53xYPr/tNmUzNU4Ng4rDdJc2+WD8NETuHKHPEz8= After creation of a new storagecluster, the keys for the OSDs of the new cluster were created, but no new key was created for Noobaa. It appears that the existing key might have been used for the new cluster too. $ vault kv list ocs Keys ---- NOOBAA_ROOT_SECRET_PATH rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0-8j44m rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0s6jj7 rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-1-b9qsg rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-2-8k6l5 rook-ceph-osd-encryption-key-ocs-deviceset-thin-1-data-0x9njv rook-ceph-osd-encryption-key-ocs-deviceset-thin-2-data-0tcb26 [fedora@vaulttest ~]$ vault kv get ocs/NOOBAA_ROOT_SECRET_PATH ======= Data ======= Key Value --- ----- rootkeyb64 uInK53xYPr/tNmUzNU4Ng4rDdJc2+WD8NETuHKHPEz8= Version of all relevant components (if applicable): OCP: 4.7.0-0.nightly-2021-02-09-003138 ocs-operator.v4.7.0-254.ci Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? No Is there any workaround available to the best of your knowledge? NA Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 2 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: No Steps to Reproduce: 1. Create two OCS clusters with cluster wide encryption enabled using KMS, such that both the clusters use the same backend path 2. Check the backend path on the vault server to verify creation of keys for OSD and noobaa Actual results: New keys were present for the OSDs but there was only one NOOBAA_ROOT_SECRET_PATH key Expected results: There should be separate NOOBAA_ROOT_SECRET_PATH keys created for each OCS cluster
Logs available here: http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/1926717/
Tested on ocs-operator.v4.7.0-263.ci Before installing ocs [fedora@vaulttest ~]$ vault kv list ocs/NOOBAA_ROOT_SECRET_PATH Keys ---- rootkeyb64-663ee31a-4a95-4842-b29b-dde4ff422b8c rootkeyb64-af9c0701-308c-4354-a81f-f6df59c3acde [fedora@vaulttest ~]$ After installing ocs [fedora@vaulttest ~]$ vault kv list ocs/NOOBAA_ROOT_SECRET_PATH Keys ---- rootkeyb64-663ee31a-4a95-4842-b29b-dde4ff422b8c rootkeyb64-7bca5296-4210-47ce-bc70-e340e98987f4 rootkeyb64-af9c0701-308c-4354-a81f-f6df59c3acde rootkeyb64-b6bdfd1d-d8ab-4b65-83cb-1563916639b9 I see 2 new keys under NOOBAA_ROOT_SECRET_PATH after creating 2 new clusters. Moving the bug to Verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Container Storage 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2041