Bug 1926717

Description of problem (please be detailed as possible and provide log
snippets):

When cluster wide encryption is enabled using KMS and the same backend path on the vault server is shared among two or more OCS clusters, only one NOOBAA_ROOT_SECRET_PATH key is created. 

Before OCS storagecluster creation, the following keys were present in the backend path, which were created as part of a previous OCS deployment.

$ vault kv list ocs
Keys
----
NOOBAA_ROOT_SECRET_PATH
rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0-8j44m
rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-1-b9qsg
rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-2-8k6l5

[fedora@vaulttest ~]$ vault kv get ocs/NOOBAA_ROOT_SECRET_PATH
======= Data =======
Key           Value
---           -----
rootkeyb64    uInK53xYPr/tNmUzNU4Ng4rDdJc2+WD8NETuHKHPEz8=

After creation of a new storagecluster, the keys for the OSDs of the new cluster were created, but no new key was created for Noobaa. It appears that the existing key might have been used for the new cluster too.

$ vault kv list ocs
Keys
----
NOOBAA_ROOT_SECRET_PATH
rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0-8j44m
rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-0s6jj7
rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-1-b9qsg
rook-ceph-osd-encryption-key-ocs-deviceset-thin-0-data-2-8k6l5
rook-ceph-osd-encryption-key-ocs-deviceset-thin-1-data-0x9njv
rook-ceph-osd-encryption-key-ocs-deviceset-thin-2-data-0tcb26

[fedora@vaulttest ~]$ vault kv get ocs/NOOBAA_ROOT_SECRET_PATH
======= Data =======
Key           Value
---           -----
rootkeyb64    uInK53xYPr/tNmUzNU4Ng4rDdJc2+WD8NETuHKHPEz8=

Version of all relevant components (if applicable):
OCP: 4.7.0-0.nightly-2021-02-09-003138
ocs-operator.v4.7.0-254.ci

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
No

Is there any workaround available to the best of your knowledge?
NA

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
No

Steps to Reproduce:
1. Create two OCS clusters with cluster wide encryption enabled using KMS, such that both the clusters use the same backend path
2. Check the backend path on the vault server to verify creation of keys for OSD and noobaa

Actual results:
New keys were present for the OSDs but there was only one NOOBAA_ROOT_SECRET_PATH key

Expected results:
There should be separate NOOBAA_ROOT_SECRET_PATH keys created for each OCS cluster