Description of problem (please be detailed as possible and provide log
snippets):

In OCS 4.8, when KV-v2 secret engine is used to store the keys for encrypted RBD PVC, on deleting the PVC, the keys are still listed in Vault. As seen from the output below the data part of the key is deleted, however the metadata is still present and needs to be deleted separately for the entire key to be removed.

$ oc get cm csi-kms-connection-details -n openshift-storage -o yaml
apiVersion: v1
data:
  1-vault: '{"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault-cluster.vault.aws.hashicorp.cloud:8200","VAULT_BACKEND_PATH":"rbd","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"admin","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":"","VAULT_SKIP_VERIFY":"true"}'


$ vault secrets list -detailed|grep rbd
rbd/   kv   kv_245f4a86   system   system   false   replicated   false   false    map[version:2]    n/a 


$ oc get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
pvc-rbd-block-1       Bound    pvc-3cefaee7-667f-4b55-af9a-f7b7fb4cd0b0   5Gi        RWX            test-pv-encryption   14m


$ oc describe pv pvc-3cefaee7-667f-4b55-af9a-f7b7fb4cd0b0 |grep VolumeHandle
    VolumeHandle:      0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e


Before deleting the PVC:
========================

$ vault kv get rbd/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e
====== Metadata ======
Key              Value
---              -----
created_time     2021-07-05T11:05:48.648644411Z
deletion_time    n/a
destroyed        false
version          1

==== Data ====
Key     Value
---     -----
data    map[passphrase:Sf89f9DXEdc7Oh8Nf3av-5f56lo=]


After deleting the PVC:
=======================

$ oc get pvc
No resources found in test namespace.

$ oc describe pv pvc-3cefaee7-667f-4b55-af9a-f7b7fb4cd0b0 
Error from server (NotFound): persistentvolumes "pvc-3cefaee7-667f-4b55-af9a-f7b7fb4cd0b0" not found

$ vault kv get rbd/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e
====== Metadata ======
Key              Value
---              -----
created_time     2021-07-05T11:05:48.648644411Z
deletion_time    2021-07-05T11:24:10.920777426Z
destroyed        false
version          1

Version of all relevant components (if applicable):
OCP: 4.8.0-0.nightly-2021-06-25-182927
OCS: ocs-operator.v4.8.0-443.ci


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
No

Is there any workaround available to the best of your knowledge?
Yes, deleting the metadata from vault, after the PV deletion deletes the keys.

$ vault kv metadata delete rbd/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e
Success! Data deleted (if it existed) at: rbd/metadata/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e


$ vault kv get rbd/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e
No value found at rbd/data/0001-0011-openshift-storage-0000000000000001-f411f3be-dd80-11eb-abbf-0a580a80026e


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
No

Steps to Reproduce:
1. Create an encryption enabled RBD storageclass with kv-v2
2. Create a PVC using the SC above and verify if the key is available in vault
3. Delete the PVC
4. Check if the key is still listed in the backend path in Vault

Actual results:
The keys in vault are not deleted 

Expected results:
The keys should be deleted.

Additional info:
Similar issue is seen for cluster wide encryption as well: https://bugzilla.redhat.com/show_bug.cgi?id=1975323