Description of problem (please be detailed as possible and provide log
snippets):

When a tenant configmap is used to override the vault namespace and backend path set in the connection details in csi-kms-connection-details configmap, the creation of PVC fails with the following error:

I0203 05:52:19.429764       1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-bbfe93c7-eb1e-4bd2-9c3b-17cfc1449005","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-bbfe93c7-eb1e-4bd2-9c3b-17cfc1449005","csi.storage.k8s.io/pvc/name":"rbd-2","csi.storage.k8s.io/pvc/namespace":"test","encrypted":"true","encryptionKMSID":"vault-ns-1","imageFeatures":"layering","imageFormat":"2","pool":"ocs-storagecluster-cephblockpool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":1}}]}
I0203 05:52:19.521234       1 connection.go:186] GRPC response: {}
I0203 05:52:19.521275       1 connection.go:187] GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed connecting to Vault: Error making API request.

URL: GET https://vault.qe.rh-ocs.com:8200/v1/sys/mounts
Code: 403. Errors:

* 1 error occurred:
	* permission denied



Version of all relevant components (if applicable):
---------------------------------------------------
OCP: 4.10.0-0.nightly-2022-02-02-220834
ODF: 4.9.2-11

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, user cannot override namespaces using tenant configmap

Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
3


Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
No


If this is a regression, please provide more details to justify this:
No


Steps to Reproduce:
1. Create an encryption enabled storageclass using vault namespaces as shown in the config below:

vault-ns-1: '{"encryptionKMSType":"vaulttokens","kmsServiceName":"vault-ns-1","vaultAddress":"https://vault.qe.rh-ocs.com:8200","vaultBackendPath":"rook","vaultCAFromSecret":"ocs-kms-ca-secret-reqlva","vaultTLSServerName":"","vaultClientCertFromSecret":"ocs-kms-client-cert-gktgp","vaultClientCertKeyFromSecret":"ocs-kms-client-key-sct2im","vaultNamespace":"odf","vaultCAFileName":"fullchain.pem","vaultClientCertFileName":"cert.pem","vaultClientCertKeyFileName":"privkey.pem","vaultAuthMethod":"token","tenantTokenName":"ceph-csi-kms-token"}'

2. Create a new namespace in OCP and create the ceph-csi-kms-token secret, which provides access to the namespace and backend path mentioned in the config above

3. Create a PVC in the namespace. Verify that the PVC is bound and the key is created in the specified path in Vault.

$ oc get pvc
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-1   Bound    pvc-11be9c9a-5c77-4d0a-983a-6276b73e0a9f   10Gi       RWO            test-1         29m

$ oc get pv pvc-11be9c9a-5c77-4d0a-983a-6276b73e0a9f -o yaml|grep volumeHandle
    volumeHandle: 0001-0011-openshift-storage-0000000000000001-915883e1-84b4-11ec-a732-0a580a830017

$ vault kv list -namespace=odf rook
Keys
----
0001-0011-openshift-storage-0000000000000001-915883e1-84b4-11ec-a732-0a580a830017

4. Create a configmap in the tenant namespace to override the vault namespace as shown below:

$ oc get cm ceph-csi-kms-config -n test -o yaml
  apiVersion: v1
  data:
    vaultBackendPath: rbd-encryption
    vaultNamespace: rbd-encryption
  kind: ConfigMap
  metadata:
    creationTimestamp: "2022-02-03T05:50:56Z"
    name: ceph-csi-kms-config
    namespace: test
    resourceVersion: "45991"
    uid: fade94c8-b4cd-4da6-9da7-442087cdcdb1

5. Update the ceph-csi-kms-token secret in the tenant namespace to provide access to the new namespace and backend path.

6. Create a new PVC



Actual results:
---------------
PVC creation fails with the following error:

$ oc get pvc -w
NAME    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-1   Bound     pvc-11be9c9a-5c77-4d0a-983a-6276b73e0a9f   10Gi       RWO            test-1         20m
rbd-2   Pending                                                                        test-1         14m


  Warning  ProvisioningFailed  23m (x11 over 31m)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5c78b86dbb-xsx9w_4aa13338-4914-4d59-b0cb-58520e81c89b  failed to provision volume with StorageClass "test-1": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed connecting to Vault: Error making API request.

URL: GET https://vault.qe.rh-ocs.com:8200/v1/sys/mounts
Code: 403. Errors:

* 1 error occurred:
           * permission denied


Expected results:
-----------------

PVC creation should be successful and the encryption keys should be stored in the vault namespace specified in the tenant configmap.


Additional info:
----------------

When the ceph-csi-kms-token secret was changed back to the original token which provided access to the namespace set in the csi-kms-connection-details configmap, the provisioning succeeded and the encryption keys were created in the first vault namespace and not in the namespace set in the tenant configmap.

$ oc get pvc 
NAME    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-1   Bound     pvc-11be9c9a-5c77-4d0a-983a-6276b73e0a9f   10Gi       RWO            test-1         41m
rbd-2   Bound     pvc-bbfe93c7-eb1e-4bd2-9c3b-17cfc1449005   10Gi       RWO            test-1         35m
rbd-3   Bound     pvc-4db8c27a-73b0-4545-bd4e-c62c66ffb23b   10Gi       RWO            test-1         17m

$ vault kv list -namespace=odf rook
Keys
----
0001-0011-openshift-storage-0000000000000001-0a138101-84b8-11ec-a732-0a580a830017
0001-0011-openshift-storage-0000000000000001-0f23e7ac-84b8-11ec-a732-0a580a830017
0001-0011-openshift-storage-0000000000000001-915883e1-84b4-11ec-a732-0a580a830017