Created attachment 1956192 [details]
Advance Setting

Description of problem (please be detailed as possible and provide log
snippests):

When setting PV-encryption, the PVC provisioning process fails to connect to the VAULT HCP instance and reports a "permission denied" error.

Version of all relevant components (if applicable): 4.13


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)? 
Cant Create PVC when pv-encryption is enabled with vault kubernetes auth method.


Is there any workaround available to the best of your knowledge?
Yes, Manually we can edit 'VaultAuthNamespace' value from 'csi-kms-connection-details' configmap and change value to the HCP vault namespace. 


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)? 1


Can this issue reproducible? 
Yes


Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Setup service accounts, auth method for the intended namespace by following steps mention in this link: https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.12/html/managing_and_allocating_storage_resources/storage-classes_rhodf#configuring-access-to-kms-using-vaulttenantsa_rhodf
2. [UI] Go to "Storage > StorageClasses".
3. [UI] Create StorageClass using "rbd" provisioner.
4. [UI] Opt for "Kubernetes" authentication method.
5. [UI] Create PVC on configured namespace on step 1.

Actual results:
PVC provisioning is failing.

Expected results:
PVC provisioning should be succeeded. 

Additional info:

Initially problem is reported via BZ https://bugzilla.redhat.com/show_bug.cgi?id=2181446 and UI team has fixed their part.

Now the problem we're facing is related to the "VaultAuthNamespace" value in the "csi-kms-connection-details" configmap.
When we enter the tenant namespace value in the "Authentication Namespace" field in the UI's advanced settings form, it is mapped to the "VaultAuthNamespace" field in the configmap after successfully adding the connection. However, when we try to provision the PVC, we receive a "Permission Denied" error. 
We have found that changing the value of "VaultAuthNamespace" to match the value of "VaultNamespace" results in successful PVC provisioning.


csi-kms-connection-details configmap
====================================
oc describe  configmap csi-kms-connection-details -n openshift-storage                                                  ─╯
Name:         csi-kms-connection-details
Namespace:    openshift-storage
Labels:       <none>
Annotations:  <none>

Data
====
encrypt-conn:
----
{"encryptionKMSType":"vaulttenantsa","kmsServiceName":"encrypt-conn","vaultAddress":"https://vault-cluster.vault.2467e33a-73f9-408b-b9ff-b0476a654d30.aws.hashicorp.cloud:8200","vaultBackendPath":"odf/","vaultTLSServerName":"","vaultCAFileName":"","vaultClientCertFileName":"","vaultClientCertKeyFileName":"","vaultAuthMethod":"kubernetes","vaultAuthPath":"/v1/auth/kubernetes/login","vaultAuthNamespace":"default","vaultNamespace":"admin"}

BinaryData
====

Events:  <none>


PVC Provisioning ERROR
======================
❯ oc describe pvc my-pvc -n default
.
.
.
  Warning  ProvisioningFailed  2m32s (x10 over 6m50s)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5c979447fd-8hdlj_0a4af886-9e74-40e6-b564-7a25cfc86728  failed to provision volume with StorageClass "encrypt-sc": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed connecting to Vault: failed to get the authentication token: Error making API request.

Namespace: default
URL: PUT https://vault-cluster.vault.2467e33a-73f9-408b-b9ff-b0476a654d30.aws.hashicorp.cloud:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied
  Normal  ExternalProvisioning  65s (x26 over 6m50s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "openshift-storage.rbd.csi.ceph.com" or manually created by system administrator