Bug 1997624

Description of problem (please be detailed as possible and provide log snippets):

In ODF 4.9, when vault namespace is set for a vault config, the creation of encrypted RBD PVC fails with the following error:

$ oc describe pvc rbd-1
Name:          rbd-1
Namespace:     test-with-ns
StorageClass:  pv-encryption-with-ns
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Used By:       <none>
Events:
  Type     Reason              Age                   From                                                                                                                Message
  ----     ------              ----                  ----                                                                                                                -------
  Normal   Provisioning        4m48s (x12 over 13m)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-84ccc64b48-24zzs_81a4112e-1a67-4bb7-9971-b0a3b0a5ed70  External provisioner is provisioning volume for claim "test-with-ns/rbd-1"
  Warning  ProvisioningFailed  4m47s (x12 over 13m)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-84ccc64b48-24zzs_81a4112e-1a67-4bb7-9971-b0a3b0a5ed70  failed to provision volume with StorageClass "pv-encryption-with-ns": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed connecting to Vault: Error making API request.

URL: GET https://vault.qe.rh-ocs.com:8200/v1/sys/mounts
Code: 403. Errors:

* 1 error occurred:
          * permission denied

$ oc get sc pv-encryption-with-ns -o yaml |grep encrypt
[...]
  name: pv-encryption-with-ns
  encrypted: "true"
  encryptionKMSID: vault


$ oc get cm csi-kms-connection-details -n openshift-storage -o yaml
apiVersion: v1
data:
  vault: '{"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"rbd-encryption","VAULT_CACERT":"ocs-kms-ca-secret-986vqu","VAULT_TLS_SERVER_NAME":"","VAULT_CLIENT_CERT":"ocs-kms-client-cert-ndz28","VAULT_CLIENT_KEY":"ocs-kms-client-key-63ey6f","VAULT_NAMESPACE":"rbd-encryption","VAULT_TOKEN_NAME":"","VAULT_CACERT_FILE":"fullchain.pem","VAULT_CLIENT_CERT_FILE":"cert.pem","VAULT_CLIENT_KEY_FILE":"privkey.pem"}'


This issue was not seen when VAULT_NAMESPACE was not set in the vault config. PV creation was successful. 

$ oc get pvc
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
rbd-1   Bound    pvc-37aa8ac5-8e9c-43bd-b604-facb027ab10e   2Gi        RWO            pv-encryption-without-ns   118m

$ oc get sc pv-encryption-without-ns -o yaml |grep encrypt
[...]
  name: pv-encryption-without-ns
  encrypted: "true"
  encryptionKMSID: 2-vault


  2-vault: '{"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"2-vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"vb-sys-v1","VAULT_CACERT":"ocs-kms-ca-secret-3mhm6q","VAULT_TLS_SERVER_NAME":"","VAULT_CLIENT_CERT":"ocs-kms-client-cert-lds4l","VAULT_CLIENT_KEY":"ocs-kms-client-key-1z3qv","VAULT_NAMESPACE":"","VAULT_TOKEN_NAME":"","VAULT_CACERT_FILE":"fullchain.pem","VAULT_CLIENT_CERT_FILE":"cert.pem","VAULT_CLIENT_KEY_FILE":"privkey.pem"}'


Version of all relevant components (if applicable):
===================================================
ocs-operator.v4.9.0-105.ci   
odf-operator.v4.9.0-105.ci 
OCP: 4.9.0-0.nightly-2021-08-25-010624


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
===================================================
Cannot create encrypted PVCs which uses Vault namespace feature


Is there any workaround available to the best of your knowledge?
================================================================
Yes, adding the VAULT_BACKEND parameter to the respective vault config in csi-kms-connection-details configmap results in the PVC getting bound

  vault: '{"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"rbd-encryption","VAULT_CACERT":"ocs-kms-ca-secret-986vqu","VAULT_TLS_SERVER_NAME":"","VAULT_CLIENT_CERT":"ocs-kms-client-cert-ndz28","VAULT_CLIENT_KEY":"ocs-kms-client-key-63ey6f","VAULT_NAMESPACE":"rbd-encryption","VAULT_TOKEN_NAME":"","VAULT_CACERT_FILE":"fullchain.pem","VAULT_CLIENT_CERT_FILE":"cert.pem","VAULT_CLIENT_KEY_FILE":"privkey.pem","VAULT_BACKEND":"kv"}'

$ oc get pvc
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS            AGE
rbd-1   Bound    pvc-8c3ae370-e3e1-41cd-9060-70bb7de990fc   2Gi        RWO            pv-encryption-with-ns   115m



Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
=====================================================================
Yes, until OCS 4.8 no extra parameters were required for PVC creation with vault namespaces. 

Steps to Reproduce:
===================
1. Create an encryption enabled SC with vault namespace set using the steps here : https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.8/html-single/managing_and_allocating_storage_resources/index#creating-a-storage-class-for-persistent-volume-encryption_rhocs 
2. Use the SC created above and create a PVC
3. Check the status of the PVC


Actual results:
===============
The PVC is stuck in pending state with the following error:

  Warning  ProvisioningFailed  4m47s (x12 over 13m)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-84ccc64b48-24zzs_81a4112e-1a67-4bb7-9971-b0a3b0a5ed70  failed to provision volume with StorageClass "pv-encryption-with-ns": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed connecting to Vault: Error making API request.

URL: GET https://vault.qe.rh-ocs.com:8200/v1/sys/mounts
Code: 403. Errors:

* 1 error occurred:
          * permission denied


Expected results:
=================
PVC should be bound

Additional info
===============
The issue is seen with both kv-v1 and kv-v2 enabled backend paths.