Description of problem (please be detailed as possible and provide log snippets): When VAULT_BACKEND is set to "kv" in the vault config where the backend path is using kv-v2, PV encryption fails. The PVC creation succeeds, however when it is attached to a pod and it tries to create the key in Vault, it fails with the following error: Warning FailedMapVolume 8s (x5 over 18s) kubelet MapVolume.SetUpDevice failed for volume "pvc-56732d4c-6203-4f9b-b2a8-0728a8f00135" : rpc error: code = Internal desc = failed to encrypt rbd image ocs-storagecluster-cephblockpool/csi-vol-8986b89d-f8f9-11eb-972f-0a580a810211: failed parsing data for get passphrase request for 0001-0011-openshift-storage-0000000000000001-8986b89d-f8f9-11eb-972f-0a580a810211 This could be because the structure of how keys are stored in kv as compared to kv-v2 is different and when use of kv APIs is enforced on a backend path using kv-v2, it fails. The error message does not clearly reflect the cause of the failure and can be improved. Version of all relevant components (if applicable): OCS: ocs-operator.v4.7.3-243.ci Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? No, but it is a trivial failure which can be avoided or corrected with proper config values Is there any workaround available to the best of your knowledge? Change VAULT_BACKEND to the correct value as that of the backend path Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 2 Can this issue reproducible? Yes Can this issue reproduce from the UI? No If this is a regression, please provide more details to justify this: No Steps to Reproduce: 1. Create a storage class with encryption enabled using a backend path with kv-v2 (https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.7/html-single/managing_and_allocating_storage_resources/index#creating-a-storage-class-for-persistent-volume-encryption_rhocs) 2. Find out the encryptionKMSID being used for the storageclass $ oc get sc <sc_name> -o yaml |grep encryptionKMSID encryptionKMSID=4-vault 3. Edit the vault config for the KMSID from the step above in the csi-kms-connection-details configmap $ oc edit cm csi-kms-connection-details -n openshift-storage 4. Set the VAULT_BACKEND parameter to "kv" as shown below 4-vault: '{"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"bug","VAULT_CACERT":"ocs-kms-ca-secret- 35i1ma","VAULT_TLS_SERVER_NAME":"","VAULT_CLIENT_CERT":"ocs-kms-client-cert-ja7ani","VAULT_CLIENT_KEY":"ocs-kms-client-key-jrtckg","VAULT_NAMESPACE":"","VAULT_TOKEN_NAME":"ocs-kms- token","VAULT_CACERT_FILE":"fullchain.pem","VAULT_CLIENT_CERT_FILE":"cert.pem","VAULT_CLIENT_KEY_FILE":"privkey.pem","VAULT_BACKEND":"kv"}' 5. Save the configmap 6. Create a PVC using the SC and attach it to a pod 7. Check the status of the PVC and Pod Actual results: The PV creation was successful, however the pod creation failed with the error given below and no key was created in Vault. Warning FailedMapVolume 8s (x5 over 18s) kubelet MapVolume.SetUpDevice failed for volume "pvc-56732d4c-6203-4f9b-b2a8-0728a8f00135" : rpc error: code = Internal desc = failed to encrypt rbd image ocs-storagecluster-cephblockpool/csi-vol-8986b89d-f8f9-11eb-972f-0a580a810211: failed parsing data for get passphrase request for 0001-0011-openshift-storage-0000000000000001-8986b89d-f8f9-11eb-972f-0a580a810211 Expected results: Either the error message should be improved to reflect the actual cause of the failure or we can prevent VAULT_BACKEND value to override the auto-detection value in situations like this, if possible Additional info: - When the values are switched, i.e., set VAULT_BACKEND to "kv-v2" for a backend path using kv, the PVC and pod creation succeeds. However the key in vault is created under the subpath /data inside the backend path 3-vault: '{"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"bug1","VAULT_CACERT":"ocs-kms-ca-secret o1icyg","VAULT_TLS_SERVER_NAME":"","VAULT_CLIENT_CERT":"ocs-kms-client-cert-h0z11d","VAULT_CLIENT_KEY":"ocs-kms-client-key-hgy4x","VAULT_NAMESPACE":"","VAULT_TOKEN_NAME":"ocs-kms- token","VAULT_CACERT_FILE":"fullchain.pem","VAULT_CLIENT_CERT_FILE":"cert.pem","VAULT_CLIENT_KEY_FILE":"privkey.pem","VAULT_BACKEND":"kv-v2"}' $ vault kv list bug1/data Keys ---- 0001-0011-openshift-storage-0000000000000001-dd67e191-f8de-11eb-972f-0a580a810211 0001-0011-openshift-storage-0000000000000001-f638f218-f8f9-11eb-972f-0a580a810211 - The same error should be observed in OCS 4.8 for PVC creation, since the key is generated in vault during PV creation in 4.8