Bug 2061713

Description of problem (please be detailed as possible and provide log
snippets):

The creation of encrypted RBD PVCs using kubernetes authentication method (using serviceaccount) fails with the following error:

$ oc describe pvc rbd-2
Name:          rbd-2
Namespace:     test-4
StorageClass:  test-pv-encryption-4
Status:        Pending
[...]

  Warning  ProvisioningFailed    75s (x10 over 5m31s)  openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-d98d9b847-vkhsn_031127f8-bf07-4c80-8a66-4e572b015807  failed to provision volume with StorageClass "test-pv-encryption-4": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed to set VAULT_AUTH_MOUNT_PATH in Vault config: path is empty

This error was seen even though the kubernetes authentication method was enabled at the default kubernetes/ path in Vault. 

$ oc get sc test-pv-encryption-4 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2022-03-08T11:07:13Z"
  name: test-pv-encryption-4
[...]
  encrypted: "true"
  encryptionKMSID: vault-sa-internal


$ oc get cm csi-kms-connection-details -n openshift-storage -o yaml
apiVersion: v1
data:
[...]
  vault-sa-internal: '{"encryptionKMSType":"vaulttenantsa","kmsServiceName":"vault-sa-internal","vaultAddress":"https://vault.default.svc.cluster.local:8200", "vaultBackendPath":"odf", "vaultTLSServerName":"","vaultNamespace":"","vaultCAFromSecret":"ocs-kms-ca-secret-4fjlyb","vaultCAFileName":"","vaultClientCertFileName":"", "vaultClientCertKeyFileName":"","vaultAuthMethod":"kubernetes","vaultAuthPath":"","vaultAuthNamespace":""}'


Commands run to configure the kubernetes authentication method in Vault:

    $ oc -n default exec -it vault-0 -- vault write -ca-cert /vault/userconfig/vault-server-tls/vault.crt auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"

    $ oc -n default exec -it vault-0 -- vault write -ca-cert /vault/userconfig/vault-server-tls/vault.crt auth/kubernetes/role/csi-kubernetes bound_service_account_names=ceph-csi-vault-sa bound_service_account_namespaces=test-4 



Version of all relevant components (if applicable):
---------------------------------------------------
OCP: 4.10.0-0.nightly-2022-03-08-002944
ODF: odf-operator.v4.10.0    full_version=4.10.0-179


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, creation of encrypted PVC fails


Is there any workaround available to the best of your knowledge?

Setting the vaultAuthPath in the csi-kms-connection-details configmap:

  vault-sa-internal: '{"encryptionKMSType":"vaulttenantsa","kmsServiceName":"vault-sa-internal","vaultAddress":"https://vault.default.svc.cluster.local:8200", "vaultBackendPath":"odf", "vaultTLSServerName":"","vaultNamespace":"","vaultCAFromSecret":"ocs-kms-ca-secret-4fjlyb","vaultCAFileName":"","vaultClientCertFileName":"", "vaultClientCertKeyFileName":"","vaultAuthMethod":"kubernetes","vaultAuthPath":"/v1/auth/kubernetes/login","vaultAuthNamespace":""}'



Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2


Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
N/A

If this is a regression, please provide more details to justify this:
Yes. For encrypted PVCs if the kubernetes authentication method was enabled at the default kubernetes/ path in Vault, then the vaultAuthPath didn't have to be explicitly specified in the KMS connection details.


Steps to Reproduce:
-------------------
1. Follow the steps to create an encrypted RBD PVC using vaulttenantsa encryptionKMSType as mentioned here: https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.9/html/managing_and_allocating_storage_resources/storage-classes_rhodf#prerequisites_for_using_literal_vaulttenantsa_literal

2. Check the status of the PVC


Actual results:
---------------

PVC is in pending state with the following error:

failed to provision volume with StorageClass "test-pv-encryption-4": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed to set VAULT_AUTH_MOUNT_PATH in Vault config: path is empty


Expected results:
-----------------
PVC should be in Bound state. Since the kubernetes authentication method is enabled in the default path, there should be no need to specify the authentication path.


Additional info:
----------------
The VAULT_AUTH_MOUNT_PATH was recently introduced in the UI for clusterwide encryption with this bug fix: https://bugzilla.redhat.com/show_bug.cgi?id=2048442