Bug 2052937

Description of problem (please be detailed as possible and provide log
snippets):

When cluster wide encryption is enabled with token auth and uses Vault namespace, the auto detection of KV version fails, resulting in the deployment to fail. The following error is seen in the rook operator logs:

2022-02-10 09:40:10.205273 E | ceph-cluster-controller: failed to reconcile CephCluster "openshift-storage/ocs-storagecluster-cephcluster". failed to reconcile cluster "ocs-storagecluster-cephcluster": failed to configure local ceph cluster: failed to perform validation before cluster creation: failed to validate kms connection details: failed to get backend version: failed to list vault system mounts: Error making API request.

URL: GET https://vault.qe.rh-ocs.com:8200/v1/sys/mounts
Code: 403. Errors:

* 1 error occurred:
	* permission denied


The curl command when run locally works:

$ curl -X GET 'https://vault.qe.rh-ocs.com:8200/v1/sys/mounts' -H 'X-Vault-Token: <token>' -H 'X-Vault-Namespace: rbd-encryption'
{"request_id":"04d3ab03-b5a1-bc5e-34fb-47e309ae5997","lease_id":"","renewable":false,"lease_duration":0,"data":{"rbd-encryption/":{"accessor":"kv_bd6075d4","config":{"default_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0},"description":"","external_entropy_access":false,"local":false,"options":{"version":"1"},"seal_wrap":false,"type":"kv","uuid":"b20672c3-1383-be25-7cae-df8d9fb45962"}},"wrap_info":null,"warnings":null,"auth":null}


$ oc get cm ocs-kms-connection-details -n openshift-storage -o yaml
apiVersion: v1
data:
  KMS_PROVIDER: vault
  KMS_SERVICE_NAME: vault
  VAULT_ADDR: https://vault.qe.rh-ocs.com:8200
  VAULT_AUTH_METHOD: token
  VAULT_BACKEND_PATH: rbd-encryption
  VAULT_CACERT: ocs-kms-ca-secret-rwt7q
  VAULT_CLIENT_CERT: ocs-kms-client-cert-x64jdk
  VAULT_CLIENT_KEY: ocs-kms-client-key-0xq8rv
  VAULT_NAMESPACE: rbd-encryption
  VAULT_TLS_SERVER_NAME: ""


Version of all relevant components (if applicable):
---------------------------------------------------
OCP: 4.10.0-0.nightly-2022-02-09-111355
ODF: odf-operator.v4.10.0   full_version=4.10.0-151


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, the deployment fails


Is there any workaround available to the best of your knowledge?
Setting the VAULT_BACKEND value in ocs-kms-connection-details resolves the issue.

$ oc get cm ocs-kms-connection-details -o yaml -n openshift-storage
apiVersion: v1
data:
  KMS_PROVIDER: vault
  KMS_SERVICE_NAME: vault
  VAULT_ADDR: https://vault.qe.rh-ocs.com:8200
  VAULT_AUTH_METHOD: token
  VAULT_BACKEND: v1
  VAULT_BACKEND_PATH: rbd-encryption
  VAULT_CACERT: ocs-kms-ca-secret-rwt7q
  VAULT_CLIENT_CERT: ocs-kms-client-cert-x64jdk
  VAULT_CLIENT_KEY: ocs-kms-client-key-0xq8rv
  VAULT_NAMESPACE: rbd-encryption
  VAULT_TLS_SERVER_NAME: ""
kind: ConfigMap


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
3

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
No

Steps to Reproduce:
1. Deploy ODF cluster with clusterwide encryption enabled with token auth, using vault namespaces

2. Check the rook operator logs


Actual results:
---------------
The deployment fails with the following error:


Expected results:
-----------------
The deployment should be successful