Description of problem (please be detailed as possible and provide log
snippets):

In OCP 4.11 + ODF 4.11, the creation of encrypted PVC using the vaulttenantsa method is failing with the following error:

  Warning  ProvisioningFailed    7s (x8 over 71s)   openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5c7ffd97-nlfdd_2d4c77af-4dad-4b7e-9bf1-92e5d11a8d3c  failed to provision volume with StorageClass "test-pv-encryption-1": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed setting up token for test/ceph-csi-vault-sa: failed to find token in ServiceAccount test/ceph-csi-vault-sa


Due to recent changes in kubernetes, that is available in OCP 4.11, the serviceaccount does not have the token secret linked to it during its creation anymore.

$ oc get sa ceph-csi-vault-sa -o yaml
apiVersion: v1
imagePullSecrets:
- name: ceph-csi-vault-sa-dockercfg-psmtt
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-06-08T12:44:51Z"
  name: ceph-csi-vault-sa
  namespace: test
  resourceVersion: "165333"
  uid: 2d96aa2d-1c9f-4bd6-a762-6c65cbf16ea1
secrets:
- name: ceph-csi-vault-sa-dockercfg-psmtt


Version of all relevant components (if applicable):
---------------------------------------------------
OCP: 4.11.0-0.nightly-2022-06-06-025509
ODF: odf-operator.v4.11.0              OpenShift Data Foundation     4.11.0               Succeeded   full_version=4.11.0-89


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Yes, using the vaulttenantsa method for PV encryption fails



Is there any workaround available to the best of your knowledge?

The secret can be linked to the tenant serviceaccount using the following command:

$ oc get secret|grep vault
ceph-csi-vault-sa-dockercfg-psmtt   kubernetes.io/dockercfg               1      2m14s
ceph-csi-vault-sa-token-2z9tp       kubernetes.io/service-account-token   4      2m14s

$ oc secrets link ceph-csi-vault-sa ceph-csi-vault-sa-token-2z9tp

$ oc get sa ceph-csi-vault-sa -o yaml
apiVersion: v1
imagePullSecrets:
- name: ceph-csi-vault-sa-dockercfg-psmtt
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-06-08T12:44:51Z"
  name: ceph-csi-vault-sa
  namespace: test
  resourceVersion: "165333"
  uid: 2d96aa2d-1c9f-4bd6-a762-6c65cbf16ea1
secrets:
- name: ceph-csi-vault-sa-dockercfg-psmtt
- name: ceph-csi-vault-sa-token-2z9tp

Once the secret is linked, the PVC creation succeeds.


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2


Can this issue reproducible?
Yes


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:
Yes


Steps to Reproduce:
-------------------

1. Follow the steps here to create an encrypted PVC using vaulttenantsa method:

https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.10/html/managing_and_allocating_storage_resources/storage-classes_rhodf#prerequisites_for_using_literal_vaulttenantsa_literal



Actual results:
---------------

The PVC creation fails with the error:

  Warning  ProvisioningFailed    7s (x8 over 71s)   openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5c7ffd97-nlfdd_2d4c77af-4dad-4b7e-9bf1-92e5d11a8d3c  failed to provision volume with StorageClass "test-pv-encryption-1": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: failed setting up token for test/ceph-csi-vault-sa: failed to find token in ServiceAccount test/ceph-csi-vault-sa


Expected results:
-----------------
PVC creation should be successful.