Created attachment 1813321 [details]
Screen recording of storageclass creation

Description of problem (please be detailed as possible and provide log
snippets):
During Storage class creation using RBD provisioner, the vault config or connection details can be modified in the storageclass creation page for encryption enabled SCs. 

However, when there is a custom KMS ID, for eg: "test-vault" instead of "1-vault", "2-vault" and so on, changing the vault connection details in the SC creation page does not get reflected in the UI and the newly created SC uses the old encryptionKMSID instead of the new one. 

The csi-kms-connection-details configmap gets updated, but the SC does not use the newly updated KMSID.

Version of all relevant components (if applicable):
OCP: 4.8.0-0.nightly-2021-08-09-135211
OCS: ocs-operator.v4.8.0

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, since the newly created SC does not use the updated vault config

Is there any workaround available to the best of your knowledge?
Not that I am aware of. Editing the parameters of an SC is not allowed afaik.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
No

Steps to Reproduce:
1. Create an encryption enabled RBD storageclass and provide the vault config.
2. Under workloads -> configmap in the openshift-storage namespace, click on csi-kms-connection-details and go to the YAML tab. The output should look similar to this:

kind: ConfigMap
apiVersion: v1
metadata:
  name: csi-kms-connection-details
  namespace: openshift-storage
[...]
data:
  1-vault: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"rbd-encryption","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"ocs/rbd","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}

3. Create another encryption enabled RBD SC and change the vault config in the SC creation page by clicking on "Change connection details" in the Connection details section

4. The csi-kms-connection-details configmap should now look like:

kind: ConfigMap
apiVersion: v1
metadata:
  name: csi-kms-connection-details
  namespace: openshift-storage
[...]
data:
  1-vault: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"rbd-encryption","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"ocs/rbd","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}
  2-vault: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.in:8201","VAULT_BACKEND_PATH":"rbd-encryption2","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"rbd-encryption","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}
  
5. Edit the configmap above, by adding a new KMSID with a custom name under the data section as shown below and click on save

kind: ConfigMap
apiVersion: v1
metadata:
  name: csi-kms-connection-details
  namespace: openshift-storage
[...]
data:
  1-vault: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"rbd-encryption","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"ocs/rbd","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}
  2-vault-2: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault-2","VAULT_ADDR":"https://vault.qe.rh-ocs.in:8201","VAULT_BACKEND_PATH":"rbd-encryption2","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"rbd-encryption","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}
  test-vault: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault-3","VAULT_ADDR":"https://vault.qe.rh-ocs.in:8203","VAULT_BACKEND_PATH":"rbd","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"test-namespace","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}

6. In the storageclass creation page, create a new encryption enabled RBD SC. In the Connection details section, edit the vault config by clicking on "Change connection details" and click on Save

7. Check if the connection details displayed is updated with the new values

Actual results:
The Connection details does not get updated with the new values. On creating the SC, it uses the old encryptionKMSID, even though the csi-kms-connection-details configmap has a new KMSID entry with the updated vault config.

kind: ConfigMap
apiVersion: v1
metadata:
  name: csi-kms-connection-details
  namespace: openshift-storage
[...]
data:
  1-vault: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"rbd-encryption","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"ocs/rbd","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}
  2-vault-2: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault-2","VAULT_ADDR":"https://vault.qe.rh-ocs.in:8201","VAULT_BACKEND_PATH":"rbd-encryption2","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"rbd-encryption","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}
  4-vault: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"kv-v2","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"kv-v2","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}
  5-vault: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault","VAULT_ADDR":"https://vault.qe.rh-ocs.com:8200","VAULT_BACKEND_PATH":"kv-v2","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"kv-v2","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}
  test-vault: >-
    {"KMS_PROVIDER":"vaulttokens","KMS_SERVICE_NAME":"vault-3","VAULT_ADDR":"https://vault.qe.rh-ocs.in:8203","VAULT_BACKEND_PATH":"rbd","VAULT_TLS_SERVER_NAME":"","VAULT_NAMESPACE":"test-namespace","VAULT_TOKEN_NAME":"ocs-kms-token","VAULT_CACERT_FILE":"","VAULT_CLIENT_CERT_FILE":"","VAULT_CLIENT_KEY_FILE":""}

$ oc get sc test-pv-encryption-3 -o yaml
allowVolumeExpansion: false
apiVersion: storage.k8s.io/v1
kind: StorageClass
[...]
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
  encrypted: "true"
  encryptionKMSID: test-vault
  imageFeatures: layering
  imageFormat: "2"
  pool: ocs-storagecluster-cephblockpool
provisioner: openshift-storage.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Expected results:
The new SC should use the updated vault connection details (encryptionKMSID) that was provided during the SC creation

Additional info: Screen recording with the reproducer steps has been attached