Description of problem:

Version-Release number of selected component (if applicable): 4.x

How reproducible:  Have existing vsphere persistent volumes with old password.  Update the vsphere-creds secret, verify the secret update has taken by checking the vsphere-cloud-credentials secret. Restart the Pod.

Steps to Reproduce:
1. Have a pod with an existing vsphere persistent volumes that won't mount because of an old, expired password
2. Update the vsphere-creds secret with new, working credentials
3. Make a small change to the cloud-provider-config configmap to trigger an update
4. Verify the secret update has taken by checking the vsphere-cloud-credentials secret
5. Pods with existing persistent volumes will still continue to fail to mount because of bad credentials

Actual results: Pods with existing persistent volumes will still continue to fail to mount because of bad credentials, even after restart the pods, the worker nodes, master nodes, anything.


Expected results: Pods will mount the preexisting persistent volume and start their container(s)

Master Log:

Node Log (of failed PODs):   Warning  FailedAttachVolume  112s (x24 over 48m)  attachdetach-controller                        AttachVolume.Attach failed for volume "pvc-63bed637-3dce-4db3-b5fc-e18f9540d2c8" : ServerFaultCode: Cannot complete login due to an incorrect user name or password.

PV Dump: 
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: vsphere-volume-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume
  creationTimestamp: "2020-08-19T21:39:49Z"
  finalizers:
  - kubernetes.io/pv-protection
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubernetes.io/createdby: {}
          f:pv.kubernetes.io/bound-by-controller: {}
          f:pv.kubernetes.io/provisioned-by: {}
      f:spec:
        f:accessModes: {}
        f:capacity:
          .: {}
          f:storage: {}
        f:claimRef:
          .: {}
          f:apiVersion: {}
          f:kind: {}
          f:name: {}
          f:namespace: {}
          f:resourceVersion: {}
          f:uid: {}
        f:persistentVolumeReclaimPolicy: {}
        f:storageClassName: {}
        f:volumeMode: {}
        f:vsphereVolume:
          .: {}
          f:fsType: {}
          f:volumePath: {}
      f:status:
        f:phase: {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-08-19T21:39:49Z"
  name: pvc-63bed637-3dce-4db3-b5fc-e18f9540d2c8
  resourceVersion: "45696"
  selfLink: /api/v1/persistentvolumes/pvc-63bed637-3dce-4db3-b5fc-e18f9540d2c8
  uid: 851eb8ec-3353-4bcf-a2cc-3b3b052865b2
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: mongodbdir-icp-mongodb-2
    namespace: ibm-common-services
    resourceVersion: "45657"
    uid: 63bed637-3dce-4db3-b5fc-e18f9540d2c8
  persistentVolumeReclaimPolicy: Delete
  storageClassName: thin
  volumeMode: Filesystem
  vsphereVolume:
    fsType: ext4
    volumePath: '[HaaS-EDGE2-RSX4-001] kubevols/arendelle-j4nqn-dynamic-pvc-63bed637-3dce-4db3-b5fc-e18f9540d2c8.vmdk'
status:
  phase: Bound

PVC Dump:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
  creationTimestamp: "2020-08-19T21:39:47Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: icp-mongodb
    release: mongodb
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:pv.kubernetes.io/bind-completed: {}
          f:pv.kubernetes.io/bound-by-controller: {}
          f:volume.beta.kubernetes.io/storage-provisioner: {}
        f:labels:
          .: {}
          f:app: {}
          f:release: {}
      f:spec:
        f:accessModes: {}
        f:resources:
          f:requests:
            .: {}
            f:storage: {}
        f:storageClassName: {}
        f:volumeMode: {}
        f:volumeName: {}
      f:status:
        f:accessModes: {}
        f:capacity:
          .: {}
          f:storage: {}
        f:phase: {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-08-19T21:39:49Z"
  name: mongodbdir-icp-mongodb-2
  namespace: ibm-common-services
  resourceVersion: "45699"
  selfLink: /api/v1/namespaces/ibm-common-services/persistentvolumeclaims/mongodbdir-icp-mongodb-2
  uid: 63bed637-3dce-4db3-b5fc-e18f9540d2c8
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: thin
  volumeMode: Filesystem
  volumeName: pvc-63bed637-3dce-4db3-b5fc-e18f9540d2c8
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  phase: Bound

StorageClass Dump (if StorageClass used by PV/PVC):
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2020-08-19T21:00:47Z"
  managedFields:
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:storageclass.kubernetes.io/is-default-class: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"5b3222a8-1760-43b9-b277-9694dd5ae287"}:
            .: {}
            f:apiVersion: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:parameters:
        .: {}
        f:diskformat: {}
      f:provisioner: {}
      f:reclaimPolicy: {}
      f:volumeBindingMode: {}
    manager: cluster-storage-operator
    operation: Update
    time: "2020-08-19T21:00:47Z"
  name: thin
  ownerReferences:
  - apiVersion: v1
    kind: clusteroperator
    name: storage
    uid: 5b3222a8-1760-43b9-b277-9694dd5ae287
  resourceVersion: "12247"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/thin
  uid: b92bb8dc-28d5-472d-b904-f2d474bac510
parameters:
  diskformat: thin
provisioner: kubernetes.io/vsphere-volume
reclaimPolicy: Delete
volumeBindingMode: Immediate

Additional info: