Bug 1874289

Description of problem:

I am working with a customer who has OpenShift 4.5 installed via VMware IPI, and persistent storage in VMware.  When they attempt to spin up pods with persistent storage, the new PersistentVolumeClaim stays in a Pending state and fails with a "Failed to provision volume with StorageClass "thin": No VM found" error.  

The customer has tested this multiple times, with the following observations.  The first time around, they had issue with Prometheus and Alert Manager but not ElasticSearch, and rebuilt their cluster.  This second time, they now have the observed issue with ElasticSearch but not Prometheus/Alert Manager.  The customer is consistently facing this issue, is now blocked, and has agreed to keep this cluster in the current state to troubleshoot further.

I've beem unable to reproduce their issue in my own lab, but I've noted two things: a) they are using VMware 6.5.0 (not 6.7U2 or higher), and b) they performed LUN snapshots and restored the OpenShift VMs from those snapshots prior to performing an upgrade / deleting pods.

Version-Release number of selected component (if applicable):
- OpenShift 4.5.5
- VMware ESXi, 6.5.0, 7388607
- VMware VMs are on Hardware version 13
- ElasticSearch is 4.5-stable from OperatorHub

How reproducible:

Consistently receiving this error in their cluster, and currently unable to recover from this state.

Steps to Reproduce:
1. Deploy ElasticSearch on OpenShift, confirm pods are up and running.
2. Shut down OpenShift VMs
3. Storage team performs snapshot of datastore - LUN backup
4. Start VMs and confirm OpenShift cluster is up and running
5. Manually test by deleting an ElasticSearch pod and see if the new pod and PVC stays pending with this error.  The customer also encountered this issue when they performed an OpenShift minor version upgrade as well each time (which results in the pod being deleted/recreated).

Actual results:
PersistentVolumeClaim stays in Pending state with "Failed to provision volume with StorageClass "thin" error.

Expected results:
PersistentVolumeClaim is able to bind to PersistentVolume via dynamic storage provisioning.

Additional info: