Description of problem: Tests started failing consistently as of May 5th in our vsphere upi job: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.5-informing#release-openshift-ocp-installer-e2e-vsphere-upi-4.5&show-stale-tests=&sort-by-failures= Version-Release number of selected component (if applicable): 4.5 How reproducible: always
tests that are failing: [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should adopt matching orphans and release non-matching pods [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should not deadlock when a pod's predecessor fails [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should provide basic identity [Suite:openshift/conformance/parallel] [Suite:k8s]
This looks like either a setup problem on vsphare or storage since the failures happening since May 5th are consistently failing with the following error: Warning FailedScheduling <unknown> default-scheduler running "VolumeBinding" filter plugin for pod "ss-0": pod has unbound immediate PersistentVolumeClaims and pods fail to run at all. I'm moving this to storage team for further debugging, feel free to pass this over if it's a vsphare configuration issue.
This is faililing on: May 11 14:04:22.774: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for ss-0: {default-scheduler } FailedScheduling: running "VolumeBinding" filter plugin for pod "ss-0": pod has unbound immediate PersistentVolumeClaims May 11 14:04:22.774: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for ss-0: {default-scheduler } FailedScheduling: running "VolumeBinding" filter plugin for pod "ss-0": pod has unbound immediate PersistentVolumeClaims May 11 14:04:22.774: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for ss-0: {default-scheduler } FailedScheduling: skip schedule deleting pod: e2e-statefulset-6352/ss-0 May 11 14:04:22.774: INFO: At 2020-05-11 13:54:11 +0000 UTC - event for datadir-ss-0: {persistentvolume-controller } ProvisioningFailed: Failed to provision volume with StorageClass "thin": folder 'ci-op-jmc1pyfi-5c9dd-k87sv' not found May 11 14:04:22.774: INFO: At 2020-05-11 13:54:11 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Pod ss-0 in StatefulSet ss successful basically same error as - https://bugzilla.redhat.com/show_bug.cgi?id=1833137 *** This bug has been marked as a duplicate of bug 1833137 ***