Bug 1833135

Summary: Basic StatefulSet functionality tests failing on vsphere upi
Product: OpenShift Container Platform Reporter: Ben Parees <bparees>
Component: StorageAssignee: aos-storage-staff <aos-storage-staff>
Status: CLOSED DUPLICATE QA Contact: Qin Ping <piqin>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.5CC: aos-bugs, hekumar, mfojtik
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: trt
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should adopt matching orphans and release non-matching pods [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should not deadlock when a pod's predecessor fails [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should provide basic identity [Suite:openshift/conformance/parallel] [Suite:k8s]
Last Closed: 2020-05-11 16:17:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ben Parees 2020-05-07 20:56:16 UTC
Description of problem:
Tests started failing consistently as of May 5th in our vsphere upi job:

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.5-informing#release-openshift-ocp-installer-e2e-vsphere-upi-4.5&show-stale-tests=&sort-by-failures=



Version-Release number of selected component (if applicable):
4.5


How reproducible:
always

Comment 1 Ben Parees 2020-05-07 20:58:49 UTC
tests that are failing:

[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should adopt matching orphans and release non-matching pods [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should not deadlock when a pod's predecessor fails [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should provide basic identity [Suite:openshift/conformance/parallel] [Suite:k8s]

Comment 2 Maciej Szulik 2020-05-11 15:42:30 UTC
This looks like either a setup problem on vsphare or storage since the failures happening since May 5th are consistently failing with the following error:

Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "ss-0": pod has unbound immediate PersistentVolumeClaims

and pods fail to run at all.

I'm moving this to storage team for further debugging, feel free to pass this over if it's a vsphare configuration issue.

Comment 3 Hemant Kumar 2020-05-11 16:17:23 UTC
This is faililing on:

May 11 14:04:22.774: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for ss-0: {default-scheduler } FailedScheduling: running "VolumeBinding" filter plugin for pod "ss-0": pod has unbound immediate PersistentVolumeClaims
May 11 14:04:22.774: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for ss-0: {default-scheduler } FailedScheduling: running "VolumeBinding" filter plugin for pod "ss-0": pod has unbound immediate PersistentVolumeClaims
May 11 14:04:22.774: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for ss-0: {default-scheduler } FailedScheduling: skip schedule deleting pod: e2e-statefulset-6352/ss-0
May 11 14:04:22.774: INFO: At 2020-05-11 13:54:11 +0000 UTC - event for datadir-ss-0: {persistentvolume-controller } ProvisioningFailed: Failed to provision volume with StorageClass "thin": folder 'ci-op-jmc1pyfi-5c9dd-k87sv' not found
May 11 14:04:22.774: INFO: At 2020-05-11 13:54:11 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Pod ss-0 in StatefulSet ss successful

basically same error as  - https://bugzilla.redhat.com/show_bug.cgi?id=1833137

*** This bug has been marked as a duplicate of bug 1833137 ***