Description of problem: [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [Suite:openshift/conformance/parallel] [Suite:k8s] https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-proxy-4.2/123 fail [k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:337]: Oct 22 06:13:00.619: Failed waiting for state update: timed out waiting for the condition Oct 22 06:02:49.860: INFO: Updating stateful set ss STEP: Rolling back update in reverse ordinal order Oct 22 06:02:49.913: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-kq2px8lt-9c5bf.origin-ci-int-aws.dev.rhcloud.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-statefulset-8656 ss-1 -- /bin/sh -x -c mv -v /tmp/index.html /usr/share/nginx/html/ || true' Oct 22 06:02:50.409: INFO: stderr: "+ mv -v /tmp/index.html /usr/share/nginx/html/\n" Oct 22 06:02:50.409: INFO: stdout: "'/tmp/index.html' -> '/usr/share/nginx/html/index.html'\n" Oct 22 06:02:50.409: INFO: stdout of mv -v /tmp/index.html /usr/share/nginx/html/ || true on ss-1: '/tmp/index.html' -> '/usr/share/nginx/html/index.html' Oct 22 06:03:00.514: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:03:00.515: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:03:00.515: INFO: Waiting for Pod e2e-statefulset-8656/ss-1 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:03:00.515: INFO: Waiting for Pod e2e-statefulset-8656/ss-2 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:03:10.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:03:10.549: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:03:10.549: INFO: Waiting for Pod e2e-statefulset-8656/ss-1 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:03:20.557: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:03:20.557: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:03:20.557: INFO: Waiting for Pod e2e-statefulset-8656/ss-1 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:03:30.559: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:03:30.559: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:03:40.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:03:40.549: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:03:50.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:03:50.548: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:04:00.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:04:00.548: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467 Oct 22 06:04:10.551: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:04:20.547: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:04:30.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:04:40.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:04:50.559: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:05:00.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:05:10.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:05:20.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:05:30.551: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:05:40.567: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:05:50.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:06:00.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:06:10.621: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:06:20.553: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:06:30.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:06:40.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:06:50.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:07:00.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:07:10.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:07:20.547: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:07:30.556: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:07:40.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:07:50.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:08:00.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:08:10.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:08:20.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:08:30.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:08:40.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:08:50.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:09:00.563: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:09:10.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:09:20.553: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:09:30.555: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:09:40.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:09:50.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:10:00.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:10:10.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:10:20.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:10:30.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:10:40.554: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:10:50.551: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:11:00.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:11:10.547: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:11:20.555: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:11:30.559: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:11:40.560: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:11:50.554: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:12:00.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:12:10.564: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:12:20.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:12:30.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:12:40.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:12:50.648: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:13:00.576: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:13:00.619: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update Oct 22 06:13:00.619: INFO: Failed waiting for state update: timed out waiting for the condition [AfterEach] [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/apps/statefulset.go:85 Oct 22 06:13:00.650: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-kq2px8lt-9c5bf.origin-ci-int-aws.dev.rhcloud.com:6443 --kubeconfig=/tmp/admin.kubeconfig describe po ss-0 --namespace=e2e-statefulset-8656' Oct 22 06:13:01.033: INFO: stderr: "" Oct 22 06:13:01.033: INFO: stdout: "Name: ss-0\nNamespace: e2e-statefulset-8656\nPriority: 0\nPriorityClassName: <none>\nNode: ip-10-0-138-167.ec2.internal/10.0.138.167\nStart Time: Tue, 22 Oct 2019 06:04:06 +0000\nLabels: baz=blah\n controller-revision-hash=ss-7fc99c6948\n foo=bar\n statefulset.kubernetes.io/pod-name=ss-0\nAnnotations: openshift.io/scc: hostmount-anyuid\nStatus: Pending\nIP: \nControlled By: StatefulSet/ss\nContainers:\n nginx:\n Container ID: \n Image: docker.io/library/nginx:1.14-alpine\n Image ID: \n Port: <none>\n Host Port: <none>\n State: Waiting\n Reason: ContainerCreating\n Ready: False\n Restart Count: 0\n Readiness: http-get http://:80/index.html delay=0s timeout=1s period=1s #success=1 #failure=1\n Environment: <none>\n Mounts:\n /data/ from datadir (rw)\n /home from home (rw)\n /var/run/secrets/kubernetes.io/serviceaccount from default-token-fbcfz (ro)\nConditions:\n Type Status\n Initialized True \n Ready False \n ContainersReady False \n PodScheduled True \nVolumes:\n datadir:\n Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\n ClaimName: datadir-ss-0\n ReadOnly: false\n home:\n Type: HostPath (bare host directory volume)\n Path: /tmp/home\n HostPathType: \n default-token-fbcfz:\n Type: Secret (a volume populated by a Secret)\n SecretName: default-token-fbcfz\n Optional: false\nQoS Class: BestEffort\nNode-Selectors: <none>\nTolerations: node.kubernetes.io/not-ready:NoExecute for 300s\n node.kubernetes.io/unreachable:NoExecute for 300s\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal Scheduled 8m55s default-scheduler Successfully assigned e2e-statefulset-8656/ss-0 to ip-10-0-138-167.ec2.internal\n Warning FailedMount 1s (x4 over 6m52s) kubelet, ip-10-0-138-167.ec2.internal Unable to mount volumes for pod \"ss-0_e2e-statefulset-8656(c1b74ad7-f491-11e9-bcbc-0ab47c9dac57)\": timeout expired waiting for volumes to attach or mount for pod \"e2e-statefulset-8656\"/\"ss-0\". list of unmounted volumes=[datadir]. list of unattached volumes=[datadir home default-token-fbcfz]\n" Oct 22 06:13:01.033: INFO: Output of kubectl describe ss-0: Name: ss-0 Namespace: e2e-statefulset-8656 Priority: 0 PriorityClassName: <none> Node: ip-10-0-138-167.ec2.internal/10.0.138.167 Start Time: Tue, 22 Oct 2019 06:04:06 +0000 Labels: baz=blah controller-revision-hash=ss-7fc99c6948 foo=bar statefulset.kubernetes.io/pod-name=ss-0 Annotations: openshift.io/scc: hostmount-anyuid Status: Pending IP: Controlled By: StatefulSet/ss Containers: nginx: Container ID: Image: docker.io/library/nginx:1.14-alpine Image ID: Port: <none> Host Port: <none> State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Readiness: http-get http://:80/index.html delay=0s timeout=1s period=1s #success=1 #failure=1 Environment: <none> Mounts: /data/ from datadir (rw) /home from home (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-fbcfz (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: datadir: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: datadir-ss-0 ReadOnly: false home: Type: HostPath (bare host directory volume) Path: /tmp/home HostPathType: default-token-fbcfz: Type: Secret (a volume populated by a Secret) SecretName: default-token-fbcfz Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 8m55s default-scheduler Successfully assigned e2e-statefulset-8656/ss-0 to ip-10-0-138-167.ec2.internal Warning FailedMount 1s (x4 over 6m52s) kubelet, ip-10-0-138-167.ec2.internal Unable to mount volumes for pod "ss-0_e2e-statefulset-8656(c1b74ad7-f491-11e9-bcbc-0ab47c9dac57)": timeout expired waiting for volumes to attach or mount for pod "e2e-statefulset-8656"/"ss-0". list of unmounted volumes=[datadir]. list of unattached volumes=[datadir home default-token-fbcfz]
Just double-check with bug #1698829, they are not the same issue. Verified pass.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:4181