Bug 1764044

Summary: [sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [Suite:openshift/conformance/parallel] [Suite:k8s]
Product: OpenShift Container Platform Reporter: Wei Duan <wduan>
Component: StorageAssignee: Jan Safranek <jsafrane>
Status: CLOSED ERRATA QA Contact: Wei Duan <wduan>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.2.0CC: aos-bugs, aos-storage-staff, fbertina, hekumar, jsafrane, scuppett
Target Milestone: ---   
Target Release: 4.2.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-12-20 00:46:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1698829    
Bug Blocks:    

Description Wei Duan 2019-10-22 08:04:06 UTC 
Description of problem:

[sig-apps] StatefulSet [k8s.io] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [Suite:openshift/conformance/parallel] [Suite:k8s]

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-proxy-4.2/123

fail [k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:337]: Oct 22 06:13:00.619: Failed waiting for state update: timed out waiting for the condition


Oct 22 06:02:49.860: INFO: Updating stateful set ss
STEP: Rolling back update in reverse ordinal order
Oct 22 06:02:49.913: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-kq2px8lt-9c5bf.origin-ci-int-aws.dev.rhcloud.com:6443 --kubeconfig=/tmp/admin.kubeconfig exec --namespace=e2e-statefulset-8656 ss-1 -- /bin/sh -x -c mv -v /tmp/index.html /usr/share/nginx/html/ || true'
Oct 22 06:02:50.409: INFO: stderr: "+ mv -v /tmp/index.html /usr/share/nginx/html/\n"
Oct 22 06:02:50.409: INFO: stdout: "'/tmp/index.html' -> '/usr/share/nginx/html/index.html'\n"
Oct 22 06:02:50.409: INFO: stdout of mv -v /tmp/index.html /usr/share/nginx/html/ || true on ss-1: '/tmp/index.html' -> '/usr/share/nginx/html/index.html'

Oct 22 06:03:00.514: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:03:00.515: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:03:00.515: INFO: Waiting for Pod e2e-statefulset-8656/ss-1 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:03:00.515: INFO: Waiting for Pod e2e-statefulset-8656/ss-2 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:03:10.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:03:10.549: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:03:10.549: INFO: Waiting for Pod e2e-statefulset-8656/ss-1 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:03:20.557: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:03:20.557: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:03:20.557: INFO: Waiting for Pod e2e-statefulset-8656/ss-1 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:03:30.559: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:03:30.559: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:03:40.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:03:40.549: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:03:50.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:03:50.548: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:04:00.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:04:00.548: INFO: Waiting for Pod e2e-statefulset-8656/ss-0 to have revision ss-7fc99c6948 update revision ss-54bd7b9467
Oct 22 06:04:10.551: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:04:20.547: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:04:30.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:04:40.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:04:50.559: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:05:00.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:05:10.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:05:20.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:05:30.551: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:05:40.567: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:05:50.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:06:00.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:06:10.621: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:06:20.553: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:06:30.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:06:40.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:06:50.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:07:00.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:07:10.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:07:20.547: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:07:30.556: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:07:40.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:07:50.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:08:00.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:08:10.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:08:20.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:08:30.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:08:40.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:08:50.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:09:00.563: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:09:10.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:09:20.553: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:09:30.555: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:09:40.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:09:50.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:10:00.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:10:10.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:10:20.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:10:30.549: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:10:40.554: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:10:50.551: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:11:00.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:11:10.547: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:11:20.555: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:11:30.559: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:11:40.560: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:11:50.554: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:12:00.550: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:12:10.564: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:12:20.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:12:30.552: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:12:40.548: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:12:50.648: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:13:00.576: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:13:00.619: INFO: Waiting for StatefulSet e2e-statefulset-8656/ss to complete update
Oct 22 06:13:00.619: INFO: Failed waiting for state update: timed out waiting for the condition
[AfterEach] [k8s.io] Basic StatefulSet functionality [StatefulSetBasic]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/apps/statefulset.go:85
Oct 22 06:13:00.650: INFO: Running '/usr/bin/kubectl --server=https://api.ci-op-kq2px8lt-9c5bf.origin-ci-int-aws.dev.rhcloud.com:6443 --kubeconfig=/tmp/admin.kubeconfig describe po ss-0 --namespace=e2e-statefulset-8656'
Oct 22 06:13:01.033: INFO: stderr: ""
Oct 22 06:13:01.033: INFO: stdout: "Name:               ss-0\nNamespace:          e2e-statefulset-8656\nPriority:           0\nPriorityClassName:  <none>\nNode:               ip-10-0-138-167.ec2.internal/10.0.138.167\nStart Time:         Tue, 22 Oct 2019 06:04:06 +0000\nLabels:             baz=blah\n                    controller-revision-hash=ss-7fc99c6948\n                    foo=bar\n                    statefulset.kubernetes.io/pod-name=ss-0\nAnnotations:        openshift.io/scc: hostmount-anyuid\nStatus:             Pending\nIP:                 \nControlled By:      StatefulSet/ss\nContainers:\n  nginx:\n    Container ID:   \n    Image:          docker.io/library/nginx:1.14-alpine\n    Image ID:       \n    Port:           <none>\n    Host Port:      <none>\n    State:          Waiting\n      Reason:       ContainerCreating\n    Ready:          False\n    Restart Count:  0\n    Readiness:      http-get http://:80/index.html delay=0s timeout=1s period=1s #success=1 #failure=1\n    Environment:    <none>\n    Mounts:\n      /data/ from datadir (rw)\n      /home from home (rw)\n      /var/run/secrets/kubernetes.io/serviceaccount from default-token-fbcfz (ro)\nConditions:\n  Type              Status\n  Initialized       True \n  Ready             False \n  ContainersReady   False \n  PodScheduled      True \nVolumes:\n  datadir:\n    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\n    ClaimName:  datadir-ss-0\n    ReadOnly:   false\n  home:\n    Type:          HostPath (bare host directory volume)\n    Path:          /tmp/home\n    HostPathType:  \n  default-token-fbcfz:\n    Type:        Secret (a volume populated by a Secret)\n    SecretName:  default-token-fbcfz\n    Optional:    false\nQoS Class:       BestEffort\nNode-Selectors:  <none>\nTolerations:     node.kubernetes.io/not-ready:NoExecute for 300s\n                 node.kubernetes.io/unreachable:NoExecute for 300s\nEvents:\n  Type     Reason       Age                 From                                   Message\n  ----     ------       ----                ----                                   -------\n  Normal   Scheduled    8m55s               default-scheduler                      Successfully assigned e2e-statefulset-8656/ss-0 to ip-10-0-138-167.ec2.internal\n  Warning  FailedMount  1s (x4 over 6m52s)  kubelet, ip-10-0-138-167.ec2.internal  Unable to mount volumes for pod \"ss-0_e2e-statefulset-8656(c1b74ad7-f491-11e9-bcbc-0ab47c9dac57)\": timeout expired waiting for volumes to attach or mount for pod \"e2e-statefulset-8656\"/\"ss-0\". list of unmounted volumes=[datadir]. list of unattached volumes=[datadir home default-token-fbcfz]\n"
Oct 22 06:13:01.033: INFO: 
Output of kubectl describe ss-0:
Name:               ss-0
Namespace:          e2e-statefulset-8656
Priority:           0
PriorityClassName:  <none>
Node:               ip-10-0-138-167.ec2.internal/10.0.138.167
Start Time:         Tue, 22 Oct 2019 06:04:06 +0000
Labels:             baz=blah
                    controller-revision-hash=ss-7fc99c6948
                    foo=bar
                    statefulset.kubernetes.io/pod-name=ss-0
Annotations:        openshift.io/scc: hostmount-anyuid
Status:             Pending
IP:                 
Controlled By:      StatefulSet/ss
Containers:
  nginx:
    Container ID:   
    Image:          docker.io/library/nginx:1.14-alpine
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Readiness:      http-get http://:80/index.html delay=0s timeout=1s period=1s #success=1 #failure=1
    Environment:    <none>
    Mounts:
      /data/ from datadir (rw)
      /home from home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-fbcfz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-ss-0
    ReadOnly:   false
  home:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/home
    HostPathType:  
  default-token-fbcfz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-fbcfz
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                 From                                   Message
  ----     ------       ----                ----                                   -------
  Normal   Scheduled    8m55s               default-scheduler                      Successfully assigned e2e-statefulset-8656/ss-0 to ip-10-0-138-167.ec2.internal
  Warning  FailedMount  1s (x4 over 6m52s)  kubelet, ip-10-0-138-167.ec2.internal  Unable to mount volumes for pod "ss-0_e2e-statefulset-8656(c1b74ad7-f491-11e9-bcbc-0ab47c9dac57)": timeout expired waiting for volumes to attach or mount for pod "e2e-statefulset-8656"/"ss-0". list of unmounted volumes=[datadir]. list of unattached volumes=[datadir home default-token-fbcfz]
Comment 7 Wei Duan 2019-12-12 03:23:14 UTC 
Just double-check with bug #1698829, they are not the same issue.
Verified pass.
Comment 11 errata-xmlrpc 2019-12-20 00:46:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4181