| Summary: | EBS volume should be detached from node when node is NotReady | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Chao Yang <chaoyang> |
| Component: | Storage | Assignee: | Hemant Kumar <hekumar> |
| Status: | CLOSED NOTABUG | QA Contact: | Jianwei Hou <jhou> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.4.0 | CC: | aos-bugs, chaoyang, eparis, hekumar |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-02-06 18:17:37 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Based on operation flow, the volume should remain mounted. Closing for now, please re-open if there is any other information. Hi I tried the step "systemctl start atomic-openshift-node" and found volume still mounted while pod is deleted 1.Create a pod using ebs volume 2.Waiting for the pod status become to running [root@ip-172-18-2-110 ~]# oc get pods NAME READY STATUS RESTARTS AGE mypod 1/1 Running 0 7m 3.Systemctl stop atomic-openshift-node 4.oc delete pod mypod 5.[root@ip-172-18-2-110 ~]# oc get pods NAME READY STATUS RESTARTS AGE mypod 1/1 Terminating 0 8m 6.Systemctl start atomic-openshift-node 7.[root@ip-172-18-2-110 ~]# oc get pods No resources found. 8.Check volumes on node, found volume did not umount [root@ip-172-18-2-76 ~]# mount | grep aws /dev/xvdba on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-0fc54161053fa2dff type ext4 (rw,relatime,seclabel,data=ordered) /dev/xvdba on /var/lib/origin/openshift.local.volumes/pods/5787cf26-d641-11e6-800d-0eb7d2142d46/volumes/kubernetes.io~aws-ebs/pvc-2c297310-d641-11e6-800d-0eb7d2142d46 type ext4 (rw,relatime,seclabel,data=ordered) Jan 9 03:13:49 ip-172-18-2-76 atomic-openshift-node: I0109 03:13:49.775075 19771 kubelet_volumes.go:152] Orphaned pod "5787cf26-d641-11e6-800d-0eb7d2142d46" found, but volumes are not cleaned up; err: <nil> Closing this as not a bug. |
Description of problem: Pod without rc will be delete after node become NotReady, but volume is still mounted on this node Version-Release number of selected component (if applicable): openshift v3.4.0.23+24b1a58 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 How reproducible: Always Steps to Reproduce: 1.Create a storageclass and pvc 2.Check pvc and pv status [root@ip-172-18-0-41 ~]# oc get pv pvc-731daec4-a655-11e6-94f8-0e178310007a -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: kubernetes.io/createdby: aws-ebs-dynamic-provisioner pv.kubernetes.io/bound-by-controller: "yes" pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs volume.beta.kubernetes.io/storage-class: slow creationTimestamp: 2016-11-09T08:21:02Z labels: failure-domain.beta.kubernetes.io/region: us-east-1 failure-domain.beta.kubernetes.io/zone: us-east-1d name: pvc-731daec4-a655-11e6-94f8-0e178310007a resourceVersion: "15519" selfLink: /api/v1/persistentvolumes/pvc-731daec4-a655-11e6-94f8-0e178310007a uid: 73bac450-a655-11e6-94f8-0e178310007a spec: accessModes: - ReadWriteOnce awsElasticBlockStore: fsType: ext4 volumeID: aws://us-east-1d/vol-83b65112 capacity: storage: 500Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: claim2 namespace: default resourceVersion: "15516" uid: 731daec4-a655-11e6-94f8-0e178310007a persistentVolumeReclaimPolicy: Delete status: phase: Bound 3.Create a pod using this pvc kind: Pod apiVersion: v1 metadata: name: mypod2 labels: name: frontendhttp spec: containers: - name: myfrontend image: jhou/hello-openshift ports: - containerPort: 80 name: "http-server" volumeMounts: - mountPath: "/tmp" name: aws volumes: - name: aws persistentVolumeClaim: claimName: claim2 3.Check pod is running 4.Stop service of atomic-openshift-node Node will become NotReady ip-172-18-1-77.ec2.internal NotReady 7h 5.Pod will be deleted 6.Using `mount | grep aws` found this volume vol-83b65112 is still mounted on this node [root@ip-172-18-1-77 ~]# mount | grep aws /dev/xvdba on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1d/vol-83b65112 type ext4 (rw,relatime,seclabel,data=ordered) Actual results: Volume is still mounted Expected results: Volume should become to unmounted Additional info: