| Summary: | Could not attach ebs volume in containered env for openshift | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Chao Yang <chaoyang> |
| Component: | Storage | Assignee: | Jan Safranek <jsafrane> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Jianwei Hou <jhou> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.2.0 | CC: | aos-bugs, bchilds, tdawson |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-06-13 18:47:04 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
This should be in OSE v3.2.0.7 that was build and pushed to qe today. Verification is failed on
openshift v3.2.0.7
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5
Pods keeps status of "ContainerCreating"
1m 1m 1 test2 Pod Normal Scheduled {default-scheduler } Successfully assigned test2 to ip-172-18-3-242.ec2.internal
1m 8s 7 test2 Pod Warning FailedMount {kubelet ip-172-18-3-242.ec2.internal} Unable to mount volumes for pod "test2_default(ca2f8bf0-f171-11e5-a9f0-0e0a208947f9)": exit status 32
1m 8s 7 test2 Pod Warning FailedSync {kubelet ip-172-18-3-242.ec2.internal} Error syncing pod, skipping: exit status 32
Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: W0324 03:36:04.198738 10565 aws.go:1043] Got assignment call for already-assigned volume: ba@vol-b139f513
Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: E0324 03:36:04.301091 10565 aws.go:1081] endAttaching on non-allocated device
Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: I0324 03:36:04.301122 10565 aws_util.go:208] Successfully attached EBS Disk "aws://us-east-1d/vol-b139f513".
Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: I0324 03:36:04.304315 10565 nsenter_mount.go:179] Failed findmnt command: exit status 1
Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: E0324 03:36:04.331983 10565 kubelet.go:1796] Unable to mount volumes for pod "test2_default(27e92b1a-f171-11e5-a9f0-0e0a208947f9)": exit status 32; skipping pod
Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: E0324 03:36:04.332001 10565 pod_workers.go:138] Error syncing pod 27e92b1a-f171-11e5-a9f0-0e0a208947f9, skipping: exit status 32
This is sort of duplicate of bug #1313210, I'll keep it open to track progress, but any discussion should happen in #1313210. Merged as origin PR: https://github.com/openshift/origin/pull/8501 I test this bug on ose, pod using aws ebs volume can be runing openshift v3.2.0.20 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 have tested and passed on openshift v3.2.0.20 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 Fixed in 3.2.0.20 |
Description of problem: Could not attach ebs volume in containered env for openshift Version-Release number of selected component (if applicable): openshift v3.2.0.6 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 How reproducible: Always Steps to Reproduce: 1. Create a dynamic provison pvc { "kind": "PersistentVolumeClaim", "apiVersion": "v1", "metadata": { "name": "claim2", "labels": { "name": "testing" }, "annotations": { "volume.alpha.kubernetes.io/storage-class": "foo" } }, "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "3Gi" } } } } 2. NAME STATUS VOLUME CAPACITY ACCESSMODES AGE claim2 Bound pv-aws-gc8n1 3Gi RWO 1h 3. Create a pod using pvc claim2 4. Check pod status Actual results: 1.volume vol-f4798356 is in-use status from aws ebs web console 2. oc get event 24m 24m 1 awspd1 Pod Normal Scheduled {default-scheduler } Successfully assigned awspd1 to ip-172-18-1-232.ec2.internal 14m 4m 2 awspd1 Pod Warning FailedMount {kubelet ip-172-18-1-232.ec2.internal} Unable to mount volumes for pod "awspd1_default(5a5163a1-f00d-11e5-83f4-0e8b1f5ad811)": Could not attach EBS Disk "aws://us-east-1d/vol-f4798356". Timeout waiting for mount paths to be created. 14m 4m 2 awspd1 Pod Warning FailedSync {kubelet ip-172-18-1-232.ec2.internal} Error syncing pod, skipping: Could not attach EBS Disk "aws://us-east-1d/vol-f4798356". Timeout waiting for mount paths to be created. Expected results: Pod should be running Additional info: Mar 22 09:06:24 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: I0322 09:06:24.862321 65540 aws.go:1070] Assigned mount device bc -> volume vol-f4798356 Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: I0322 09:06:25.149716 65540 aws.go:1318] AttachVo lume request returned { Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: AttachTime: 2016-03-22 09:06:25.145 +0000 UTC, Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: Device: "/dev/xvdbc", Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: InstanceId: "i-e2e35379", Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: State: "attaching", Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: VolumeId: "vol-f4798356" Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: } Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: I0322 09:06:25.260695 65540 aws.go:1193] Waiting for volume state: actual=attaching, desired=attached Mar 22 09:06:26 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: I0322 09:06:26.311786 65540 aws.go:1193] Waiting for volume state: actual=attaching, desired=attached Mar 22 09:06:27 ip-172-18-1-232.ec2.internal kernel: blkfront: xvdbc: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; Mar 22 09:06:27 ip-172-18-1-232.ec2.internal kernel: xvdbc: unknown partition table