Hide Forgot
Description of problem: Could not attach ebs volume in containered env for openshift Version-Release number of selected component (if applicable): openshift v3.2.0.6 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 How reproducible: Always Steps to Reproduce: 1. Create a dynamic provison pvc { "kind": "PersistentVolumeClaim", "apiVersion": "v1", "metadata": { "name": "claim2", "labels": { "name": "testing" }, "annotations": { "volume.alpha.kubernetes.io/storage-class": "foo" } }, "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "3Gi" } } } } 2. NAME STATUS VOLUME CAPACITY ACCESSMODES AGE claim2 Bound pv-aws-gc8n1 3Gi RWO 1h 3. Create a pod using pvc claim2 4. Check pod status Actual results: 1.volume vol-f4798356 is in-use status from aws ebs web console 2. oc get event 24m 24m 1 awspd1 Pod Normal Scheduled {default-scheduler } Successfully assigned awspd1 to ip-172-18-1-232.ec2.internal 14m 4m 2 awspd1 Pod Warning FailedMount {kubelet ip-172-18-1-232.ec2.internal} Unable to mount volumes for pod "awspd1_default(5a5163a1-f00d-11e5-83f4-0e8b1f5ad811)": Could not attach EBS Disk "aws://us-east-1d/vol-f4798356". Timeout waiting for mount paths to be created. 14m 4m 2 awspd1 Pod Warning FailedSync {kubelet ip-172-18-1-232.ec2.internal} Error syncing pod, skipping: Could not attach EBS Disk "aws://us-east-1d/vol-f4798356". Timeout waiting for mount paths to be created. Expected results: Pod should be running Additional info: Mar 22 09:06:24 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: I0322 09:06:24.862321 65540 aws.go:1070] Assigned mount device bc -> volume vol-f4798356 Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: I0322 09:06:25.149716 65540 aws.go:1318] AttachVo lume request returned { Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: AttachTime: 2016-03-22 09:06:25.145 +0000 UTC, Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: Device: "/dev/xvdbc", Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: InstanceId: "i-e2e35379", Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: State: "attaching", Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: VolumeId: "vol-f4798356" Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: } Mar 22 09:06:25 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: I0322 09:06:25.260695 65540 aws.go:1193] Waiting for volume state: actual=attaching, desired=attached Mar 22 09:06:26 ip-172-18-1-232.ec2.internal atomic-openshift-node[65494]: I0322 09:06:26.311786 65540 aws.go:1193] Waiting for volume state: actual=attaching, desired=attached Mar 22 09:06:27 ip-172-18-1-232.ec2.internal kernel: blkfront: xvdbc: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; Mar 22 09:06:27 ip-172-18-1-232.ec2.internal kernel: xvdbc: unknown partition table
Fixed by https://github.com/openshift/origin/pull/8182
This should be in OSE v3.2.0.7 that was build and pushed to qe today.
Verification is failed on openshift v3.2.0.7 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5 Pods keeps status of "ContainerCreating" 1m 1m 1 test2 Pod Normal Scheduled {default-scheduler } Successfully assigned test2 to ip-172-18-3-242.ec2.internal 1m 8s 7 test2 Pod Warning FailedMount {kubelet ip-172-18-3-242.ec2.internal} Unable to mount volumes for pod "test2_default(ca2f8bf0-f171-11e5-a9f0-0e0a208947f9)": exit status 32 1m 8s 7 test2 Pod Warning FailedSync {kubelet ip-172-18-3-242.ec2.internal} Error syncing pod, skipping: exit status 32 Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: W0324 03:36:04.198738 10565 aws.go:1043] Got assignment call for already-assigned volume: ba@vol-b139f513 Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: E0324 03:36:04.301091 10565 aws.go:1081] endAttaching on non-allocated device Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: I0324 03:36:04.301122 10565 aws_util.go:208] Successfully attached EBS Disk "aws://us-east-1d/vol-b139f513". Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: I0324 03:36:04.304315 10565 nsenter_mount.go:179] Failed findmnt command: exit status 1 Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: E0324 03:36:04.331983 10565 kubelet.go:1796] Unable to mount volumes for pod "test2_default(27e92b1a-f171-11e5-a9f0-0e0a208947f9)": exit status 32; skipping pod Mar 24 03:36:04 ip-172-18-3-242.ec2.internal atomic-openshift-node[10519]: E0324 03:36:04.332001 10565 pod_workers.go:138] Error syncing pod 27e92b1a-f171-11e5-a9f0-0e0a208947f9, skipping: exit status 32
This is sort of duplicate of bug #1313210, I'll keep it open to track progress, but any discussion should happen in #1313210.
Merged as origin PR: https://github.com/openshift/origin/pull/8501
I test this bug on ose, pod using aws ebs volume can be runing openshift v3.2.0.20 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5
have tested and passed on openshift v3.2.0.20 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5
Fixed in 3.2.0.20