Description of problem: Pods will not deploy due to timeout issues. The following is an example of what's found in events Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "mysql-3-5iqui"/"servicejs". list of unattached/unmounted volumes=[mysql-data] Unable to mount volumes for pod "mysql-3-5iqui_servicejs(1437caee-f3dd-11e6-be40-0e3d364e19a5)": timeout expired waiting for volumes to attach/mount for pod "mysql-3-5iqui"/"servicejs". list of unattached/unmounted volumes=[mysql-data] Version-Release number of selected component (if applicable): OpenShift Master: v3.4.1.8 (online version 3.4.1.14) Kubernetes Master: v1.4.0+776c994 How reproducible: intermittently Steps to Reproduce: 1. Deploy an application that also creates a PVC 2. Wait for pod to deploy Actual results: See the above messages in the events ("timeout expired waiting for volumes to attach/mount for pod "mysql-3-5iqui"/"servicejs". list of unattached/unmounted volumes=[mysql-data]") and pod deployment fails Expected results: Pod deployment should succeed, and PV should be attached Additional info: Breakout from https://bugzilla.redhat.com/show_bug.cgi?id=1423066#c30 This was determined to not be a recurrence, and indeed a separate bug.
wgordon@dhcp129-36:~$ oc get pvc -o yaml mongodb apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-class: ebs creationTimestamp: 2017-03-10T16:03:23Z labels: app: nodejs-mongo-persistent template: nodejs-mongo-persistent name: mongodb namespace: wgordon resourceVersion: "975773187" selfLink: /api/v1/namespaces/wgordon/persistentvolumeclaims/mongodb uid: 16c7c4c9-05ab-11e7-a65b-0e3d364e19a5 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi volumeName: pvc-16c7c4c9-05ab-11e7-a65b-0e3d364e19a5 status: accessModes: - ReadWriteOnce capacity: storage: 1Gi phase: Bound wgordon@dhcp129-36:~$ oc describe pvc mongodb Name: mongodb Namespace: wgordon StorageClass: ebs Status: Bound Volume: pvc-16c7c4c9-05ab-11e7-a65b-0e3d364e19a5 Labels: app=nodejs-mongo-persistent template=nodejs-mongo-persistent Capacity: 1Gi Access Modes: RWO No events. wgordon@dhcp129-36:~$ oc describe pod mongodb-1-uh46b Name: mongodb-1-uh46b Namespace: wgordon Security Policy: restricted Node: ip-172-31-2-80.ec2.internal/172.31.2.80 Start Time: Fri, 10 Mar 2017 11:04:10 -0500 Labels: deployment=mongodb-1 deploymentconfig=mongodb name=mongodb Status: Pending IP: Controllers: ReplicationController/mongodb-1 Containers: mongodb: Container ID: Image: registry.access.redhat.com/rhscl/mongodb-32-rhel7@sha256:43f27a496396934c2aca166a566bf69455ee1edb2207e4fc208a3b998f665f7e Image ID: Port: 27017/TCP Limits: cpu: 1 memory: 512Mi Requests: cpu: 60m memory: 307Mi State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Liveness: tcp-socket :27017 delay=30s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [/bin/sh -i -c mongo 127.0.0.1:27017/$MONGODB_DATABASE -u $MONGODB_USER -p $MONGODB_PASSWORD --eval="quit()"] delay=3s timeout=1s period=10s #success=1 #failure=3 Volume Mounts: /var/lib/mongodb/data from mongodb-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-ayrqv (ro) Environment Variables: MONGODB_USER: <set to the key 'database-user' in secret 'nodejs-mongo-persistent'> MONGODB_PASSWORD: <set to the key 'database-password' in secret 'nodejs-mongo-persistent'> MONGODB_DATABASE: sampledb MONGODB_ADMIN_PASSWORD: <set to the key 'database-admin-password' in secret 'nodejs-mongo-persistent'> Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: mongodb-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: mongodb ReadOnly: false default-token-ayrqv: Type: Secret (a volume populated by a Secret) SecretName: default-token-ayrqv QoS Class: Burstable Tolerations: <none> Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 7m 7m 1 {default-scheduler } Normal Scheduled Successfully assigned mongodb-1-uh46b to ip-172-31-2-80.ec2.internal 5m 1m 3 {kubelet ip-172-31-2-80.ec2.internal} Warning FailedMount Unable to mount volumes for pod "mongodb-1-uh46b_wgordon(327cbab6-05ab-11e7-be06-0e63b9c1c48f)": timeout expired waiting for volumes to attach/mount for pod "mongodb-1-uh46b"/"wgordon". list of unattached/unmounted volumes=[mongodb-data] 5m 1m 3 {kubelet ip-172-31-2-80.ec2.internal} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "mongodb-1-uh46b"/"wgordon". list of unattached/unmounted volumes=[mongodb-data]
get pvc - http://pastebin.test.redhat.com/470492 describe pvc - http://pastebin.test.redhat.com/470493 describe pod - http://pastebin.test.redhat.com/470499 Current version: openshift v3.4.1.10 kubernetes v1.4.0+776c994
I am going to close this as duplicate of #1436324 because same as that bug, in this bug also volume refuses to attached to new pod because volume is stuck in 'attaching' state: *** This bug has been marked as a duplicate of bug 1436324 ***