Bug 1435424
| Summary: | Pods timeout waiting to attach PV | ||
|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Will Gordon <wgordon> |
| Component: | Storage | Assignee: | Hemant Kumar <hekumar> |
| Status: | CLOSED DUPLICATE | QA Contact: | Jianwei Hou <jhou> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.x | CC: | aos-bugs, aos-storage-staff, hekumar |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-04-03 17:57:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
wgordon@dhcp129-36:~$ oc get pvc -o yaml mongodb
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-class: ebs
creationTimestamp: 2017-03-10T16:03:23Z
labels:
app: nodejs-mongo-persistent
template: nodejs-mongo-persistent
name: mongodb
namespace: wgordon
resourceVersion: "975773187"
selfLink: /api/v1/namespaces/wgordon/persistentvolumeclaims/mongodb
uid: 16c7c4c9-05ab-11e7-a65b-0e3d364e19a5
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
volumeName: pvc-16c7c4c9-05ab-11e7-a65b-0e3d364e19a5
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
phase: Bound
wgordon@dhcp129-36:~$ oc describe pvc mongodb
Name: mongodb
Namespace: wgordon
StorageClass: ebs
Status: Bound
Volume: pvc-16c7c4c9-05ab-11e7-a65b-0e3d364e19a5
Labels: app=nodejs-mongo-persistent
template=nodejs-mongo-persistent
Capacity: 1Gi
Access Modes: RWO
No events.
wgordon@dhcp129-36:~$ oc describe pod mongodb-1-uh46b
Name: mongodb-1-uh46b
Namespace: wgordon
Security Policy: restricted
Node: ip-172-31-2-80.ec2.internal/172.31.2.80
Start Time: Fri, 10 Mar 2017 11:04:10 -0500
Labels: deployment=mongodb-1
deploymentconfig=mongodb
name=mongodb
Status: Pending
IP:
Controllers: ReplicationController/mongodb-1
Containers:
mongodb:
Container ID:
Image: registry.access.redhat.com/rhscl/mongodb-32-rhel7@sha256:43f27a496396934c2aca166a566bf69455ee1edb2207e4fc208a3b998f665f7e
Image ID:
Port: 27017/TCP
Limits:
cpu: 1
memory: 512Mi
Requests:
cpu: 60m
memory: 307Mi
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: tcp-socket :27017 delay=30s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/sh -i -c mongo 127.0.0.1:27017/$MONGODB_DATABASE -u $MONGODB_USER -p $MONGODB_PASSWORD --eval="quit()"] delay=3s timeout=1s period=10s #success=1 #failure=3
Volume Mounts:
/var/lib/mongodb/data from mongodb-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ayrqv (ro)
Environment Variables:
MONGODB_USER: <set to the key 'database-user' in secret 'nodejs-mongo-persistent'>
MONGODB_PASSWORD: <set to the key 'database-password' in secret 'nodejs-mongo-persistent'>
MONGODB_DATABASE: sampledb
MONGODB_ADMIN_PASSWORD: <set to the key 'database-admin-password' in secret 'nodejs-mongo-persistent'>
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
mongodb-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: mongodb
ReadOnly: false
default-token-ayrqv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ayrqv
QoS Class: Burstable
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
7m 7m 1 {default-scheduler } Normal Scheduled Successfully assigned mongodb-1-uh46b to ip-172-31-2-80.ec2.internal
5m 1m 3 {kubelet ip-172-31-2-80.ec2.internal} Warning FailedMount Unable to mount volumes for pod "mongodb-1-uh46b_wgordon(327cbab6-05ab-11e7-be06-0e63b9c1c48f)": timeout expired waiting for volumes to attach/mount for pod "mongodb-1-uh46b"/"wgordon". list of unattached/unmounted volumes=[mongodb-data]
5m 1m 3 {kubelet ip-172-31-2-80.ec2.internal} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "mongodb-1-uh46b"/"wgordon". list of unattached/unmounted volumes=[mongodb-data]
get pvc - http://pastebin.test.redhat.com/470492 describe pvc - http://pastebin.test.redhat.com/470493 describe pod - http://pastebin.test.redhat.com/470499 Current version: openshift v3.4.1.10 kubernetes v1.4.0+776c994 I am going to close this as duplicate of #1436324 because same as that bug, in this bug also volume refuses to attached to new pod because volume is stuck in 'attaching' state: *** This bug has been marked as a duplicate of bug 1436324 *** |
Description of problem: Pods will not deploy due to timeout issues. The following is an example of what's found in events Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "mysql-3-5iqui"/"servicejs". list of unattached/unmounted volumes=[mysql-data] Unable to mount volumes for pod "mysql-3-5iqui_servicejs(1437caee-f3dd-11e6-be40-0e3d364e19a5)": timeout expired waiting for volumes to attach/mount for pod "mysql-3-5iqui"/"servicejs". list of unattached/unmounted volumes=[mysql-data] Version-Release number of selected component (if applicable): OpenShift Master: v3.4.1.8 (online version 3.4.1.14) Kubernetes Master: v1.4.0+776c994 How reproducible: intermittently Steps to Reproduce: 1. Deploy an application that also creates a PVC 2. Wait for pod to deploy Actual results: See the above messages in the events ("timeout expired waiting for volumes to attach/mount for pod "mysql-3-5iqui"/"servicejs". list of unattached/unmounted volumes=[mysql-data]") and pod deployment fails Expected results: Pod deployment should succeed, and PV should be attached Additional info: Breakout from https://bugzilla.redhat.com/show_bug.cgi?id=1423066#c30 This was determined to not be a recurrence, and indeed a separate bug.