| Summary: | DB pod keeps in RunContainerError state and fail to create docker container with API error (500) | ||
|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Wenjing Zheng <wzheng> |
| Component: | Containers | Assignee: | Abhishek Gupta <abhgupta> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Wenjing Zheng <wzheng> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.x | CC: | abhgupta, aos-bugs, jokerman, mmccomas |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-05-23 15:10:36 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Don't see where this is a docker issue? This is specific to Online Developer Preview, where we are specifically blocking docker from provisioning volumes within /var/lib/docker/volumes for pods that specify a VOLUME directive but the pod spec does not include a volume mount. Assigning this bug to myself for now to avoid confusion. The deployment "pre" hook uses the openshift/mysql-55-centos7:latest image and I believe the image relies on some VOLUME for its data. Since the "pre" hook does not specify a PVC or an emptyDir volume to mount against it, docker falls back to provisioning a volume directory on its own. This is something that is explicitly prevented in Online and hence the hook pod fails. You need to provide a PVC or an emptyDir for your database image for it to work. Note from Dan Mace: You need to explicitly specify which volumes you want to carry over to the hook pod via execNewPod.volumes. This should be a list of names of volumes from the template pod's volume list. Yes, after providing pvs or emptyDir, no such error, will verify this bug. Thanks for your clarification. |
Description of problem: Below error appears for db pod: Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- <invalid> <invalid> 1 {default-scheduler } Normal Scheduled Successfully assigned database-1-hook-pre to ip-172-31-15-140.ec2.internal <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/3e5355385db4ae06b649f81d4cedd28c3ef6268445b762984ce6257a72d03f96: permission denied <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/3e5355385db4ae06b649f81d4cedd28c3ef6268445b762984ce6257a72d03f96: permission denied\n" Version-Release number of selected component (if applicable): kubernetes v1.2.0-alpha.7-703-gbc4550d Docker 1.8.2-el7, build a01dc02/1.8.2 kernel 3.10.0-327.10.1.el7.x86_64 https://console.dev-preview-int.openshift.com/console/ How reproducible: always Steps to Reproduce: 1. Create a project 2. oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/sample-app/application-template-stibuild.json 3. Check pod status, especially db pod Actual results: [wzheng@openshiftqe test]$ oc get pods NAME READY STATUS RESTARTS AGE database-1-deploy 1/1 Running 0 1m database-1-hook-pre 0/1 RunContainerError 0 58s frontend-1-ha4bb 1/1 Running 0 <invalid> frontend-1-uo058 1/1 Running 0 <invalid> ruby-sample-build-1-build 0/1 Completed 0 1m [wzheng@openshiftqe test]$ oc describe pods database-1-hook-pre Name: database-1-hook-pre Namespace: wzheng2 Image(s): openshift/mysql-55-centos7:latest Node: ip-172-31-15-140.ec2.internal/172.31.15.140 Start Time: Thu, 17 Mar 2016 15:16:27 +0800 Labels: openshift.io/deployer-pod-for.name=database-1 Status: Pending Reason: Message: IP: 10.1.7.61 Controllers: <none> Containers: lifecycle: Container ID: Image: openshift/mysql-55-centos7:latest Image ID: Port: Command: /bin/true QoS Tier: cpu: BestEffort memory: BestEffort State: Waiting Reason: RunContainerError Ready: False Restart Count: 0 Environment Variables: CUSTOM_VAR1: custom_value1 MYSQL_USER: user6YJ MYSQL_PASSWORD: AYn1DNJn MYSQL_DATABASE: root OPENSHIFT_DEPLOYMENT_NAME: database-1 OPENSHIFT_DEPLOYMENT_NAMESPACE: wzheng2 Conditions: Type Status Ready False Volumes: default-token-ka49f: Type: Secret (a secret that should populate this volume) SecretName: default-token-ka49f Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- <invalid> <invalid> 1 {default-scheduler } Normal Scheduled Successfully assigned database-1-hook-pre to ip-172-31-15-140.ec2.internal <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/3e5355385db4ae06b649f81d4cedd28c3ef6268445b762984ce6257a72d03f96: permission denied <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/3e5355385db4ae06b649f81d4cedd28c3ef6268445b762984ce6257a72d03f96: permission denied\n" <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/55860555b9de6f13eb3aa62a9f24c1fb49900e9dbe007eb2c4d819bb167a17e8: permission denied <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/55860555b9de6f13eb3aa62a9f24c1fb49900e9dbe007eb2c4d819bb167a17e8: permission denied\n" <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/32c3cfc0b917ed78ff8d1ba7a196c35f4d7a22477cb2325c7c2aead91b44572e: permission denied <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/32c3cfc0b917ed78ff8d1ba7a196c35f4d7a22477cb2325c7c2aead91b44572e: permission denied\n" <invalid> <invalid> 4 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Normal Pulling pulling image "openshift/mysql-55-centos7:latest" <invalid> <invalid> 4 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Normal Pulled Successfully pulled image "openshift/mysql-55-centos7:latest" <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/5789239d348e46e0f226d34b3b925e916160ebf9a77e932fdf80932acb0746cf: permission denied\n" <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/5789239d348e46e0f226d34b3b925e916160ebf9a77e932fdf80932acb0746cf: permission denied Expected results: The db pod should be running and succeed to create docker container. Additional info: