Hide Forgot
Description of problem: Below error appears for db pod: Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- <invalid> <invalid> 1 {default-scheduler } Normal Scheduled Successfully assigned database-1-hook-pre to ip-172-31-15-140.ec2.internal <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/3e5355385db4ae06b649f81d4cedd28c3ef6268445b762984ce6257a72d03f96: permission denied <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/3e5355385db4ae06b649f81d4cedd28c3ef6268445b762984ce6257a72d03f96: permission denied\n" Version-Release number of selected component (if applicable): kubernetes v1.2.0-alpha.7-703-gbc4550d Docker 1.8.2-el7, build a01dc02/1.8.2 kernel 3.10.0-327.10.1.el7.x86_64 https://console.dev-preview-int.openshift.com/console/ How reproducible: always Steps to Reproduce: 1. Create a project 2. oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/sample-app/application-template-stibuild.json 3. Check pod status, especially db pod Actual results: [wzheng@openshiftqe test]$ oc get pods NAME READY STATUS RESTARTS AGE database-1-deploy 1/1 Running 0 1m database-1-hook-pre 0/1 RunContainerError 0 58s frontend-1-ha4bb 1/1 Running 0 <invalid> frontend-1-uo058 1/1 Running 0 <invalid> ruby-sample-build-1-build 0/1 Completed 0 1m [wzheng@openshiftqe test]$ oc describe pods database-1-hook-pre Name: database-1-hook-pre Namespace: wzheng2 Image(s): openshift/mysql-55-centos7:latest Node: ip-172-31-15-140.ec2.internal/172.31.15.140 Start Time: Thu, 17 Mar 2016 15:16:27 +0800 Labels: openshift.io/deployer-pod-for.name=database-1 Status: Pending Reason: Message: IP: 10.1.7.61 Controllers: <none> Containers: lifecycle: Container ID: Image: openshift/mysql-55-centos7:latest Image ID: Port: Command: /bin/true QoS Tier: cpu: BestEffort memory: BestEffort State: Waiting Reason: RunContainerError Ready: False Restart Count: 0 Environment Variables: CUSTOM_VAR1: custom_value1 MYSQL_USER: user6YJ MYSQL_PASSWORD: AYn1DNJn MYSQL_DATABASE: root OPENSHIFT_DEPLOYMENT_NAME: database-1 OPENSHIFT_DEPLOYMENT_NAMESPACE: wzheng2 Conditions: Type Status Ready False Volumes: default-token-ka49f: Type: Secret (a secret that should populate this volume) SecretName: default-token-ka49f Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- <invalid> <invalid> 1 {default-scheduler } Normal Scheduled Successfully assigned database-1-hook-pre to ip-172-31-15-140.ec2.internal <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/3e5355385db4ae06b649f81d4cedd28c3ef6268445b762984ce6257a72d03f96: permission denied <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/3e5355385db4ae06b649f81d4cedd28c3ef6268445b762984ce6257a72d03f96: permission denied\n" <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/55860555b9de6f13eb3aa62a9f24c1fb49900e9dbe007eb2c4d819bb167a17e8: permission denied <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/55860555b9de6f13eb3aa62a9f24c1fb49900e9dbe007eb2c4d819bb167a17e8: permission denied\n" <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/32c3cfc0b917ed78ff8d1ba7a196c35f4d7a22477cb2325c7c2aead91b44572e: permission denied <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/32c3cfc0b917ed78ff8d1ba7a196c35f4d7a22477cb2325c7c2aead91b44572e: permission denied\n" <invalid> <invalid> 4 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Normal Pulling pulling image "openshift/mysql-55-centos7:latest" <invalid> <invalid> 4 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Normal Pulled Successfully pulled image "openshift/mysql-55-centos7:latest" <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "lifecycle" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/5789239d348e46e0f226d34b3b925e916160ebf9a77e932fdf80932acb0746cf: permission denied\n" <invalid> <invalid> 1 {kubelet ip-172-31-15-140.ec2.internal} spec.containers{lifecycle} Warning Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/5789239d348e46e0f226d34b3b925e916160ebf9a77e932fdf80932acb0746cf: permission denied Expected results: The db pod should be running and succeed to create docker container. Additional info:
Don't see where this is a docker issue?
This is specific to Online Developer Preview, where we are specifically blocking docker from provisioning volumes within /var/lib/docker/volumes for pods that specify a VOLUME directive but the pod spec does not include a volume mount. Assigning this bug to myself for now to avoid confusion.
The deployment "pre" hook uses the openshift/mysql-55-centos7:latest image and I believe the image relies on some VOLUME for its data. Since the "pre" hook does not specify a PVC or an emptyDir volume to mount against it, docker falls back to provisioning a volume directory on its own. This is something that is explicitly prevented in Online and hence the hook pod fails. You need to provide a PVC or an emptyDir for your database image for it to work.
Note from Dan Mace: You need to explicitly specify which volumes you want to carry over to the hook pod via execNewPod.volumes. This should be a list of names of volumes from the template pod's volume list.
Yes, after providing pvs or emptyDir, no such error, will verify this bug. Thanks for your clarification.