Hide Forgot
Description of problem: After create app using template mongodb-example template, the mongodb pod cannot startup successfully Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1. oc new-app --template=nodejs-mongodb-example 2. check the pod status oc get pod Actual results: pod cannot start up [haoran@cheetah ~]$ oc describe pod mongodb-1-2yc4e Name: mongodb-1-2yc4e Namespace: haowang Image(s): registry.access.redhat.com/rhscl/mongodb-26-rhel7:latest Node: ip-172-31-15-138.ec2.internal/172.31.15.138 Start Time: Tue, 22 Mar 2016 10:37:27 +0800 Labels: app=nodejs-mongodb-example,deployment=mongodb-1,deploymentconfig=mongodb,name=mongodb Status: Pending Reason: Message: IP: 10.1.3.169 Replication Controllers: mongodb-1 (1/1 replicas created) Containers: mongodb: Container ID: Image: registry.access.redhat.com/rhscl/mongodb-26-rhel7:latest Image ID: QoS Tier: cpu: Burstable memory: Burstable Limits: cpu: 1 memory: 512Mi Requests: cpu: 60m memory: 322122547200m State: Waiting Reason: RunContainerError Ready: False Restart Count: 0 Environment Variables: MONGODB_USER: user0M2 MONGODB_PASSWORD: oAkqah4PlOA7Dd7U MONGODB_DATABASE: sampledb MONGODB_ADMIN_PASSWORD: F4MEl6y6RyHkpjCj Conditions: Type Status Ready False Volumes: mongodb-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: mongodb ReadOnly: false default-token-5sunj: Type: Secret (a secret that should populate this volume) SecretName: default-token-5sunj Events: FirstSeen LastSeen Count From SubobjectPath Reason Message ───────── ──────── ───── ──── ───────────── ────── ─────── 1m 1m 1 {default-scheduler } Scheduled Successfully assigned mongodb-1-2yc4e to ip-172-31-15-138.ec2.internal 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} spec.containers{mongodb} Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/09327824c43da8449a25d65d95a9195205f0a90e3269e8d82285d1a132dd134e: permission denied 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/09327824c43da8449a25d65d95a9195205f0a90e3269e8d82285d1a132dd134e: permission denied\n" 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} spec.containers{mongodb} Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/93158e9ac9103386138146307716fa54cc0f5e1920776e90236bcb84751e920f: permission denied 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/93158e9ac9103386138146307716fa54cc0f5e1920776e90236bcb84751e920f: permission denied\n" 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/3db02d140fadf79e3888512ee7b270a948974d2977e1b213dbf3f6abd113cbe4: permission denied\n" 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} spec.containers{mongodb} Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/3db02d140fadf79e3888512ee7b270a948974d2977e1b213dbf3f6abd113cbe4: permission denied 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/755b457608fe3a2dba8918d6fadd696b9bd9cf0a74b4fc538bd18dc8c707aa70: permission denied\n" 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} spec.containers{mongodb} Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/755b457608fe3a2dba8918d6fadd696b9bd9cf0a74b4fc538bd18dc8c707aa70: permission denied 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/f5992c3967583963bd4782a425b00d42e04dcdb321cf4876d5c8256b8648e868: permission denied\n" 1m 1m 1 {kubelet ip-172-31-15-138.ec2.internal} spec.containers{mongodb} Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/f5992c3967583963bd4782a425b00d42e04dcdb321cf4876d5c8256b8648e868: permission denied 51s 51s 1 {kubelet ip-172-31-15-138.ec2.internal} spec.containers{mongodb} Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/98eed57517501dc72f6456cc741836059c2a1e5c26f738404e2f34250ae1bd1d: permission denied 51s 51s 1 {kubelet ip-172-31-15-138.ec2.internal} FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/98eed57517501dc72f6456cc741836059c2a1e5c26f738404e2f34250ae1bd1d: permission denied\n" 38s 38s 1 {kubelet ip-172-31-15-138.ec2.internal} spec.containers{mongodb} Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/60e80943e312b9b976e21383297f0344df599544b82885ef44c33260d9bca682: permission denied 38s 38s 1 {kubelet ip-172-31-15-138.ec2.internal} FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/60e80943e312b9b976e21383297f0344df599544b82885ef44c33260d9bca682: permission denied\n" 25s 25s 1 {kubelet ip-172-31-15-138.ec2.internal} FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/7f735be6c4312c3eb27a8366e4543c5c72bb956ef9cfe1c1dc742016cb3a181c: permission denied\n" 25s 25s 1 {kubelet ip-172-31-15-138.ec2.internal} spec.containers{mongodb} Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/7f735be6c4312c3eb27a8366e4543c5c72bb956ef9cfe1c1dc742016cb3a181c: permission denied 13s 13s 1 {kubelet ip-172-31-15-138.ec2.internal} FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/17a2e5e50f7586a23821c904caf99fdeba4f4b275d095572ea0c5c486dcd72c9: permission denied\n" 13s 13s 1 {kubelet ip-172-31-15-138.ec2.internal} spec.containers{mongodb} Failed Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/17a2e5e50f7586a23821c904caf99fdeba4f4b275d095572ea0c5c486dcd72c9: permission denied Expected results: pod can start up Additional info: [haoran@cheetah ~]$ oc get pvc -o json { "kind": "List", "apiVersion": "v1", "metadata": {}, "items": [ { "kind": "PersistentVolumeClaim", "apiVersion": "v1", "metadata": { "name": "mongodb", "namespace": "haowang2", "selfLink": "/api/v1/namespaces/haowang2/persistentvolumeclaims/mongodb", "uid": "3c701f1d-eff7-11e5-b4ba-0aa949b0be09", "resourceVersion": "2732231", "creationTimestamp": "2016-03-22T06:28:05Z", "labels": { "app": "nodejs-mongodb-example", "template": "nodejs-mongodb-example" }, "annotations": { "openshift.io/generated-by": "OpenShiftNewApp" } }, "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "512Mi" } }, "volumeName": "pv-1-dev-preview-int-master-00081-vol-2a989583" }, "status": { "phase": "Bound", "accessModes": [ "ReadWriteOnce" ], "capacity": { "storage": "1Gi" } } } ] }
cakephp-mysql-example also has the same issue: [wzheng@openshiftqe-laptop test]$ oc get pvc mysql -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: creationTimestamp: 2016-03-22T06:53:27Z labels: template: cakephp-mysql-example name: mysql namespace: wzheng resourceVersion: "2738345" selfLink: /api/v1/namespaces/wzheng/persistentvolumeclaims/mysql uid: c7796950-effa-11e5-af5c-0a40e3e0a381 spec: accessModes: - ReadWriteOnce resources: requests: storage: 512Mi volumeName: pv-2-dev-preview-int-master-00081-vol-7f9499d6 status: accessModes: - ReadWriteOnce capacity: storage: 2Gi phase: Bound
dancer-mysql-example also has such issue: Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/3018134ffee066f4bfe62a34dd1c4ac83a5fb1fe4822bc40b1ccc68233518e85: permission denied $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES AGE database Bound pv-5-dev-preview-int-master-00081-vol-9a979a33 5Gi RWO 1m
The template will be modified to use emptyDir instead based on an offline discussion in an email thread.
Don't know if this came up in the offline discussion Abhishek, but the various ephemeral versions of the languange quickstarts off of https://github.com/openshift (i.e. https://github.com/openshift/cakephp-ex, dancer-ex, etc.) have the memory limit and other various changes that were dropped in https://github.com/openshift/online. Online dev preview could just use those.
The templates have been modified to use emptyDir now - please try again with the examples.
Created attachment 1141512 [details] nodejs-mongodb-example template on the online env I can't find the template are using emptyDir now, seem the attachment template get from online env.
You can use any of the templates that were fixed by this PR --> https://github.com/openshift/origin/pull/8248 As an example, all quickstarts inside have been modified --> https://github.com/openshift/origin/tree/master/examples/quickstarts/
(In reply to Abhishek Gupta from comment #7) > You can use any of the templates that were fixed by this PR --> > https://github.com/openshift/origin/pull/8248 > > As an example, all quickstarts inside have been modified --> > https://github.com/openshift/origin/tree/master/examples/quickstarts/ I know this pr, I want using this bug to also track the installation for online picked the correct template, so I will not using the template in the git repo, I will use the template installed in the online system.
The templates that are included in the Online environment do not use emptyDir and use persistent volumes instead. If this bug is to ensure that the available templates in Online work, you can go ahead and test them - they are expected to work now. If you run into issues with the template now, these would be different/new issues and we can debug further. I was referring to the change made in the PR, where the default origin templates are now using emptyDir instead of relying on docker to provision the volume directory. This allows them to work without changes in the Online environment.
(In reply to Abhishek Gupta from comment #9) > The templates that are included in the Online environment do not use > emptyDir and use persistent volumes instead. If this bug is to ensure that > the available templates in Online work, you can go ahead and test them - > they are expected to work now. If you run into issues with the template now, > these would be different/new issues and we can debug further. > > I was referring to the change made in the PR, where the default origin > templates are now using emptyDir instead of relying on docker to provision > the volume directory. This allows them to work without changes in the Online > environment. The original reason I created this bug is to track the installed online templates,the template under your pr was fine , but the installed templates not. as you said , the template using pvc, but in the bug description , it is still using the docker volume, the template have pvc defined, and after create app , pvc are bound , I think I found out why pvc is bound but container is still using docker volume , see the templates here: https://github.com/openshift/online/blob/master/templates/origin-and-exs/rails-postgresql.json#L387 all the templates under online repo are expected mount to "/var/lib/pgsql/data" , instead, we should make sure mount to the destination same as the Dockerfile defined, for mongodb , we should use https://github.com/openshift/mongodb/blob/master/2.6/Dockerfile.rhel7#L63 , otherwise , when creating container , it will still forbidden to create the docker volume. Could you please create pr to correct this ?
PR here: https://github.com/openshift/online/pull/78
After the pr merged, and templates recreated by dakini, templates works well now.