Bug 1320016

Summary: [online]nodejs-mongodb-example template cannot work well
Product: OpenShift Online Reporter: Wang Haoran <haowang>
Component: WebsiteAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED CURRENTRELEASE QA Contact: Yanping Zhang <yanpzhan>
Severity: medium Docs Contact:
Priority: high    
Version: 3.xCC: aos-bugs, gmontero, jokerman, mmccomas, wzheng
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-23 15:08:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
nodejs-mongodb-example template on the online env none

Description Wang Haoran 2016-03-22 06:37:16 UTC
Description of problem:
After create app using template mongodb-example template, the mongodb pod cannot startup successfully

Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1. oc new-app --template=nodejs-mongodb-example
2. check the pod status
  oc get pod

Actual results:
pod cannot start up
 [haoran@cheetah ~]$ oc describe pod mongodb-1-2yc4e
Name:				mongodb-1-2yc4e
Namespace:			haowang
Image(s):			registry.access.redhat.com/rhscl/mongodb-26-rhel7:latest
Node:				ip-172-31-15-138.ec2.internal/172.31.15.138
Start Time:			Tue, 22 Mar 2016 10:37:27 +0800
Labels:				app=nodejs-mongodb-example,deployment=mongodb-1,deploymentconfig=mongodb,name=mongodb
Status:				Pending
Reason:				
Message:			
IP:				10.1.3.169
Replication Controllers:	mongodb-1 (1/1 replicas created)
Containers:
  mongodb:
    Container ID:	
    Image:		registry.access.redhat.com/rhscl/mongodb-26-rhel7:latest
    Image ID:		
    QoS Tier:
      cpu:	Burstable
      memory:	Burstable
    Limits:
      cpu:	1
      memory:	512Mi
    Requests:
      cpu:		60m
      memory:		322122547200m
    State:		Waiting
      Reason:		RunContainerError
    Ready:		False
    Restart Count:	0
    Environment Variables:
      MONGODB_USER:		user0M2
      MONGODB_PASSWORD:		oAkqah4PlOA7Dd7U
      MONGODB_DATABASE:		sampledb
      MONGODB_ADMIN_PASSWORD:	F4MEl6y6RyHkpjCj
Conditions:
  Type		Status
  Ready 	False 
Volumes:
  mongodb-data:
    Type:	PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:	mongodb
    ReadOnly:	false
  default-token-5sunj:
    Type:	Secret (a secret that should populate this volume)
    SecretName:	default-token-5sunj
Events:
  FirstSeen	LastSeen	Count	From					SubobjectPath			Reason		Message
  ─────────	────────	─────	────					─────────────			──────		───────
  1m		1m		1	{default-scheduler }							Scheduled	Successfully assigned mongodb-1-2yc4e to ip-172-31-15-138.ec2.internal
  1m		1m		1	{kubelet ip-172-31-15-138.ec2.internal}	spec.containers{mongodb}	Failed		Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/09327824c43da8449a25d65d95a9195205f0a90e3269e8d82285d1a132dd134e: permission denied

  1m	1m	1	{kubelet ip-172-31-15-138.ec2.internal}		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/09327824c43da8449a25d65d95a9195205f0a90e3269e8d82285d1a132dd134e: permission denied\n"

  1m	1m	1	{kubelet ip-172-31-15-138.ec2.internal}	spec.containers{mongodb}	Failed	Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/93158e9ac9103386138146307716fa54cc0f5e1920776e90236bcb84751e920f: permission denied

  1m	1m	1	{kubelet ip-172-31-15-138.ec2.internal}		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/93158e9ac9103386138146307716fa54cc0f5e1920776e90236bcb84751e920f: permission denied\n"

  1m	1m	1	{kubelet ip-172-31-15-138.ec2.internal}		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/3db02d140fadf79e3888512ee7b270a948974d2977e1b213dbf3f6abd113cbe4: permission denied\n"

  1m	1m	1	{kubelet ip-172-31-15-138.ec2.internal}	spec.containers{mongodb}	Failed	Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/3db02d140fadf79e3888512ee7b270a948974d2977e1b213dbf3f6abd113cbe4: permission denied

  1m	1m	1	{kubelet ip-172-31-15-138.ec2.internal}		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/755b457608fe3a2dba8918d6fadd696b9bd9cf0a74b4fc538bd18dc8c707aa70: permission denied\n"

  1m	1m	1	{kubelet ip-172-31-15-138.ec2.internal}	spec.containers{mongodb}	Failed	Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/755b457608fe3a2dba8918d6fadd696b9bd9cf0a74b4fc538bd18dc8c707aa70: permission denied

  1m	1m	1	{kubelet ip-172-31-15-138.ec2.internal}		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/f5992c3967583963bd4782a425b00d42e04dcdb321cf4876d5c8256b8648e868: permission denied\n"

  1m	1m	1	{kubelet ip-172-31-15-138.ec2.internal}	spec.containers{mongodb}	Failed	Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/f5992c3967583963bd4782a425b00d42e04dcdb321cf4876d5c8256b8648e868: permission denied

  51s	51s	1	{kubelet ip-172-31-15-138.ec2.internal}	spec.containers{mongodb}	Failed	Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/98eed57517501dc72f6456cc741836059c2a1e5c26f738404e2f34250ae1bd1d: permission denied

  51s	51s	1	{kubelet ip-172-31-15-138.ec2.internal}		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/98eed57517501dc72f6456cc741836059c2a1e5c26f738404e2f34250ae1bd1d: permission denied\n"

  38s	38s	1	{kubelet ip-172-31-15-138.ec2.internal}	spec.containers{mongodb}	Failed	Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/60e80943e312b9b976e21383297f0344df599544b82885ef44c33260d9bca682: permission denied

  38s	38s	1	{kubelet ip-172-31-15-138.ec2.internal}		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/60e80943e312b9b976e21383297f0344df599544b82885ef44c33260d9bca682: permission denied\n"

  25s	25s	1	{kubelet ip-172-31-15-138.ec2.internal}		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/7f735be6c4312c3eb27a8366e4543c5c72bb956ef9cfe1c1dc742016cb3a181c: permission denied\n"

  25s	25s	1	{kubelet ip-172-31-15-138.ec2.internal}	spec.containers{mongodb}	Failed	Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/7f735be6c4312c3eb27a8366e4543c5c72bb956ef9cfe1c1dc742016cb3a181c: permission denied

  13s	13s	1	{kubelet ip-172-31-15-138.ec2.internal}		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "mongodb" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/17a2e5e50f7586a23821c904caf99fdeba4f4b275d095572ea0c5c486dcd72c9: permission denied\n"

  13s	13s	1	{kubelet ip-172-31-15-138.ec2.internal}	spec.containers{mongodb}	Failed	Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/17a2e5e50f7586a23821c904caf99fdeba4f4b275d095572ea0c5c486dcd72c9: permission denied
Expected results:
pod can start up 

Additional info:

[haoran@cheetah ~]$ oc get pvc -o json
{
    "kind": "List",
    "apiVersion": "v1",
    "metadata": {},
    "items": [
        {
            "kind": "PersistentVolumeClaim",
            "apiVersion": "v1",
            "metadata": {
                "name": "mongodb",
                "namespace": "haowang2",
                "selfLink": "/api/v1/namespaces/haowang2/persistentvolumeclaims/mongodb",
                "uid": "3c701f1d-eff7-11e5-b4ba-0aa949b0be09",
                "resourceVersion": "2732231",
                "creationTimestamp": "2016-03-22T06:28:05Z",
                "labels": {
                    "app": "nodejs-mongodb-example",
                    "template": "nodejs-mongodb-example"
                },
                "annotations": {
                    "openshift.io/generated-by": "OpenShiftNewApp"
                }
            },
            "spec": {
                "accessModes": [
                    "ReadWriteOnce"
                ],
                "resources": {
                    "requests": {
                        "storage": "512Mi"
                    }
                },
                "volumeName": "pv-1-dev-preview-int-master-00081-vol-2a989583"
            },
            "status": {
                "phase": "Bound",
                "accessModes": [
                    "ReadWriteOnce"
                ],
                "capacity": {
                    "storage": "1Gi"
                }
            }
        }
    ]
}

Comment 1 Wenjing Zheng 2016-03-22 06:59:41 UTC
cakephp-mysql-example also has the same issue: 
[wzheng@openshiftqe-laptop test]$ oc get pvc mysql -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  creationTimestamp: 2016-03-22T06:53:27Z
  labels:
    template: cakephp-mysql-example
  name: mysql
  namespace: wzheng
  resourceVersion: "2738345"
  selfLink: /api/v1/namespaces/wzheng/persistentvolumeclaims/mysql
  uid: c7796950-effa-11e5-af5c-0a40e3e0a381
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 512Mi
  volumeName: pv-2-dev-preview-int-master-00081-vol-7f9499d6
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  phase: Bound

Comment 2 Wenjing Zheng 2016-03-22 07:08:30 UTC
dancer-mysql-example also has such issue: Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/3018134ffee066f4bfe62a34dd1c4ac83a5fb1fe4822bc40b1ccc68233518e85: permission denied
$ oc get pvc
NAME       STATUS    VOLUME                                           CAPACITY   ACCESSMODES   AGE
database   Bound     pv-5-dev-preview-int-master-00081-vol-9a979a33   5Gi        RWO           1m

Comment 3 Abhishek Gupta 2016-03-23 23:09:57 UTC
The template will be modified to use emptyDir instead based on an offline discussion in an email thread.

Comment 4 Gabe Montero 2016-03-24 13:17:57 UTC
Don't know if this came up in the offline discussion Abhishek, but the various ephemeral versions of the languange quickstarts off of https://github.com/openshift (i.e. https://github.com/openshift/cakephp-ex, dancer-ex, etc.) have the memory limit and other various changes that were dropped in https://github.com/openshift/online.  Online dev preview could just use those.

Comment 5 Abhishek Gupta 2016-03-29 23:04:22 UTC
The templates have been modified to use emptyDir now - please try again with the examples.

Comment 6 Wang Haoran 2016-03-30 01:27:34 UTC
Created attachment 1141512 [details]
nodejs-mongodb-example template on the online env

I can't find the template are using emptyDir now, seem the attachment template get from online env.

Comment 7 Abhishek Gupta 2016-03-30 02:23:04 UTC
You can use any of the templates that were fixed by this PR --> https://github.com/openshift/origin/pull/8248

As an example, all quickstarts inside have been modified --> https://github.com/openshift/origin/tree/master/examples/quickstarts/

Comment 8 Wang Haoran 2016-03-30 02:32:44 UTC
(In reply to Abhishek Gupta from comment #7)
> You can use any of the templates that were fixed by this PR -->
> https://github.com/openshift/origin/pull/8248
> 
> As an example, all quickstarts inside have been modified -->
> https://github.com/openshift/origin/tree/master/examples/quickstarts/

I know this pr, I want using this bug to also track the installation for online picked the correct template, so I will not using the template in the git repo, I will use the template installed in the online system.

Comment 9 Abhishek Gupta 2016-03-30 02:44:18 UTC
The templates that are included in the Online environment do not use emptyDir and use persistent volumes instead. If this bug is to ensure that the available templates in Online work, you can go ahead and test them - they are expected to work now. If you run into issues with the template now, these would be different/new issues and we can debug further.

I was referring to the change made in the PR, where the default origin templates are now using emptyDir instead of relying on docker to provision the volume directory. This allows them to work without changes in the Online environment.

Comment 10 Wang Haoran 2016-03-30 02:58:41 UTC
(In reply to Abhishek Gupta from comment #9)
> The templates that are included in the Online environment do not use
> emptyDir and use persistent volumes instead. If this bug is to ensure that
> the available templates in Online work, you can go ahead and test them -
> they are expected to work now. If you run into issues with the template now,
> these would be different/new issues and we can debug further.
> 
> I was referring to the change made in the PR, where the default origin
> templates are now using emptyDir instead of relying on docker to provision
> the volume directory. This allows them to work without changes in the Online
> environment.

The original reason I created this bug is to track the installed online templates,the template under your pr was fine , but the installed templates not.
as you said , the template using pvc, but in the bug description , it is still using the docker volume, the template have pvc defined, and after create app , pvc are bound , I think I found out why pvc is bound but container is still using docker volume , see the templates here:
https://github.com/openshift/online/blob/master/templates/origin-and-exs/rails-postgresql.json#L387  all the templates under online repo are expected mount to "/var/lib/pgsql/data" , instead, we should make sure mount to the destination same as the Dockerfile defined, for mongodb , we should use https://github.com/openshift/mongodb/blob/master/2.6/Dockerfile.rhel7#L63  , otherwise , when creating container , it will still forbidden to create the docker volume.

Could you please create pr to correct this ?

Comment 11 Wang Haoran 2016-03-30 05:20:51 UTC
PR here: https://github.com/openshift/online/pull/78

Comment 12 Wang Haoran 2016-03-30 15:40:46 UTC
After the pr merged, and templates recreated by dakini, templates works well now.