Bug 1336318

Summary: [online]Failed to create gitserver pod
Product: OpenShift Online Reporter: XiuJuan Wang <xiuwang>
Component: BuildAssignee: Cesar Wong <cewong>
Status: CLOSED CURRENTRELEASE QA Contact: Wenjing Zheng <wzheng>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.xCC: aos-bugs, bparees, jokerman, mfojtik, mmccomas
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-23 17:31:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description XiuJuan Wang 2016-05-16 07:28:46 UTC 
Description of problem:
Due to 'Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/e502a13b9ce0e665bf77b912fb0a4d7fc46fd83ad8519858b7fc97f3bebceb7e: permission denied' error, can't create git server pod

Works before in int env(older than openshift master:v3.2.0.40)

Version-Release number of selected component (if applicable):
dev-preview-stg (openshift master:v3.2.0.44)

openshift/origin-gitserver (imageid:28042b7dc0fd)
How reproducible:


Steps to Reproduce:
1.Create the Git Server
$oc create  -f https://raw.githubusercontent.com/openshift/origin/master/examples/gitserver/gitserver.yaml
2.Grant edit access to the git service account
$oc policy add-role-to-user openshift-online:edit -z git 

3.Check pod
# oc  get  pods 
NAME           READY     STATUS              RESTARTS   AGE
git-1-deploy   0/1       Error               0          31m
git-2-deploy   1/1       Running             0          <invalid>
git-2-w5dfg    0/1       RunContainerError   0          <invalid>


Actual results:

Events:
  FirstSeen	LastSeen	Count	From					SubobjectPath		Type	Reason		Message
  ---------	--------	-----	----					-------------		--------------		-------
  <invalid>	<invalid>	1	{default-scheduler }						Normal	Scheduled	Successfully assigned git-2-w5dfg to ip-172-31-9-165.ec2.internal
  <invalid>	<invalid>	1	{kubelet ip-172-31-9-165.ec2.internal}	spec.containers{git}	WarningFailed		Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/045e603cba13564651cf3b95f1689468c125a1dfcd023e6b7042ffb86d02fc7f: permission denied

  <invalid>	<invalid>	1	{kubelet ip-172-31-9-165.ec2.internal}		Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "git" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/045e603cba13564651cf3b95f1689468c125a1dfcd023e6b7042ffb86d02fc7f: permission denied\n"


Expected results:
Gitserver pod should be running.

Additional info:

Have added emptyDir in gitserver.yaml.

          volumeMounts:
          - mountPath: /var/lib/git/
            name: git
        volumes:
        - name: git
          emptyDir: {}
    triggers:
    - type: ConfigChange
Comment 1 Cesar Wong 2016-05-16 17:43:36 UTC 
The gitserver image currently declares 2 volumes:
1) /var/lib/origin (inherited from openshift/origin)
2) /var/lib/git

Currently, it's possible to get it running on openshift online by creating separate pvc's for them:

oc volume --add --type=pvc --name=git    --claim-size=1G --overwrite dc/git
oc volume --add --type=pvc --name=origin --claim-size=1G --mount-path=/var/lib/origin  dc/git

There's really no use for the origin volume, so I will change the parent image of the gitserver to be openshift/origin-base instead of openshift/origin. 

I will also state in the instructions that if you want to create the git server online, you must do it with a pvc claim... after running 'oc create -f gitserver.yaml', you'll need to run:

oc volume --add --type=pvc --name=git    --claim-size=1G --overwrite dc/git
Comment 2 XiuJuan Wang 2016-05-17 06:00:01 UTC 
envs:
dev-preview-stg (openshift master:v3.2.0.44)
dev-preview-int (openshift master:v3.2.0.40)

The workaround doesn't work.Still meet the "API error (500): mkdir /var/lib/docker/volumes/045e603cba13564651cf3b95f1689468c125a1dfcd023e6b7042ffb86d02fc7f: permission denied\n" error

$oc  get pvc 
NAME        STATUS    VOLUME         CAPACITY   ACCESSMODES   AGE
pvc-x9hh3   Bound     pv-aws-ujuze   1Gi        RWO           21m
$oc  get  dc git  -o yaml | grep pvc 
          claimName: pvc-x9hh3

$ oc  get pods 
NAME           READY     STATUS              RESTARTS   AGE
git-1-deploy   0/1       DeadlineExceeded    0          23m
git-2-deploy   0/1       DeadlineExceeded    0          21m
git-3-deploy   0/1       DeadlineExceeded    0          17m
git-4-deploy   0/1       Error               0          12m
git-5-78ttx    0/1       RunContainerError   0          2m
git-5-deploy   1/1       Running             0          2m
Comment 3 XiuJuan Wang 2016-05-17 07:48:28 UTC 
Sorry, should create pvc for /var/lib/origin too.Git server pod could be running now.
$oc create -f  gitserver.yaml
$oc volume --add --type=pvc --name=git    --claim-size=1G --overwrite dc/git
$oc volume --add --type=pvc --name=origin --claim-size=1G --mount-path=/var/lib/origin  dc/git
Comment 4 Cesar Wong 2016-05-17 13:25:50 UTC 
Fix in https://github.com/openshift/origin/pull/8902
Comment 5 XiuJuan Wang 2016-05-23 10:00:24 UTC 
@Cesar

Pod could be running in online envs with gitserver-persistent.yaml.

But online envs (dev-preview-stg and dev-preview-int) don't allow docker build.So will fail to create bc with docker strategy when push initial code to gitserver repo.


remote:     error: buildconfigs "ruby-hello-world" is forbidden: build strategy Docker is not allowed
remote:     deploymentconfig "ruby-hello-world" created
remote:     service "ruby-hello-world" created
remote: --> Failed
To http://git-xiuwang.b795.dev-preview-stg.openshiftapps.com/ruby-hello-world.git
 * [new branch]      master -> master

Can we change the docker build to s2i build for gitserver?
Comment 6 Cesar Wong 2016-05-25 20:33:04 UTC 
@XiuJuan I've submitted PR https://github.com/openshift/origin/pull/9031
which allows you to set an environment variable on the git server deployment that controls your default strategy.

For the online case, we would need to set it to 'docker'.
Comment 7 Cesar Wong 2016-05-25 20:41:45 UTC 
Sorry, for online, you need to set it to 'source'
Comment 8 XiuJuan Wang 2016-06-01 09:09:42 UTC 
@cewong After seting the env to source, the created app is still using dockerstrategy when push initial code to git server repo.

$ oc get dc  -o yaml | grep  -A 1 BUILD
          - name: BUILD_STRATEGY
            value: source


remote:     * An image stream will be created as "ruby-22-centos7:latest" that will track the source image
remote:     * A Docker build using source code from http://git:8080/ruby-hello-world.git will be created
remote:       * The resulting image will be pushed to image stream "ruby-hello-world:latest"
remote:       * Every time "ruby-22-centos7:latest" changes a new build will be triggered
remote:     * This image will be deployed in deployment config "ruby-hello-world"
remote:     * Port 8080 will be load balanced by service "ruby-hello-world"
remote:       * Other containers can access this service through the hostname "ruby-hello-world"
remote: 
remote: --> Creating resources with label app=ruby-hello-world ...
remote:     imagestream "ruby-22-centos7" created
remote:     imagestream "ruby-hello-world" created
remote:     error: buildconfigs "ruby-hello-world" is forbidden: build strategy Docker is not allowed

Delay so long time to check this,my bad
Comment 9 Cesar Wong 2016-06-01 12:37:24 UTC 
@XiuJuan it looks like the latest version of origin-gitserver has not been pushed to Docker hub. I pulled the openshift/origin-gitserver:latest and its /var/lib/git-hooks/post-receive doesn't include my latest change:
https://github.com/openshift/origin/blob/master/examples/gitserver/hooks/post-receive#L30

I will check later today to see if a new image is pushed that contains my change.
Comment 10 Cesar Wong 2016-06-02 03:00:34 UTC 
Looks like the gitserver image has been updated. Please try again with the latest image.
Comment 11 XiuJuan Wang 2016-06-02 03:34:54 UTC 
@cewong, works now.
You can move this bug to on-qa now.Thanks
Comment 12 XiuJuan Wang 2016-06-02 05:47:50 UTC 
template: https://raw.githubusercontent.com/openshift/origin/master/examples/gitserver/gitserver-persistent.yaml
Could create git server pod with above template when set BUILD_STRATEGY=source in online env.
Move this bug to verified.