Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1336318 - [online]Failed to create gitserver pod
Summary: [online]Failed to create gitserver pod
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Build
Version: 3.x
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Cesar Wong
QA Contact: Wenjing Zheng
Depends On:
TreeView+ depends on / blocked
Reported: 2016-05-16 07:28 UTC by XiuJuan Wang
Modified: 2016-06-23 17:31 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2016-06-23 17:31:43 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description XiuJuan Wang 2016-05-16 07:28:46 UTC
Description of problem:
Due to 'Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/e502a13b9ce0e665bf77b912fb0a4d7fc46fd83ad8519858b7fc97f3bebceb7e: permission denied' error, can't create git server pod

Works before in int env(older than openshift master:v3.2.0.40)

Version-Release number of selected component (if applicable):
dev-preview-stg (openshift master:v3.2.0.44)

openshift/origin-gitserver (imageid:28042b7dc0fd)
How reproducible:

Steps to Reproduce:
1.Create the Git Server
$oc create  -f https://raw.githubusercontent.com/openshift/origin/master/examples/gitserver/gitserver.yaml
2.Grant edit access to the git service account
$oc policy add-role-to-user openshift-online:edit -z git 

3.Check pod
# oc  get  pods 
NAME           READY     STATUS              RESTARTS   AGE
git-1-deploy   0/1       Error               0          31m
git-2-deploy   1/1       Running             0          <invalid>
git-2-w5dfg    0/1       RunContainerError   0          <invalid>

Actual results:

  FirstSeen	LastSeen	Count	From					SubobjectPath		Type	Reason		Message
  ---------	--------	-----	----					-------------		--------------		-------
  <invalid>	<invalid>	1	{default-scheduler }						Normal	Scheduled	Successfully assigned git-2-w5dfg to ip-172-31-9-165.ec2.internal
  <invalid>	<invalid>	1	{kubelet ip-172-31-9-165.ec2.internal}	spec.containers{git}	WarningFailed		Failed to create docker container with error: API error (500): mkdir /var/lib/docker/volumes/045e603cba13564651cf3b95f1689468c125a1dfcd023e6b7042ffb86d02fc7f: permission denied

  <invalid>	<invalid>	1	{kubelet ip-172-31-9-165.ec2.internal}		Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "git" with RunContainerError: "runContainer: API error (500): mkdir /var/lib/docker/volumes/045e603cba13564651cf3b95f1689468c125a1dfcd023e6b7042ffb86d02fc7f: permission denied\n"

Expected results:
Gitserver pod should be running.

Additional info:

Have added emptyDir in gitserver.yaml.

          - mountPath: /var/lib/git/
            name: git
        - name: git
          emptyDir: {}
    - type: ConfigChange

Comment 1 Cesar Wong 2016-05-16 17:43:36 UTC
The gitserver image currently declares 2 volumes:
1) /var/lib/origin (inherited from openshift/origin)
2) /var/lib/git

Currently, it's possible to get it running on openshift online by creating separate pvc's for them:

oc volume --add --type=pvc --name=git    --claim-size=1G --overwrite dc/git
oc volume --add --type=pvc --name=origin --claim-size=1G --mount-path=/var/lib/origin  dc/git

There's really no use for the origin volume, so I will change the parent image of the gitserver to be openshift/origin-base instead of openshift/origin. 

I will also state in the instructions that if you want to create the git server online, you must do it with a pvc claim... after running 'oc create -f gitserver.yaml', you'll need to run:

oc volume --add --type=pvc --name=git    --claim-size=1G --overwrite dc/git

Comment 2 XiuJuan Wang 2016-05-17 06:00:01 UTC
dev-preview-stg (openshift master:v3.2.0.44)
dev-preview-int (openshift master:v3.2.0.40)

The workaround doesn't work.Still meet the "API error (500): mkdir /var/lib/docker/volumes/045e603cba13564651cf3b95f1689468c125a1dfcd023e6b7042ffb86d02fc7f: permission denied\n" error

$oc  get pvc 
pvc-x9hh3   Bound     pv-aws-ujuze   1Gi        RWO           21m
$oc  get  dc git  -o yaml | grep pvc 
          claimName: pvc-x9hh3

$ oc  get pods 
NAME           READY     STATUS              RESTARTS   AGE
git-1-deploy   0/1       DeadlineExceeded    0          23m
git-2-deploy   0/1       DeadlineExceeded    0          21m
git-3-deploy   0/1       DeadlineExceeded    0          17m
git-4-deploy   0/1       Error               0          12m
git-5-78ttx    0/1       RunContainerError   0          2m
git-5-deploy   1/1       Running             0          2m

Comment 3 XiuJuan Wang 2016-05-17 07:48:28 UTC
Sorry, should create pvc for /var/lib/origin too.Git server pod could be running now.
$oc create -f  gitserver.yaml
$oc volume --add --type=pvc --name=git    --claim-size=1G --overwrite dc/git
$oc volume --add --type=pvc --name=origin --claim-size=1G --mount-path=/var/lib/origin  dc/git

Comment 4 Cesar Wong 2016-05-17 13:25:50 UTC
Fix in https://github.com/openshift/origin/pull/8902

Comment 5 XiuJuan Wang 2016-05-23 10:00:24 UTC

Pod could be running in online envs with gitserver-persistent.yaml.

But online envs (dev-preview-stg and dev-preview-int) don't allow docker build.So will fail to create bc with docker strategy when push initial code to gitserver repo.

remote:     error: buildconfigs "ruby-hello-world" is forbidden: build strategy Docker is not allowed
remote:     deploymentconfig "ruby-hello-world" created
remote:     service "ruby-hello-world" created
remote: --> Failed
To http://git-xiuwang.b795.dev-preview-stg.openshiftapps.com/ruby-hello-world.git
 * [new branch]      master -> master

Can we change the docker build to s2i build for gitserver?

Comment 6 Cesar Wong 2016-05-25 20:33:04 UTC
@XiuJuan I've submitted PR https://github.com/openshift/origin/pull/9031
which allows you to set an environment variable on the git server deployment that controls your default strategy.

For the online case, we would need to set it to 'docker'.

Comment 7 Cesar Wong 2016-05-25 20:41:45 UTC
Sorry, for online, you need to set it to 'source'

Comment 8 XiuJuan Wang 2016-06-01 09:09:42 UTC
@cewong After seting the env to source, the created app is still using dockerstrategy when push initial code to git server repo.

$ oc get dc  -o yaml | grep  -A 1 BUILD
          - name: BUILD_STRATEGY
            value: source

remote:     * An image stream will be created as "ruby-22-centos7:latest" that will track the source image
remote:     * A Docker build using source code from http://git:8080/ruby-hello-world.git will be created
remote:       * The resulting image will be pushed to image stream "ruby-hello-world:latest"
remote:       * Every time "ruby-22-centos7:latest" changes a new build will be triggered
remote:     * This image will be deployed in deployment config "ruby-hello-world"
remote:     * Port 8080 will be load balanced by service "ruby-hello-world"
remote:       * Other containers can access this service through the hostname "ruby-hello-world"
remote: --> Creating resources with label app=ruby-hello-world ...
remote:     imagestream "ruby-22-centos7" created
remote:     imagestream "ruby-hello-world" created
remote:     error: buildconfigs "ruby-hello-world" is forbidden: build strategy Docker is not allowed

Delay so long time to check this,my bad

Comment 9 Cesar Wong 2016-06-01 12:37:24 UTC
@XiuJuan it looks like the latest version of origin-gitserver has not been pushed to Docker hub. I pulled the openshift/origin-gitserver:latest and its /var/lib/git-hooks/post-receive doesn't include my latest change:

I will check later today to see if a new image is pushed that contains my change.

Comment 10 Cesar Wong 2016-06-02 03:00:34 UTC
Looks like the gitserver image has been updated. Please try again with the latest image.

Comment 11 XiuJuan Wang 2016-06-02 03:34:54 UTC
@cewong, works now.
You can move this bug to on-qa now.Thanks

Comment 12 XiuJuan Wang 2016-06-02 05:47:50 UTC
template: https://raw.githubusercontent.com/openshift/origin/master/examples/gitserver/gitserver-persistent.yaml
Could create git server pod with above template when set BUILD_STRATEGY=source in online env.
Move this bug to verified.

Note You need to log in before you can comment on or make changes to this bug.