Bug 1293252

Summary: Can not push images to docker-registry
Product: OpenShift Container Platform Reporter: Johnny Liu <jialiu>
Component: Image RegistryAssignee: Scott Dodson <sdodson>
Status: CLOSED ERRATA QA Contact: Wei Sun <wsun>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.1.0CC: aos-bugs, bleanhar, jokerman, sdodson
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-26 19:20:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Johnny Liu 2015-12-21 08:47:31 UTC 
Description of problem:
sti build failed due to unable to push image to docker-registry.

Version-Release number of selected component (if applicable):
atomic-openshift-3.1.1.0-1.git.0.8632732.el7aos.x86_64
AtomicOpenShift/3.1/2015-12-19.3
openshift3/ose-docker-registry:v3.1.1.0 (421ec8528377)

How reproducible:
Always

Steps to Reproduce:
1. Setup env and install docker-registry
2. Log in as a common user
3. New an app, trigger sti build automatically.
$ oc new-app https://github.com/openshift/simple-openshift-sinatra-sti.git -i openshift/ruby
$ oc build-logs simple-openshift-sinatra-sti-1

Actual results:
docker push failed.
$ oc build-logs simple-openshift-sinatra-sti-1
<--snip-->
I1221 03:06:58.061702       1 sti.go:315] Successfully built 172.30.53.111:5000/jialiu/simple-openshift-sinatra-sti:latest
I1221 03:06:58.118370       1 cleanup.go:23] Removing temporary directory /tmp/s2i-build912895855
I1221 03:06:58.118414       1 fs.go:117] Removing directory '/tmp/s2i-build912895855'
I1221 03:06:58.123525       1 cfg.go:44] Locating docker auth for image 172.30.53.111:5000/jialiu/simple-openshift-sinatra-sti:latest and type PUSH_DOCKERCFG_PATH
I1221 03:06:58.123561       1 cfg.go:80] Using Docker authentication configuration in '/var/run/secrets/openshift.io/push/.dockercfg'
I1221 03:06:58.123861       1 cfg.go:66] Using serviceaccount user for Docker authentication for image 172.30.53.111:5000/jialiu/simple-openshift-sinatra-sti:latest
I1221 03:06:58.123916       1 sti.go:214] Using provided push secret for pushing 172.30.53.111:5000/jialiu/simple-openshift-sinatra-sti:latest image
I1221 03:06:58.123936       1 sti.go:218] Pushing 172.30.53.111:5000/jialiu/simple-openshift-sinatra-sti:latest image ...
The push refers to a repository [172.30.53.111:5000/jialiu/simple-openshift-sinatra-sti] (len: 1)
Preparing
I1221 03:06:58.151209       1 sti.go:223] Registry server Address: 
I1221 03:06:58.151234       1 sti.go:224] Registry server User Name: serviceaccount
I1221 03:06:58.151244       1 sti.go:225] Registry server Email: serviceaccount
I1221 03:06:58.151252       1 sti.go:230] Registry server Password: <<non-empty>>
F1221 03:06:58.151268       1 builder.go:185] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error


docker-registry logs:
# oc logs docker-registry-1-ub5aa
<---snip-->
time="2015-12-21T03:06:58.132159515-05:00" level=debug msg="authorizing request" go.version=go1.4.2 http.request.host="172.30.53.111:5000" http.request.id=19cae056-05c3-4c84-a50c-dd7dbe0dc7c5 http.request.method=GET http.request.remoteaddr="192.168.0.213:55341" http.request.uri="/v2/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=b3c18885-5919-49df-bd1f-956b9b0f0f77 
time="2015-12-21T03:06:58.132290948-05:00" level=error msg="error authorizing context: authorization header with basic token required" go.version=go1.4.2 http.request.host="172.30.53.111:5000" http.request.id=19cae056-05c3-4c84-a50c-dd7dbe0dc7c5 http.request.method=GET http.request.remoteaddr="192.168.0.213:55341" http.request.uri="/v2/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=b3c18885-5919-49df-bd1f-956b9b0f0f77 
192.168.0.213 - - [21/Dec/2015:03:06:58 -0500] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64"
time="2015-12-21T03:06:58.140142967-05:00" level=debug msg="authorizing request" go.version=go1.4.2 http.request.host="172.30.53.111:5000" http.request.id=09fc00f7-89d6-45b1-bcd1-1eabb8b1076e http.request.method=POST http.request.remoteaddr="192.168.0.213:55342" http.request.uri="/v2/jialiu/simple-openshift-sinatra-sti/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=b3c18885-5919-49df-bd1f-956b9b0f0f77 vars.name="jialiu/simple-openshift-sinatra-sti" 
time="2015-12-21T03:06:58.140311867-05:00" level=debug msg="Origin auth: checking for access to repository:jialiu/simple-openshift-sinatra-sti:pull" go.version=go1.4.2 http.request.host="172.30.53.111:5000" http.request.id=09fc00f7-89d6-45b1-bcd1-1eabb8b1076e http.request.method=POST http.request.remoteaddr="192.168.0.213:55342" http.request.uri="/v2/jialiu/simple-openshift-sinatra-sti/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=b3c18885-5919-49df-bd1f-956b9b0f0f77 vars.name="jialiu/simple-openshift-sinatra-sti" 
time="2015-12-21T03:06:58.145518341-05:00" level=debug msg="Origin auth: checking for access to repository:jialiu/simple-openshift-sinatra-sti:push" go.version=go1.4.2 http.request.host="172.30.53.111:5000" http.request.id=09fc00f7-89d6-45b1-bcd1-1eabb8b1076e http.request.method=POST http.request.remoteaddr="192.168.0.213:55342" http.request.uri="/v2/jialiu/simple-openshift-sinatra-sti/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=b3c18885-5919-49df-bd1f-956b9b0f0f77 vars.name="jialiu/simple-openshift-sinatra-sti" 
time="2015-12-21T03:06:58.150431152-05:00" level=error msg="error initializing repository middleware: DOCKER_REGISTRY_URL is required" go.version=go1.4.2 http.request.host="172.30.53.111:5000" http.request.id=09fc00f7-89d6-45b1-bcd1-1eabb8b1076e http.request.method=POST http.request.remoteaddr="192.168.0.213:55342" http.request.uri="/v2/jialiu/simple-openshift-sinatra-sti/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64" instance.id=b3c18885-5919-49df-bd1f-956b9b0f0f77 vars.name="jialiu/simple-openshift-sinatra-sti" 
192.168.0.213 - - [21/Dec/2015:03:06:58 -0500] "POST /v2/jialiu/simple-openshift-sinatra-sti/blobs/uploads/ HTTP/1.1" 500 70 "" "docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.3.1.el7.x86_64 os/linux arch/amd64"
<---snip-->

Expected results:
Sti build push images successfully.


Additional info:
This issue does NOT happen on released version - 3.1.0.4
Comment 2 Michal Minar 2015-12-21 16:35:13 UTC 
Dockerfile in our dist-gits need to be modified to reflect upstream changes. Rename of REGISTRY_URL environment variable to DOCKER_REGISTRY_URL in particular [1].

[1] https://github.com/openshift/origin/commit/bb6781fab81cb123035b5fa7607359f32ddbd74c
Comment 4 Johnny Liu 2015-12-22 02:18:28 UTC 
Verified this bug with the latest openshift3/ose-docker-registry (21b8d7a085c2) image, and PASS.


data is pushed successfully.
Comment 6 errata-xmlrpc 2016-01-26 19:20:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070