Bug 1370045 - Can' t push image to registry 2.4.1 with gcs backend
Summary: Can' t push image to registry 2.4.1 with gcs backend
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Alexey Gladkov
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-25 07:09 UTC by zhou ying
Modified: 2016-09-27 09:45 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-27 09:45:57 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC
Github openshift origin pull 10640 'None' 'closed' 'BZ 1370045: Fix pushing to GCS storage' 2019-11-27 17:41:16 UTC

Description zhou ying 2016-08-25 07:09:49 UTC
Description of problem:
When push image to integrade docker-registry which with  GCS backend, always met error: build error: Failed to push image: blob upload unknown

Version-Release number of selected component (if applicable):
openshift v3.3.0.25+d2ac65e-dirty
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git
docker-registry:version=v2.4.1
docker Version: docker-1.10.3-46.el7.11.x86_64

How reproducible:
always 

Steps to Reproduce:
1. Setup OpenShift on GCE and registry with gcs backend;
#oc rsh docker-registry-6-wewpw
sh-4.2$ cat /etc/registryconfig/config.yml 
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    layerinfo: inmemory
  gcs:
    bucket: ose-qe-docker-registry
    keyfile: /etc/registryconfig/key.json
    rootdirectory: /registry-20160825013039
auth:
  openshift:
    realm: openshift
middleware:
  repository:
    - name: openshift
      options:
        pullthrough: true
  registry:
     - name: openshift
  storage:
     - name: openshift
2. Login OpenShift and build images

Actual results:
2. Build always failed with error: build error: Failed to push image: blob upload unknown

logs from registry:
#oc build-logs cakephp-example-1 -n install-test
Pushing image 172.30.36.227:5000/install-test/cakephp-example:latest ...
Pushed 0/4 layers, 3% complete
Pushed 0/4 layers, 38% complete
Pushed 0/4 layers, 73% complete
Pushed 0/4 layers, 87% complete
Pushed 0/4 layers, 23% complete
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount@example.org
Registry server Password: <<non-empty>>
error: build error: Failed to push image: blob upload unknown

time="2016-08-25T02:15:53.034160745-04:00" level=info msg="upload resumed at wrong offest: 1955 != 0" go.version=go1.6.2 http.request.host="172.30.176.172:5000" http.request.id=55328835-60df-4af7-8711-5900ac0c4028 http.request.method=PUT http.request.remoteaddr="10.1.1.1:56366" http.request.uri="/v2/zhouy/ruby-ex/blobs/uploads/eaaeb57f-309f-4cbf-8884-35d9122055e3?_state=O9aqc7CGTX2pozAKu_aaI4H0vGjVQD2qfZEU9lLpDCV7Ik5hbWUiOiJ6aG91eS9ydWJ5LWV4IiwiVVVJRCI6ImVhYWViNTdmLTMwOWYtNGNiZi04ODg0LTM1ZDkxMjIwNTVlMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOC0yNVQwNjoxNTo0OFoifQ%3D%3D&digest=sha256%3A4fbe156e6dc8d00f242f536fe56bb1aec1d35116ef7c41e2970008c7e04af30a" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/8491c12-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=2cf11f15-8611-4f6a-a25f-e9dc0bb0fe26 vars.name="zhouy/ruby-ex" vars.uuid=eaaeb57f-309f-4cbf-8884-35d9122055e3
time="2016-08-25T02:15:53.407949705-04:00" level=debug msg="could not stat layer link \"sha256:3d8673bd162aafaf70e94029801a409fa3210363707ea1e7956b6f8ff9351cc8\" in repository \"zhouy/ruby-ex\": unknown blob" go.version=go1.6.2 http.request.host="172.30.176.172:5000" http.request.id=18a3c7af-0fa2-4144-a7bc-44f130b20588 http.request.method=HEAD http.request.remoteaddr="10.1.1.1:56368" http.request.uri="/v2/zhouy/ruby-ex/blobs/sha256:3d8673bd162aafaf70e94029801a409fa3210363707ea1e7956b6f8ff9351cc8" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/8491c12-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=2cf11f15-8611-4f6a-a25f-e9dc0bb0fe26 vars.digest="sha256:3d8673bd162aafaf70e94029801a409fa3210363707ea1e7956b6f8ff9351cc8" vars.name="zhouy/ruby-ex" 
time="2016-08-25T02:15:54.167328431-04:00" level=error msg="response completed with error" err.code=unknown err.detail="unexpected EOF" err.message="unknown error" go.version=go1.6.2 http.request.host="172.30.176.172:5000" http.request.id=ccefc4b5-4b7d-458b-82e8-af48d36040d1 http.request.method=PATCH http.request.remoteaddr="10.1.1.1:56352" http.request.uri="/v2/zhouy/ruby-ex/blobs/uploads/b59de20b-17be-4302-a41e-a3441f66637c?_state=AdDH-6pXN1_OVO92gCSihVXDaksmvrug5gZFjIyD1Ad7Ik5hbWUiOiJ6aG91eS9ydWJ5LWV4IiwiVVVJRCI6ImI1OWRlMjBiLTE3YmUtNDMwMi1hNDFlLWEzNDQxZjY2NjM3YyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOC0yNVQwNjoxNTo0OC4yODgyODU2MjJaIn0%3D" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/8491c12-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=5.050923305s http.response.status=500 http.response.written=70 instance.id=2cf11f15-8611-4f6a-a25f-e9dc0bb0fe26 vars.name="zhouy/ruby-ex" vars.uuid=b59de20b-17be-4302-a41e-a3441f66637c 
10.1.1.1 - - [25/Aug/2016:02:15:54 -0400] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1"
10.1.1.1 - - [25/Aug/2016:02:15:54 -0400] "GET /healthz HTTP/1.1" 200 0 "" "Go-http-client/1.1"
time="2016-08-25T02:15:54.444840834-04:00" level=debug msg="gcs.PutContent(\"/docker/registry/v2/repositories/zhouy/ruby-ex/_uploads/c00226a0-12b2-4684-9296-77e0c2c1efb3/hashstates/sha256/0\")" go.version=go1.6.2 http.request.host="172.30.176.172:5000" http.request.id=77218122-05a3-4355-a342-3539cd8a76be http.request.method=POST http.request.remoteaddr="10.1.1.1:56374" http.request.uri="/v2/zhouy/ruby-ex/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/8491c12-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" instance.id=2cf11f15-8611-4f6a-a25f-e9dc0bb0fe26 trace.duration=296.636854ms trace.file="/builddir/build/BUILD/atomic-openshift-git-0.9a704ee/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=3f57882a-231e-46e5-bcf0-2679468bbe1c trace.line=95 vars.name="zhouy/ruby-ex" 
10.1.1.1 - - [25/Aug/2016:02:15:48 -0400] "PATCH /v2/zhouy/ruby-ex/blobs/uploads/89afbfee-edab-443d-b07f-50961a51606f?_state=UbYGLnxNMPJT6ukMG9vHkQEk8x1RrVyqv3TJsxE5Brd7Ik5hbWUiOiJ6aG91eS9ydWJ5LWV4IiwiVVVJRCI6Ijg5YWZiZmVlLWVkYWItNDQzZC1iMDdmLTUwOTYxYTUxNjA2ZiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOC0yNVQwNjoxNTo0OC4wODMwNTY1MTZaIn0%3D HTTP/1.1" 500 70 "" "docker/1.10.3 go/go1.6.2 git-commit/8491c12-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
time="2016-08-25T02:15:54.680856575-04:00" level=error msg="response completed with error" err.code=unknown err.detail="unexpected EOF" err.message="unknown error" go.version=go1.6.2 http.request.host="172.30.176.172:5000" http.request.id=e1ee51f5-c078-4934-8da2-53f268c31bd7 http.request.method=PATCH http.request.remoteaddr="10.1.1.1:56348" http.request.uri="/v2/zhouy/ruby-ex/blobs/uploads/89afbfee-edab-443d-b07f-50961a51606f?_state=UbYGLnxNMPJT6ukMG9vHkQEk8x1RrVyqv3TJsxE5Brd7Ik5hbWUiOiJ6aG91eS9ydWJ5LWV4IiwiVVVJRCI6Ijg5YWZiZmVlLWVkYWItNDQzZC1iMDdmLTUwOTYxYTUxNjA2ZiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNi0wOC0yNVQwNjoxNTo0OC4wODMwNTY1MTZaIn0%3D" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/8491c12-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=5.720544956s http.response.status=500 http.response.written=70 instance.id=2cf11f15-8611-4f6a-a25f-e9dc0bb0fe26 vars.name="zhouy/ruby-ex" vars.uuid=89afbfee-edab-443d-b07f-50961a51606f 
10.1.1.1 - - [25/Aug/2016:02:15:53 -0400] "POST /v2/zhouy/ruby-ex/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/1.10.3 go/go1.6.2 git-commit/8491c12-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64"
time="2016-08-25T02:15:54.699310283-04:00" level=info msg="response completed" go.version=go1.6.2 http.request.host="172.30.176.172:5000" http.request.id=77218122-05a3-4355-a342-3539cd8a76be http.request.method=POST http.request.remoteaddr="10.1.1.1:56374" http.request.uri="/v2/zhouy/ruby-ex/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/8491c12-unsupported kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" http.response.duration=838.770665ms http.response.status=202 http.response.written=0 instance.id=2cf11f15-8611-4f6a-a25f-e9dc0bb0fe26

Expected results:
2. Push image succeed.

Additional info:
after change to local filesystem instead of GCS, we can push images succeed

Comment 1 Michal Fojtik 2016-08-25 09:37:24 UTC
Clayton, any idea what can possibly cause this?

Comment 4 zhou ying 2016-08-26 00:59:01 UTC
Michal:
  
  If needed , I could setup an env to reproduce this issue.

Comment 5 Michal Fojtik 2016-08-26 08:28:43 UTC
The fix has been merged to origin, we need to re-test in the latest puddle.

Comment 7 zhou ying 2016-08-29 03:07:23 UTC
Confirmed with latest puddle, the issue has fixed:
openshift version
openshift v3.3.0.26
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git

Comment 9 errata-xmlrpc 2016-09-27 09:45:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933


Note You need to log in before you can comment on or make changes to this bug.