Description of problem: When push image to docker-registry which with swift as backend, sometimes the push will fail. Version-Release number of selected component (if applicable): openshift version openshift v3.4.0.38 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 ose-docker-registry:v3.4.0.38 62c83a5aa335 docker Version: docker-common-1.12.5-7.el7.x86_64 How reproducible: 20% Steps to Reproduce: 1. Setup registry with swift as backend; 2. Login OpenShift and create project; 3. Tag and push image to docker-registry `docker tag bmeng/caddy-docker docker-registry-default.1221-gfx.qe.rhcloud.com/zhouyt/caddy:latest` `docker push docker-registry-default.1221-gfx.qe.rhcloud.com/zhouyt/caddy` The push refers to a repository [docker-registry-default.1221-gfx.qe.rhcloud.com/zhouyt/caddy] (len: 1) 9f7e1bdd68f9: Pushed 50d015339b28: Pushed eedc425d4821: Pushed 1f98fc7a5558: Pushed 245b955bb8ef: Pushed 4d3d1306f0ec: Pushed f1d1e61c29f8: Pushed 17529d4bbbc9: Pushing [===========================> ] 11.98 MB/21.59 MB Received unexpected HTTP status: 500 Internal Server Error Actual results: 3. Sometimes the push failed , the reigstry has logs: time="2016-12-21T06:25:59.334080359Z" level=error msg="unable to delete layer upload resources \"/docker/registry/v2/repositories/zhouyt/caddy/_uploads/ca914a82-d42e-4b96-9fc1-70efb2ccbfb9\": swift: HTTP Error: 503: 503 Service Unavailable" go.version=go1.7.4 http.request.host=docker-registry-default.1221-gfx.qe.rhcloud.com http.request.id=6fa109c0-6471-4a02-87c0-75c0c5378db8 http.request.method=PUT http.request.remoteaddr="10.128.0.1:56750" http.request.uri="/v2/zhouyt/caddy/blobs/uploads/ca914a82-d42e-4b96-9fc1-70efb2ccbfb9?_state=8GL_Fc00Na3OVPigaZsSDYf0k528f5WGAW7XoXAM7wh7Ik5hbWUiOiJ6aG91eXQvY2FkZHkiLCJVVUlEIjoiY2E5MTRhODItZDQyZS00Yjk2LTlmYzEtNzBlZmIyY2NiZmI5IiwiT2Zmc2V0IjoyNDkyODcwLCJTdGFydGVkQXQiOiIyMDE2LTEyLTIxVDA2OjI1OjQyWiJ9&digest=sha256%3Abfc185be0245882f955f50de01e4702b286d1e8f35e73182bef075233a7fddae" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/4.0.4-301.fc22.x86_64 os/linux arch/amd64" instance.id=54707cec-7767-49e9-871f-ed4f1f9dc87a vars.name="zhouyt/caddy" vars.uuid=ca914a82-d42e-4b96-9fc1-70efb2ccbfb9 time="2016-12-21T06:25:59.334161493Z" level=error msg="unknown error completing upload: driver.Error{DriverName:\"swift\", Enclosed:(*swift.Error)(0xc420889b80)}" go.version=go1.7.4 http.request.host=docker-registry-default.1221-gfx.qe.rhcloud.com http.request.id=6fa109c0-6471-4a02-87c0-75c0c5378db8 http.request.method=PUT http.request.remoteaddr="10.128.0.1:56750" http.request.uri="/v2/zhouyt/caddy/blobs/uploads/ca914a82-d42e-4b96-9fc1-70efb2ccbfb9?_state=8GL_Fc00Na3OVPigaZsSDYf0k528f5WGAW7XoXAM7wh7Ik5hbWUiOiJ6aG91eXQvY2FkZHkiLCJVVUlEIjoiY2E5MTRhODItZDQyZS00Yjk2LTlmYzEtNzBlZmIyY2NiZmI5IiwiT2Zmc2V0IjoyNDkyODcwLCJTdGFydGVkQXQiOiIyMDE2LTEyLTIxVDA2OjI1OjQyWiJ9&digest=sha256%3Abfc185be0245882f955f50de01e4702b286d1e8f35e73182bef075233a7fddae" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/4.0.4-301.fc22.x86_64 os/linux arch/amd64" instance.id=54707cec-7767-49e9-871f-ed4f1f9dc87a vars.name="zhouyt/caddy" vars.uuid=ca914a82-d42e-4b96-9fc1-70efb2ccbfb9 time="2016-12-21T06:25:59.334245835Z" level=debug msg="(*blobWriter).Cancel" go.version=go1.7.4 http.request.host=docker-registry-default.1221-gfx.qe.rhcloud.com http.request.id=6fa109c0-6471-4a02-87c0-75c0c5378db8 http.request.method=PUT http.request.remoteaddr="10.128.0.1:56750" http.request.uri="/v2/zhouyt/caddy/blobs/uploads/ca914a82-d42e-4b96-9fc1-70efb2ccbfb9?_state=8GL_Fc00Na3OVPigaZsSDYf0k528f5WGAW7XoXAM7wh7Ik5hbWUiOiJ6aG91eXQvY2FkZHkiLCJVVUlEIjoiY2E5MTRhODItZDQyZS00Yjk2LTlmYzEtNzBlZmIyY2NiZmI5IiwiT2Zmc2V0IjoyNDkyODcwLCJTdGFydGVkQXQiOiIyMDE2LTEyLTIxVDA2OjI1OjQyWiJ9&digest=sha256%3Abfc185be0245882f955f50de01e4702b286d1e8f35e73182bef075233a7fddae" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/4.0.4-301.fc22.x86_64 os/linux arch/amd64" instance.id=54707cec-7767-49e9-871f-ed4f1f9dc87a vars.name="zhouyt/caddy" vars.uuid=ca914a82-d42e-4b96-9fc1-70efb2ccbfb9 time="2016-12-21T06:25:59.334294113Z" level=error msg="error canceling upload after error: already closed" go.version=go1.7.4 http.request.host=docker-registry-default.1221-gfx.qe.rhcloud.com http.request.id=6fa109c0-6471-4a02-87c0-75c0c5378db8 http.request.method=PUT http.request.remoteaddr="10.128.0.1:56750" http.request.uri="/v2/zhouyt/caddy/blobs/uploads/ca914a82-d42e-4b96-9fc1-70efb2ccbfb9?_state=8GL_Fc00Na3OVPigaZsSDYf0k528f5WGAW7XoXAM7wh7Ik5hbWUiOiJ6aG91eXQvY2FkZHkiLCJVVUlEIjoiY2E5MTRhODItZDQyZS00Yjk2LTlmYzEtNzBlZmIyY2NiZmI5IiwiT2Zmc2V0IjoyNDkyODcwLCJTdGFydGVkQXQiOiIyMDE2LTEyLTIxVDA2OjI1OjQyWiJ9&digest=sha256%3Abfc185be0245882f955f50de01e4702b286d1e8f35e73182bef075233a7fddae" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/4.0.4-301.fc22.x86_64 os/linux arch/amd64" instance.id=54707cec-7767-49e9-871f-ed4f1f9dc87a vars.name="zhouyt/caddy" vars.uuid=ca914a82-d42e-4b96-9fc1-70efb2ccbfb9 time="2016-12-21T06:26:03.058030698Z" level=debug msg="swift.PutContent(\"/docker/registry/v2/repositories/zhouyt/caddy/_uploads/ca914a82-d42e-4b96-9fc1-70efb2ccbfb9/hashstates/sha256/2492870\")" go.version=go1.7.4 http.request.host=docker-registry-default.1221-gfx.qe.rhcloud.com http.request.id=6fa109c0-6471-4a02-87c0-75c0c5378db8 http.request.method=PUT http.request.remoteaddr="10.128.0.1:56750" http.request.uri="/v2/zhouyt/caddy/blobs/uploads/ca914a82-d42e-4b96-9fc1-70efb2ccbfb9?_state=8GL_Fc00Na3OVPigaZsSDYf0k528f5WGAW7XoXAM7wh7Ik5hbWUiOiJ6aG91eXQvY2FkZHkiLCJVVUlEIjoiY2E5MTRhODItZDQyZS00Yjk2LTlmYzEtNzBlZmIyY2NiZmI5IiwiT2Zmc2V0IjoyNDkyODcwLCJTdGFydGVkQXQiOiIyMDE2LTEyLTIxVDA2OjI1OjQyWiJ9&digest=sha256%3Abfc185be0245882f955f50de01e4702b286d1e8f35e73182bef075233a7fddae" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/4.0.4-301.fc22.x86_64 os/linux arch/amd64" instance.id=54707cec-7767-49e9-871f-ed4f1f9dc87a trace.duration=3.723570127s trace.file="/builddir/build/BUILD/atomic-openshift-git-0.8561cba/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=d0394f3b-ab01-41af-802d-21d55a262c6b trace.line=95 vars.name="zhouyt/caddy" vars.uuid=ca914a82-d42e-4b96-9fc1-70efb2ccbfb9 time="2016-12-21T06:26:03.058425739Z" level=error msg="response completed with error" err.code=unknown err.detail="swift: HTTP Error: 503: 503 Service Unavailable" err.message="unknown error" go.version=go1.7.4 http.request.host=docker-registry-default.1221-gfx.qe.rhcloud.com http.request.id=6fa109c0-6471-4a02-87c0-75c0c5378db8 http.request.method=PUT http.request.remoteaddr="10.128.0.1:56750" http.request.uri="/v2/zhouyt/caddy/blobs/uploads/ca914a82-d42e-4b96-9fc1-70efb2ccbfb9?_state=8GL_Fc00Na3OVPigaZsSDYf0k528f5WGAW7XoXAM7wh7Ik5hbWUiOiJ6aG91eXQvY2FkZHkiLCJVVUlEIjoiY2E5MTRhODItZDQyZS00Yjk2LTlmYzEtNzBlZmIyY2NiZmI5IiwiT2Zmc2V0IjoyNDkyODcwLCJTdGFydGVkQXQiOiIyMDE2LTEyLTIxVDA2OjI1OjQyWiJ9&digest=sha256%3Abfc185be0245882f955f50de01e4702b286d1e8f35e73182bef075233a7fddae" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/4.0.4-301.fc22.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=14.027557529s http.response.status=500 http.response.written=170 instance.id=54707cec-7767-49e9-871f-ed4f1f9dc87a vars.name="zhouyt/caddy" vars.uuid=ca914a82-d42e-4b96-9fc1-70efb2ccbfb9 Expected results: 3, All the push should succeed Additional info:
Today push image to docker-registry failed again, the backend is gcs, the reproducible ratio is about 10%. The env info: openshift version openshift v3.4.0.38 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 ose-docker-registry:v3.4.0.38 docker Version: docker-common-1.12.5-8.el7.x86_64 The logs from docker-registry: time="2017-01-06T09:03:06.998574658Z" level=debug msg="gcs.GetContent(\"/docker/registry/v2/repositories/dyan-test/nodejs-mongodb-example/_layers/sha256/8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636/link\")" go.version=go1.7.4 http.request.host="172.30.77.6:5000" http.request.id=1fc32b65-118a-4eff-8c33-5cc1d07bc5e1 http.request.method=HEAD http.request.remoteaddr="10.129.0.1:35156" http.request.uri="/v2/dyan-test/nodejs-mongodb-example/blobs/sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636" http.request.useragent="docker/1.12.5 go/go1.7.4 git-commit/1d8f205 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(go-dockerclient)" instance.id=9fe1098d-7db3-476a-86cf-06617c98b287 trace.duration=92.101775ms trace.file="/builddir/build/BUILD/atomic-openshift-git-0.8561cba/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=9dd7d91f-8ffb-43cc-b1d1-cb1b5955002d trace.line=82 vars.digest="sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636" vars.name="dyan-test/nodejs-mongodb-example" time="2017-01-06T09:03:06.998705762Z" level=debug msg="could not stat layer link \"sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636\" in repository \"dyan-test/nodejs-mongodb-example\": unknown blob" go.version=go1.7.4 http.request.host="172.30.77.6:5000" http.request.id=1fc32b65-118a-4eff-8c33-5cc1d07bc5e1 http.request.method=HEAD http.request.remoteaddr="10.129.0.1:35156" http.request.uri="/v2/dyan-test/nodejs-mongodb-example/blobs/sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636" http.request.useragent="docker/1.12.5 go/go1.7.4 git-commit/1d8f205 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(go-dockerclient)" instance.id=9fe1098d-7db3-476a-86cf-06617c98b287 vars.digest="sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636" vars.name="dyan-test/nodejs-mongodb-example" time="2017-01-06T09:03:07.119679473Z" level=debug msg="gcs.Stat(\"/docker/registry/v2/blobs/sha256/8f/8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636/data\")" go.version=go1.7.4 http.request.host="172.30.77.6:5000" http.request.id=1fc32b65-118a-4eff-8c33-5cc1d07bc5e1 http.request.method=HEAD http.request.remoteaddr="10.129.0.1:35156" http.request.uri="/v2/dyan-test/nodejs-mongodb-example/blobs/sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636" http.request.useragent="docker/1.12.5 go/go1.7.4 git-commit/1d8f205 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(go-dockerclient)" instance.id=9fe1098d-7db3-476a-86cf-06617c98b287 trace.duration=120.849505ms trace.file="/builddir/build/BUILD/atomic-openshift-git-0.8561cba/_output/local/go/src/github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/openshift/origin/vendor/github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat" trace.id=04e4eb1d-7fc1-49c0-b2a8-6d7dcc784fbd trace.line=137 vars.digest="sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636" vars.name="dyan-test/nodejs-mongodb-example" time="2017-01-06T09:03:07.134620077Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636 err.message="blob unknown to registry" go.version=go1.7.4 http.request.host="172.30.77.6:5000" http.request.id=1fc32b65-118a-4eff-8c33-5cc1d07bc5e1 http.request.method=HEAD http.request.remoteaddr="10.129.0.1:35156" http.request.uri="/v2/dyan-test/nodejs-mongodb-example/blobs/sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636" http.request.useragent="docker/1.12.5 go/go1.7.4 git-commit/1d8f205 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(go-dockerclient)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=235.529777ms http.response.status=404 http.response.written=157 instance.id=9fe1098d-7db3-476a-86cf-06617c98b287 vars.digest="sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636" vars.name="dyan-test/nodejs-mongodb-example" 10.129.0.1 - - [06/Jan/2017:09:03:06 +0000] "HEAD /v2/dyan-test/nodejs-mongodb-example/blobs/sha256:8ffaa6f9bd8cd084bcbf7ea04e3a40f727dfb91afe6a9cc391023bbf7f678636 HTTP/1.1" 404 157 "" "docker/1.12.5 go/go1.7.4 git-commit/1d8f205 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(go-dockerclient)"
Can you please re-test with https://bugzilla.redhat.com/show_bug.cgi?id=1408661 fix applied? (or confirm this also happen on master).
time="2016-12-21T06:25:59.334161493Z" level=error msg="unknown error completing upload: driver.Error{DriverName:\"swift\", Enclosed:(*swift.Error)(0xc420889b80)}" go.version=go1.7.4 http.request.host=docker-registry-default.1221-gfx.qe.rhcloud.com http.request.id=6fa109c0-6471-4a02-87c0-75c0c5378db8 http.request.method=PUT http.request.remoteaddr="10.128.0.1:56750" http.request.uri="/v2/zhouyt/caddy/blobs/uploads/ca914a82-d42e-4b96-9fc1-70efb2ccbfb9?_state=8GL_Fc00Na3OVPigaZsSDYf0k528f5WGAW7XoXAM7wh7Ik5hbWUiOiJ6aG91eXQvY2FkZHkiLCJVVUlEIjoiY2E5MTRhODItZDQyZS00Yjk2LTlmYzEtNzBlZmIyY2NiZmI5IiwiT2Zmc2V0IjoyNDkyODcwLCJTdGFydGVkQXQiOiIyMDE2LTEyLTIxVDA2OjI1OjQyWiJ9&digest=sha256%3Abfc185be0245882f955f50de01e4702b286d1e8f35e73182bef075233a7fddae" http.request.useragent="docker/1.9.1 go/go1.4.2 git-commit/a34a1d5 kernel/4.0.4-301.fc22.x86_64 os/linux arch/amd64" instance.id=54707cec-7767-49e9-871f-ed4f1f9dc87a vars.name="zhouyt/caddy" vars.uuid=ca914a82-d42e-4b96-9fc1-70efb2ccbfb9 it looks like network issues. > Today push image to docker-registry failed again, the backend is gcs, the reproducible ratio is about 10%. This is a something different. It looks like the problem which was fixed yesterday at https://github.com/openshift/origin/pull/12395. Can you try to reproduce this on origin master ?
Confirmed on origin master with S3 as backend, can't reproduce this issue.
Can we verify this?
Can't reproduce this issue with latest OCP3.5: openshift version openshift v3.5.0.17+c55cf2b kubernetes v1.5.2+43a9be4 etcd 3.1.0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0884