Description of problem: When use the S3 as storage, try to pull or push images both failed Try pull images through remote registry will prompt: image not found; Push images will meet: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error Version-Release number of selected component (if applicable): openshift v1.1.3-536-gda71953 kubernetes v1.2.0-alpha.7-703-gbc4550d etcd 2.2.5 How reproducible: always Steps to Reproduce: 1. Use the config file to create screts: `oc secrets new registry config.yml` cat config.yml version: 0.1 log: level: debug http: addr: :5000 storage: cache: layerinfo: inmemory s3: accesskey: XXXXXXXXXXXXXXXXXX secretkey: RYYYYYYYYYYYYYYYYYYYYYYYYYYYYY region: us-east-1 bucket: s3://openshift-qe-registry-testing-bucket1/ encrypt: true secure: true v4auth: false rootdirectory: /registry auth: openshift: realm: openshift middleware: repository: - name: openshift options: pullthrough: true 2. Update the integrated docker-registry's dc , to use the S3 as storage: `oc env dc/docker-registry REGISTRY_CONFIGURATION_PATH=/config/config.yml` `oc volume dc/docker-registry --add --name=config -m /config --type=secret --secret-name=registry` 3. Wait for the docker-registry to deploy 4. Login OpenShift and create project; 5. Start build; 6. Tag a ImageStream; `oc tag docker.io/zhouying7780/hello-test hellotest:v1` 7. Docker login the integrated docker-registry by the user token; 8. Try to pull the image by command: `docker pull 172.30.239.9:5000/zhouy/hellotest:v1` Actual results: 5. Build will failed, with error:builder.go:204] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error 8. Pull will failed,with error:Error: image zhouy/hellotest:v1 not found Expected results: The docker-registry works well, could push or pull images; Additional info: By s3cmd can access the s3 bucket: [root@ip-172-18-6-56 amd64]# s3cmd put README.md s3://openshift-qe-registry-testing-bucket1 upload: 'README.md' -> 's3://openshift-qe-registry-testing-bucket1/README.md' [1 of 1] 12648 of 12648 100% in 0s 118.41 kB/s done [root@ip-172-18-6-56 amd64]# s3cmd ls s3://openshift-qe-registry-testing-bucket1 2016-03-09 03:24 12648 s3://openshift-qe-registry-testing-bucket1/README.md But the docker-registry pod has logs: time="2016-03-09T03:15:26.230874448Z" level=debug msg="s3.GetContent(\"/docker/registry/v2/repositories/zhouy/hellotest/_layers/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef/link\")" go.version=go1.4.2 http.request.host="172.30.239.9:5000" http.request.id=89178353-f0c8-4f3e-9de4-436973865601 http.request.method=HEAD http.request.remoteaddr="172.18.6.56:33651" http.request.uri="/v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64" instance.id=9c49f839-2553-47ee-8a90-da2300bfdf4c trace.duration=4.823170297s trace.file="/go/src/github.com/openshift/origin/Godeps/_workspace/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=0bdf5ed2-bdef-4770-8900-9b9ccbc107f7 trace.line=82 vars.digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" vars.name="zhouy/hellotest" time="2016-03-09T03:15:26.230935106Z" level=error msg="Failed to find blob \"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef\": driver.Error{DriverName:\"s3\", Enclosed:(*s3.Error)(0xc2099cf0e0)}" go.version=go1.4.2 http.request.host="172.30.239.9:5000" http.request.id=89178353-f0c8-4f3e-9de4-436973865601 http.request.method=HEAD http.request.remoteaddr="172.18.6.56:33651" http.request.uri="/v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64" instance.id=9c49f839-2553-47ee-8a90-da2300bfdf4c vars.digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" vars.name="zhouy/hellotest" time="2016-03-09T03:15:26.230961991Z" level=error msg="response completed with error" err.code=UNKNOWN err.detail="s3: The specified bucket does not exist" err.message="unknown error" go.version=go1.4.2 http.request.host="172.30.239.9:5000" http.request.id=461b3474-f692-4890-a19a-f4d0206f25eb http.request.method=HEAD http.request.remoteaddr="172.18.6.56:33652" http.request.uri="/v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.861301693s http.response.status=500 http.response.written=325 instance.id=9c49f839-2553-47ee-8a90-da2300bfdf4c vars.digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" vars.name="zhouy/hellotest" 172.18.6.56 - - [09/Mar/2016:03:15:21 +0000] "HEAD /v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef HTTP/1.1" 500 325 "" "docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64"
"6. Tag a ImageStream; `oc tag docker.io/zhouying7780/hello-test hellotest:v1` 7. Docker login the integrated docker-registry by the user token; 8. Try to pull the image by command: `docker pull 172.30.239.9:5000/zhouy/hellotest:v1`" This is not related to S3 and is https://github.com/openshift/origin/issues/7792. Can you please post the output of `oc logs build/<name-of-your-build> --tail 6`? Only the last line is not enough. Also are you running OpenShift locally or in a EC2 instance?
Actually nevermind about the oc logs command. I think I have just reproduced this. Looking...
Can you post the output of `oc logs dc/docker-registry` after you see the 500 error? It would be really helpful to debug this.
I can confirm that docker push is working against our registry backed with S3. http://pastebin.com/iVxMZEuk You most probably see the 500 due to wrong credentials (why I was seeing 500).
Michail, Thanks, I've confirmed with ami devenv_rhel_3734, in config.yml, the bucket should be like this: bucket: openshift-qe-registry-testing-bucket1 now the issue has fixed. [root@ip-172-18-11-61 amd64]# openshift version openshift v1.1.4-16-gb5da002 kubernetes v1.2.0-origin-41-g91d3e75 etcd 2.2.5 [root@ip-172-18-11-61 amd64]# docker tag -f busybox 172.30.22.13:5000/zhouy/stream:v122 [root@ip-172-18-11-61 amd64]# docker push 172.30.22.13:5000/zhouy/stream:v122 The push refers to a repository [172.30.22.13:5000/zhouy/stream] (len: 1) 559d41a5eba1: Pushed bb2571406a55: Pushed v122: digest: sha256:c0aa37999df88fa0b7c4f4035f4e77d54469b30476d0fb6c4d5206b65eced788 size: 2734 [root@ip-172-18-11-61 amd64]# docker exec -it 0830b81503a5 more /config/config.yml version: 0.1 log: level: debug http: addr: :5000 storage: cache: layerinfo: inmemory version: 0.1 log: level: debug http: addr: :5000 storage: cache: layerinfo: inmemory s3: accesskey: xxxxxxxxxxx secretkey: yyyyyyyyyyyyyyyyyyyyyyy region: us-east-1 bucket: openshift-qe-registry-testing-bucket1 encrypt: true secure: true v4auth: false rootdirectory: /registry auth: openshift: realm: openshift middleware: repository: - name: openshift options: pullthrough: true