Bug 1315948
| Summary: | [platformmanagement_public_595]When use the S3 as the integrated docker-registry storage can't push or pull images successfully | ||
|---|---|---|---|
| Product: | OKD | Reporter: | zhou ying <yinzhou> |
| Component: | Image Registry | Assignee: | Michail Kargakis <mkargaki> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Wei Sun <wsun> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.x | CC: | aos-bugs, pweil, yinzhou |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-05-12 17:12:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
"6. Tag a ImageStream; `oc tag docker.io/zhouying7780/hello-test hellotest:v1` 7. Docker login the integrated docker-registry by the user token; 8. Try to pull the image by command: `docker pull 172.30.239.9:5000/zhouy/hellotest:v1`" This is not related to S3 and is https://github.com/openshift/origin/issues/7792. Can you please post the output of `oc logs build/<name-of-your-build> --tail 6`? Only the last line is not enough. Also are you running OpenShift locally or in a EC2 instance? Actually nevermind about the oc logs command. I think I have just reproduced this. Looking... Can you post the output of `oc logs dc/docker-registry` after you see the 500 error? It would be really helpful to debug this. I can confirm that docker push is working against our registry backed with S3. http://pastebin.com/iVxMZEuk You most probably see the 500 due to wrong credentials (why I was seeing 500). Michail,
Thanks, I've confirmed with ami devenv_rhel_3734, in config.yml, the bucket should be like this:
bucket: openshift-qe-registry-testing-bucket1
now the issue has fixed.
[root@ip-172-18-11-61 amd64]# openshift version
openshift v1.1.4-16-gb5da002
kubernetes v1.2.0-origin-41-g91d3e75
etcd 2.2.5
[root@ip-172-18-11-61 amd64]# docker tag -f busybox 172.30.22.13:5000/zhouy/stream:v122
[root@ip-172-18-11-61 amd64]# docker push 172.30.22.13:5000/zhouy/stream:v122
The push refers to a repository [172.30.22.13:5000/zhouy/stream] (len: 1)
559d41a5eba1: Pushed
bb2571406a55: Pushed
v122: digest: sha256:c0aa37999df88fa0b7c4f4035f4e77d54469b30476d0fb6c4d5206b65eced788 size: 2734
[root@ip-172-18-11-61 amd64]# docker exec -it 0830b81503a5 more /config/config.yml
version: 0.1
log:
level: debug
http:
addr: :5000
storage:
cache:
layerinfo: inmemory
version: 0.1
log:
level: debug
http:
addr: :5000
storage:
cache:
layerinfo: inmemory
s3:
accesskey: xxxxxxxxxxx
secretkey: yyyyyyyyyyyyyyyyyyyyyyy
region: us-east-1
bucket: openshift-qe-registry-testing-bucket1
encrypt: true
secure: true
v4auth: false
rootdirectory: /registry
auth:
openshift:
realm: openshift
middleware:
repository:
- name: openshift
options:
pullthrough: true
|
Description of problem: When use the S3 as storage, try to pull or push images both failed Try pull images through remote registry will prompt: image not found; Push images will meet: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error Version-Release number of selected component (if applicable): openshift v1.1.3-536-gda71953 kubernetes v1.2.0-alpha.7-703-gbc4550d etcd 2.2.5 How reproducible: always Steps to Reproduce: 1. Use the config file to create screts: `oc secrets new registry config.yml` cat config.yml version: 0.1 log: level: debug http: addr: :5000 storage: cache: layerinfo: inmemory s3: accesskey: XXXXXXXXXXXXXXXXXX secretkey: RYYYYYYYYYYYYYYYYYYYYYYYYYYYYY region: us-east-1 bucket: s3://openshift-qe-registry-testing-bucket1/ encrypt: true secure: true v4auth: false rootdirectory: /registry auth: openshift: realm: openshift middleware: repository: - name: openshift options: pullthrough: true 2. Update the integrated docker-registry's dc , to use the S3 as storage: `oc env dc/docker-registry REGISTRY_CONFIGURATION_PATH=/config/config.yml` `oc volume dc/docker-registry --add --name=config -m /config --type=secret --secret-name=registry` 3. Wait for the docker-registry to deploy 4. Login OpenShift and create project; 5. Start build; 6. Tag a ImageStream; `oc tag docker.io/zhouying7780/hello-test hellotest:v1` 7. Docker login the integrated docker-registry by the user token; 8. Try to pull the image by command: `docker pull 172.30.239.9:5000/zhouy/hellotest:v1` Actual results: 5. Build will failed, with error:builder.go:204] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error 8. Pull will failed,with error:Error: image zhouy/hellotest:v1 not found Expected results: The docker-registry works well, could push or pull images; Additional info: By s3cmd can access the s3 bucket: [root@ip-172-18-6-56 amd64]# s3cmd put README.md s3://openshift-qe-registry-testing-bucket1 upload: 'README.md' -> 's3://openshift-qe-registry-testing-bucket1/README.md' [1 of 1] 12648 of 12648 100% in 0s 118.41 kB/s done [root@ip-172-18-6-56 amd64]# s3cmd ls s3://openshift-qe-registry-testing-bucket1 2016-03-09 03:24 12648 s3://openshift-qe-registry-testing-bucket1/README.md But the docker-registry pod has logs: time="2016-03-09T03:15:26.230874448Z" level=debug msg="s3.GetContent(\"/docker/registry/v2/repositories/zhouy/hellotest/_layers/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef/link\")" go.version=go1.4.2 http.request.host="172.30.239.9:5000" http.request.id=89178353-f0c8-4f3e-9de4-436973865601 http.request.method=HEAD http.request.remoteaddr="172.18.6.56:33651" http.request.uri="/v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64" instance.id=9c49f839-2553-47ee-8a90-da2300bfdf4c trace.duration=4.823170297s trace.file="/go/src/github.com/openshift/origin/Godeps/_workspace/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=0bdf5ed2-bdef-4770-8900-9b9ccbc107f7 trace.line=82 vars.digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" vars.name="zhouy/hellotest" time="2016-03-09T03:15:26.230935106Z" level=error msg="Failed to find blob \"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef\": driver.Error{DriverName:\"s3\", Enclosed:(*s3.Error)(0xc2099cf0e0)}" go.version=go1.4.2 http.request.host="172.30.239.9:5000" http.request.id=89178353-f0c8-4f3e-9de4-436973865601 http.request.method=HEAD http.request.remoteaddr="172.18.6.56:33651" http.request.uri="/v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64" instance.id=9c49f839-2553-47ee-8a90-da2300bfdf4c vars.digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" vars.name="zhouy/hellotest" time="2016-03-09T03:15:26.230961991Z" level=error msg="response completed with error" err.code=UNKNOWN err.detail="s3: The specified bucket does not exist" err.message="unknown error" go.version=go1.4.2 http.request.host="172.30.239.9:5000" http.request.id=461b3474-f692-4890-a19a-f4d0206f25eb http.request.method=HEAD http.request.remoteaddr="172.18.6.56:33652" http.request.uri="/v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.861301693s http.response.status=500 http.response.written=325 instance.id=9c49f839-2553-47ee-8a90-da2300bfdf4c vars.digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" vars.name="zhouy/hellotest" 172.18.6.56 - - [09/Mar/2016:03:15:21 +0000] "HEAD /v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef HTTP/1.1" 500 325 "" "docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64"