Bug 1315948 - [platformmanagement_public_595]When use the S3 as the integrated docker-registry storage can't push or pull images successfully
[platformmanagement_public_595]When use the S3 as the integrated docker-regis...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Image Registry (Show other bugs)
3.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Michail Kargakis
Wei Sun
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-09 00:38 EST by zhou ying
Modified: 2016-05-12 13:12 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-12 13:12:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description zhou ying 2016-03-09 00:38:19 EST
Description of problem:
When use the S3 as storage, try to pull or push  images both failed
Try pull images through remote registry will prompt: image not found;
Push images will meet: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error

Version-Release number of selected component (if applicable):
openshift v1.1.3-536-gda71953
kubernetes v1.2.0-alpha.7-703-gbc4550d
etcd 2.2.5

How reproducible:
always

Steps to Reproduce:
1. Use the config file to create screts:
 `oc secrets new registry config.yml`
cat config.yml 
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    layerinfo: inmemory
  s3:
    accesskey: XXXXXXXXXXXXXXXXXX
    secretkey: RYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
    region: us-east-1
    bucket: s3://openshift-qe-registry-testing-bucket1/
    encrypt: true
    secure: true
    v4auth: false
    rootdirectory: /registry
auth:
  openshift:
    realm: openshift
middleware:
  repository:
    - name: openshift
      options:
        pullthrough: true

2. Update the integrated docker-registry's dc , to use the S3 as storage:
  `oc env dc/docker-registry  REGISTRY_CONFIGURATION_PATH=/config/config.yml`
  `oc volume dc/docker-registry --add --name=config -m /config --type=secret --secret-name=registry`

3. Wait for the docker-registry to deploy

4. Login OpenShift and create project;
5. Start build;
6. Tag a ImageStream;
   `oc tag docker.io/zhouying7780/hello-test hellotest:v1`
7. Docker login the integrated docker-registry by the user token;
8. Try to pull the image by command:
   `docker pull 172.30.239.9:5000/zhouy/hellotest:v1`

Actual results:
5. Build will failed, with error:builder.go:204] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error
8. Pull will failed,with error:Error: image zhouy/hellotest:v1 not found

Expected results:
The docker-registry works well, could push or pull images;


Additional info:
By s3cmd can access the s3 bucket:
[root@ip-172-18-6-56 amd64]# s3cmd put README.md s3://openshift-qe-registry-testing-bucket1
upload: 'README.md' -> 's3://openshift-qe-registry-testing-bucket1/README.md'  [1 of 1]
 12648 of 12648   100% in    0s   118.41 kB/s  done
[root@ip-172-18-6-56 amd64]# s3cmd ls s3://openshift-qe-registry-testing-bucket1
2016-03-09 03:24     12648   s3://openshift-qe-registry-testing-bucket1/README.md

But the docker-registry pod has logs:
time="2016-03-09T03:15:26.230874448Z" level=debug msg="s3.GetContent(\"/docker/registry/v2/repositories/zhouy/hellotest/_layers/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef/link\")" go.version=go1.4.2 http.request.host="172.30.239.9:5000" http.request.id=89178353-f0c8-4f3e-9de4-436973865601 http.request.method=HEAD http.request.remoteaddr="172.18.6.56:33651" http.request.uri="/v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64" instance.id=9c49f839-2553-47ee-8a90-da2300bfdf4c trace.duration=4.823170297s trace.file="/go/src/github.com/openshift/origin/Godeps/_workspace/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).GetContent" trace.id=0bdf5ed2-bdef-4770-8900-9b9ccbc107f7 trace.line=82 vars.digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" vars.name="zhouy/hellotest" 
time="2016-03-09T03:15:26.230935106Z" level=error msg="Failed to find blob \"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef\": driver.Error{DriverName:\"s3\", Enclosed:(*s3.Error)(0xc2099cf0e0)}" go.version=go1.4.2 http.request.host="172.30.239.9:5000" http.request.id=89178353-f0c8-4f3e-9de4-436973865601 http.request.method=HEAD http.request.remoteaddr="172.18.6.56:33651" http.request.uri="/v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64" instance.id=9c49f839-2553-47ee-8a90-da2300bfdf4c vars.digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" vars.name="zhouy/hellotest" 
time="2016-03-09T03:15:26.230961991Z" level=error msg="response completed with error" err.code=UNKNOWN err.detail="s3: The specified bucket does not exist" err.message="unknown error" go.version=go1.4.2 http.request.host="172.30.239.9:5000" http.request.id=461b3474-f692-4890-a19a-f4d0206f25eb http.request.method=HEAD http.request.remoteaddr="172.18.6.56:33652" http.request.uri="/v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.861301693s http.response.status=500 http.response.written=325 instance.id=9c49f839-2553-47ee-8a90-da2300bfdf4c vars.digest="sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" vars.name="zhouy/hellotest" 
172.18.6.56 - - [09/Mar/2016:03:15:21 +0000] "HEAD /v2/zhouy/hellotest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef HTTP/1.1" 500 325 "" "docker/1.9.1 go/go1.4.2 kernel/3.10.0-229.7.2.el7.x86_64 os/linux arch/amd64"
Comment 1 Michail Kargakis 2016-03-16 12:54:48 EDT
"6. Tag a ImageStream;
   `oc tag docker.io/zhouying7780/hello-test hellotest:v1`
7. Docker login the integrated docker-registry by the user token;
8. Try to pull the image by command:
   `docker pull 172.30.239.9:5000/zhouy/hellotest:v1`"

This is not related to S3 and is https://github.com/openshift/origin/issues/7792.

Can you please post the output of `oc logs build/<name-of-your-build> --tail 6`? Only the last line is not enough. Also are you running OpenShift locally or in a EC2 instance?
Comment 2 Michail Kargakis 2016-03-16 12:58:10 EDT
Actually nevermind about the oc logs command. I think I have just reproduced this. Looking...
Comment 3 Michail Kargakis 2016-03-16 14:42:18 EDT
Can you post the output of `oc logs dc/docker-registry` after you see the 500 error? It would be really helpful to debug this.
Comment 4 Michail Kargakis 2016-03-16 15:09:05 EDT
I can confirm that docker push is working against our registry backed with S3.

http://pastebin.com/iVxMZEuk

You most probably see the 500 due to wrong credentials (why I was seeing 500).
Comment 5 zhou ying 2016-03-17 05:43:53 EDT
Michail,
   Thanks, I've confirmed with ami devenv_rhel_3734, in config.yml, the bucket should be like this:
bucket: openshift-qe-registry-testing-bucket1

now the issue has fixed.
[root@ip-172-18-11-61 amd64]# openshift version
openshift v1.1.4-16-gb5da002
kubernetes v1.2.0-origin-41-g91d3e75
etcd 2.2.5

[root@ip-172-18-11-61 amd64]# docker tag -f busybox 172.30.22.13:5000/zhouy/stream:v122
[root@ip-172-18-11-61 amd64]# docker push 172.30.22.13:5000/zhouy/stream:v122
The push refers to a repository [172.30.22.13:5000/zhouy/stream] (len: 1)
559d41a5eba1: Pushed 
bb2571406a55: Pushed 
v122: digest: sha256:c0aa37999df88fa0b7c4f4035f4e77d54469b30476d0fb6c4d5206b65eced788 size: 2734



[root@ip-172-18-11-61 amd64]# docker exec -it  0830b81503a5 more /config/config.yml
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    layerinfo: inmemory
version: 0.1
log:
  level: debug
http:
  addr: :5000
storage:
  cache:
    layerinfo: inmemory
  s3:
    accesskey: xxxxxxxxxxx
    secretkey: yyyyyyyyyyyyyyyyyyyyyyy
    region: us-east-1
    bucket: openshift-qe-registry-testing-bucket1
    encrypt: true
    secure: true
    v4auth: false
    rootdirectory: /registry
auth:
  openshift:
    realm: openshift
middleware:
  repository:
    - name: openshift
      options:
        pullthrough: true

Note You need to log in before you can comment on or make changes to this bug.