Description of problem: - When we configured docker-distribution to use RADOS GW w/ s3 interface, docker push does not finish. Version-Release number of selected component (if applicable): - ceph-radosgw-12.2.4-30.el7cp.x86_64 - docker-1.13.1-63.git94f4240.el7.x86_64 - docker-distribution-2.6.2-2.git48294d9.el7.x86_64 How reproducible: 100% Steps to Reproduce: 0. Install and setup RADOS GW - NOTE: Settings RADOS GW is fine, as putting data by s3cmd worked fine. 1. Setup docker-distribution 1-1. Configure config.yml # cat /etc/docker-distribution/registry/config.yml version: 0.1 log: fields: service: registry storage: cache: layerinfo: inmemory cache: blobdescriptor: inmemory s3: accesskey: testkey secretkey: testsecret region: us bucket: docker-registry rootdirectory: / regionendpoint: http://knakayam-ceph-c2.example.com secure: false http: addr: 0.0.0.0:5000 debug: addr: localhost:5002 headers: X-Content-Type-Options: [nosniff] 1-2. Restart service # systemctl restart docker-distribution.service 2. Push image to the docker-registry # docker push 127.0.0.1:5000/busybox Actual results: - docker push failed with following error # docker push 127.0.0.1:5000/busybox The push refers to a repository [127.0.0.1:5000/busybox] 8e9a7d50b12c: Retrying in 5 seconds (after several attempt) # docker push 127.0.0.1:5000/busybox The push refers to a repository [127.0.0.1:5000/busybox] 8e9a7d50b12c: Pushing 1.378 MB received unexpected HTTP status: 500 Internal Server Error Expected results: - dokcker push worked fine. Additional info: - Putting data via s3cmd works fine. - Using swift API worked fine. - After got 500 error, if I checked ceph side, the data exists: ~~~ files/docker/registry/v2/repositories/busybox/_layers/sha256/22c2dd5ee85dc01136051800684b0bf30016a3082f97093c806152bf43d4e089/link files/docker/registry/v2/repositories/busybox/_layers/sha256/75a0e65efd518b9bcac8a8287e5c7032bc81f8cbfbe03271fd049b81ab26119b/link files/docker/registry/v2/repositories/busybox/_manifests/revisions/sha256/1bd6df27274fef1dd36eb529d0f4c8033f61c675d6b04213dd913f902f7cafb5/link files/docker/registry/v2/repositories/busybox/_manifests/tags/latest/current/link files/docker/registry/v2/repositories/busybox/_manifests/tags/latest/index/sha256/1bd6df27274fef1dd36eb529d0f4c8033f61c675d6b04213dd913f902f7cafb5/link ~~~ - `journalctl -f -u docker-distribution` during the push: http://pastebin.test.redhat.com/620933 - `journalctl -f -u docker` during the push: http://pastebin.test.redhat.com/620934
I am reporting this issue as Ceph component, because docker-distribution could push to actual S3 storage. Only RADOS GW w/ S3 interface has this issue.
Hi Matt, could you please give us initial feedback? If it should forward to docker registry or docker team, please let us know.
Created attachment 1510255 [details] rgw log at rgw debug level 20
Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate. Regards, Giri