.Pushing a docker image to the Ceph Object Gateway over s3 does not complete
In certain situations when configuring `docker-distribution` to use Ceph Object Gateway with the s3 interface the `docker push` command does not complete. Instead the command fails with an HTTP 500 error.
There is no workaround at this time.
DescriptionKenjiro Nakayama
2018-07-20 01:27:43 UTC
Description of problem:
- When we configured docker-distribution to use RADOS GW w/ s3 interface, docker push does not finish.
Version-Release number of selected component (if applicable):
- ceph-radosgw-12.2.4-30.el7cp.x86_64
- docker-1.13.1-63.git94f4240.el7.x86_64
- docker-distribution-2.6.2-2.git48294d9.el7.x86_64
How reproducible: 100%
Steps to Reproduce:
0. Install and setup RADOS GW
- NOTE: Settings RADOS GW is fine, as putting data by s3cmd worked fine.
1. Setup docker-distribution
1-1. Configure config.yml
# cat /etc/docker-distribution/registry/config.yml
version: 0.1
log:
fields:
service: registry
storage:
cache:
layerinfo: inmemory
cache:
blobdescriptor: inmemory
s3:
accesskey: testkey
secretkey: testsecret
region: us
bucket: docker-registry
rootdirectory: /
regionendpoint: http://knakayam-ceph-c2.example.com
secure: false
http:
addr: 0.0.0.0:5000
debug:
addr: localhost:5002
headers:
X-Content-Type-Options: [nosniff]
1-2. Restart service
# systemctl restart docker-distribution.service
2. Push image to the docker-registry
# docker push 127.0.0.1:5000/busybox
Actual results:
- docker push failed with following error
# docker push 127.0.0.1:5000/busybox
The push refers to a repository [127.0.0.1:5000/busybox]
8e9a7d50b12c: Retrying in 5 seconds
(after several attempt)
# docker push 127.0.0.1:5000/busybox
The push refers to a repository [127.0.0.1:5000/busybox]
8e9a7d50b12c: Pushing 1.378 MB
received unexpected HTTP status: 500 Internal Server Error
Expected results:
- dokcker push worked fine.
Additional info:
- Putting data via s3cmd works fine.
- Using swift API worked fine.
- After got 500 error, if I checked ceph side, the data exists:
~~~
files/docker/registry/v2/repositories/busybox/_layers/sha256/22c2dd5ee85dc01136051800684b0bf30016a3082f97093c806152bf43d4e089/link
files/docker/registry/v2/repositories/busybox/_layers/sha256/75a0e65efd518b9bcac8a8287e5c7032bc81f8cbfbe03271fd049b81ab26119b/link
files/docker/registry/v2/repositories/busybox/_manifests/revisions/sha256/1bd6df27274fef1dd36eb529d0f4c8033f61c675d6b04213dd913f902f7cafb5/link
files/docker/registry/v2/repositories/busybox/_manifests/tags/latest/current/link
files/docker/registry/v2/repositories/busybox/_manifests/tags/latest/index/sha256/1bd6df27274fef1dd36eb529d0f4c8033f61c675d6b04213dd913f902f7cafb5/link
~~~
- `journalctl -f -u docker-distribution` during the push: http://pastebin.test.redhat.com/620933
- `journalctl -f -u docker` during the push: http://pastebin.test.redhat.com/620934
I am reporting this issue as Ceph component, because docker-distribution could push to actual S3 storage. Only RADOS GW w/ S3 interface has this issue.
Description of problem: - When we configured docker-distribution to use RADOS GW w/ s3 interface, docker push does not finish. Version-Release number of selected component (if applicable): - ceph-radosgw-12.2.4-30.el7cp.x86_64 - docker-1.13.1-63.git94f4240.el7.x86_64 - docker-distribution-2.6.2-2.git48294d9.el7.x86_64 How reproducible: 100% Steps to Reproduce: 0. Install and setup RADOS GW - NOTE: Settings RADOS GW is fine, as putting data by s3cmd worked fine. 1. Setup docker-distribution 1-1. Configure config.yml # cat /etc/docker-distribution/registry/config.yml version: 0.1 log: fields: service: registry storage: cache: layerinfo: inmemory cache: blobdescriptor: inmemory s3: accesskey: testkey secretkey: testsecret region: us bucket: docker-registry rootdirectory: / regionendpoint: http://knakayam-ceph-c2.example.com secure: false http: addr: 0.0.0.0:5000 debug: addr: localhost:5002 headers: X-Content-Type-Options: [nosniff] 1-2. Restart service # systemctl restart docker-distribution.service 2. Push image to the docker-registry # docker push 127.0.0.1:5000/busybox Actual results: - docker push failed with following error # docker push 127.0.0.1:5000/busybox The push refers to a repository [127.0.0.1:5000/busybox] 8e9a7d50b12c: Retrying in 5 seconds (after several attempt) # docker push 127.0.0.1:5000/busybox The push refers to a repository [127.0.0.1:5000/busybox] 8e9a7d50b12c: Pushing 1.378 MB received unexpected HTTP status: 500 Internal Server Error Expected results: - dokcker push worked fine. Additional info: - Putting data via s3cmd works fine. - Using swift API worked fine. - After got 500 error, if I checked ceph side, the data exists: ~~~ files/docker/registry/v2/repositories/busybox/_layers/sha256/22c2dd5ee85dc01136051800684b0bf30016a3082f97093c806152bf43d4e089/link files/docker/registry/v2/repositories/busybox/_layers/sha256/75a0e65efd518b9bcac8a8287e5c7032bc81f8cbfbe03271fd049b81ab26119b/link files/docker/registry/v2/repositories/busybox/_manifests/revisions/sha256/1bd6df27274fef1dd36eb529d0f4c8033f61c675d6b04213dd913f902f7cafb5/link files/docker/registry/v2/repositories/busybox/_manifests/tags/latest/current/link files/docker/registry/v2/repositories/busybox/_manifests/tags/latest/index/sha256/1bd6df27274fef1dd36eb529d0f4c8033f61c675d6b04213dd913f902f7cafb5/link ~~~ - `journalctl -f -u docker-distribution` during the push: http://pastebin.test.redhat.com/620933 - `journalctl -f -u docker` during the push: http://pastebin.test.redhat.com/620934