Bug 1604979 - docker push does not complete when docker-distribution uses RADOS GW w/ s3 interface
Summary: docker push does not complete when docker-distribution uses RADOS GW w/ s3 in...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 3.2
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z2
: 3.3
Assignee: Daniel Gryniewicz
QA Contact: Tejas
URL:
Whiteboard:
Depends On:
Blocks: 1629656
TreeView+ depends on / blocked
 
Reported: 2018-07-20 01:27 UTC by Kenjiro Nakayama
Modified: 2022-03-13 15:16 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
.Pushing a docker image to the Ceph Object Gateway over s3 does not complete In certain situations when configuring `docker-distribution` to use Ceph Object Gateway with the s3 interface the `docker push` command does not complete. Instead the command fails with an HTTP 500 error. There is no workaround at this time.
Clone Of:
Environment:
Last Closed: 2019-08-23 10:13:31 UTC
Embargoed:


Attachments (Terms of Use)
rgw log at rgw debug level 20 (305.18 KB, text/plain)
2018-11-30 20:11 UTC, Matt Benjamin (redhat)
no flags Details

Description Kenjiro Nakayama 2018-07-20 01:27:43 UTC
Description of problem:
- When we configured docker-distribution to use RADOS GW w/ s3 interface, docker push does not finish.

Version-Release number of selected component (if applicable):
- ceph-radosgw-12.2.4-30.el7cp.x86_64
- docker-1.13.1-63.git94f4240.el7.x86_64
- docker-distribution-2.6.2-2.git48294d9.el7.x86_64

How reproducible: 100%

Steps to Reproduce:

0. Install and setup RADOS GW
- NOTE: Settings RADOS GW is fine, as putting data by s3cmd worked fine.

1. Setup docker-distribution

1-1. Configure config.yml

   # cat /etc/docker-distribution/registry/config.yml
   version: 0.1
   log:
     fields:
       service: registry
   storage:
       cache:
           layerinfo: inmemory
       cache:
         blobdescriptor: inmemory
       s3:
         accesskey: testkey
         secretkey: testsecret
         region: us
         bucket: docker-registry
         rootdirectory: /
         regionendpoint: http://knakayam-ceph-c2.example.com
         secure: false
   http:
     addr: 0.0.0.0:5000
     debug:
       addr: localhost:5002
     headers:
       X-Content-Type-Options: [nosniff]

1-2. Restart service

  # systemctl restart docker-distribution.service

2. Push image to the docker-registry

  # docker push 127.0.0.1:5000/busybox

Actual results:
- docker push failed with following error

  # docker push 127.0.0.1:5000/busybox
  The push refers to a repository [127.0.0.1:5000/busybox]
  8e9a7d50b12c: Retrying in 5 seconds

(after several attempt)

  # docker push 127.0.0.1:5000/busybox
  The push refers to a repository [127.0.0.1:5000/busybox]
  8e9a7d50b12c: Pushing 1.378 MB
  received unexpected HTTP status: 500 Internal Server Error

Expected results:
- dokcker push worked fine.

Additional info:
- Putting data via s3cmd works fine.
- Using swift API worked fine.
- After got 500 error, if I checked ceph side, the data exists:

~~~
  files/docker/registry/v2/repositories/busybox/_layers/sha256/22c2dd5ee85dc01136051800684b0bf30016a3082f97093c806152bf43d4e089/link
  files/docker/registry/v2/repositories/busybox/_layers/sha256/75a0e65efd518b9bcac8a8287e5c7032bc81f8cbfbe03271fd049b81ab26119b/link
  files/docker/registry/v2/repositories/busybox/_manifests/revisions/sha256/1bd6df27274fef1dd36eb529d0f4c8033f61c675d6b04213dd913f902f7cafb5/link
  files/docker/registry/v2/repositories/busybox/_manifests/tags/latest/current/link
  files/docker/registry/v2/repositories/busybox/_manifests/tags/latest/index/sha256/1bd6df27274fef1dd36eb529d0f4c8033f61c675d6b04213dd913f902f7cafb5/link
~~~

- `journalctl -f -u docker-distribution` during the push: http://pastebin.test.redhat.com/620933
- `journalctl -f -u docker` during the push: http://pastebin.test.redhat.com/620934

Comment 3 Kenjiro Nakayama 2018-07-20 01:29:15 UTC
I am reporting this issue as Ceph component, because docker-distribution could push to actual S3 storage. Only RADOS GW w/ S3 interface has this issue.

Comment 4 Kenjiro Nakayama 2018-08-03 02:22:49 UTC
Hi Matt, could you please give us initial feedback? If it should forward to docker registry or docker team, please let us know.

Comment 8 Matt Benjamin (redhat) 2018-11-30 20:11:55 UTC
Created attachment 1510255 [details]
rgw log at rgw debug level 20

Comment 12 Giridhar Ramaraju 2019-08-05 13:08:26 UTC
Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate. 

Regards,
Giri

Comment 13 Giridhar Ramaraju 2019-08-05 13:09:54 UTC
Updating the QA Contact to a Hemant. Hemant will be rerouting them to the appropriate QE Associate. 

Regards,
Giri


Note You need to log in before you can comment on or make changes to this bug.