Bug 1368461

Description of problem:
This very well may be a bug with the docker registry and not skopeo/atomic, but wanted to open a bug to track the issue.

If you used 'atomic images delete --remote' to delete an image, while the image becomes immediately unavailable on the remote registry you cannot re-push the image to the registry even after garbage collection.

tcpdump shows the atomic client communicating to the registry when the push is attempted.

Version-Release number of selected component (if applicable):
atomic 1.11.0
skopeo version 0.1.14-dev
registry github.com/docker/distribution v2.5.0 (latest registry container)

How reproducible:
100%

Steps to Reproduce:
Setup

Host A - docker 1.10, running latest docker.io/registry as of last thurs. (rhel-base:5000)

docker run -d -e REGISTRY_STORAGE_DELETE_ENABLED=true -p 5000:5000 docker.io/registry

Host B running docker 1.10 with Atomic 1.11 and skopeo 0.1.14

Host C running docker 1.10

All hosts are RHEL.
Procedure

1. on Host C, build image from Dockerfile and tag with host A registry info:

FROM busybox
entrypoint /bin/who

[root@atomic-registry builder]# docker build -t rhel-base:5000/busyboxwhom .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM busybox
 ---> 2b8fd9751c4c
Step 2 : ENTRYPOINT /bin/who
 ---> Using cache
 ---> 0a18ce01aa55
Successfully built 0a18ce01aa55

2. Push from Host C to Host A

[root@atomic-registry builder]# docker push rhel-base:5000/busyboxwhom
The push refers to a repository [rhel-base:5000/busyboxwhom]
8ac8bfaff55a: Layer already exists
latest: digest: sha256:9d7f22bb0cbb71bc7bc03cfbda4592c21aedfb461d27d0dbd56fd0b6eb4ba77d size: 505

3. On host B, call 'atomic images delete --remote' on image on A - indicates that the image is tagged for deletion

[root@fedora_vm ~]# atomic images delete --remote rhel-base:5000/busyboxwhom
Do you wish to delete ['rhel-base:5000/busyboxwhom']? (y/N) y
Image docker://rhel-base:5000/busyboxwhom marked for deletion

4. On host A, call garbage collection for running registry container - it reports blob deletion

[root@rhel-base build-image]# docker exec -it e1ee1c36997a bin/registry garbage-collect /etc/docker/registry/config.yml
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/8d/8ddc19f16526912237dd8af81971d5e4dd0587907234be2b83e249518d5b673f  go.version=go1.6.3 instance.id=3a148b58-0249-4391-8f09-457ebbec5230
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4  go.version=go1.6.3 instance.id=3a148b58-0249-4391-8f09-457ebbec5230
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/c4/c43b7b7f0e356a6c202705e96f640cffa32b15c4178d5ff67d73012b9b43ffe5  go.version=go1.6.3 instance.id=3a148b58-0249-4391-8f09-457ebbec5230

5. On host C, delete image build in step 1 then rebuild it with same Docker file.

[root@atomic-registry builder]# docker images
REPOSITORY                                        TAG                 IMAGE ID            CREATED             SIZE
docker.io/openshift/origin-docker-registry        latest              579eac409085        24 minutes ago      349.1 MB
docker.io/busybox                                 latest              2b8fd9751c4c        7 weeks ago         1.093 MB
[root@atomic-registry builder]# docker rmi 579eac409085
Untagged: docker.io/openshift/origin-docker-registry:latest
Deleted: sha256:579eac409085392c9ca498566d9b61531245ffd63e6078ff99eaf729e3c7036a
Deleted: sha256:34db1304880695eabcb7c8247a58db70e46c739032a74cb8ea8dee023275e321
Deleted: sha256:ebe5d2de1b01e30ce43a5602317c6ce0ad8df41cbb6f47dd5c2b308cd0e12a9d
[root@atomic-registry builder]# docker build -t rhel-base:5000/busyboxwhom .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM busybox
 ---> 2b8fd9751c4c
Step 2 : ENTRYPOINT /bin/who
 ---> Using cache
 ---> 0a18ce01aa55
Successfully built 0a18ce01aa55

6. Attempt to push image to registry on A - get error that layer exists

[root@atomic-registry builder]# docker push rhel-base:5000/busyboxwhom
The push refers to a repository [rhel-base:5000/busyboxwhom]
8ac8bfaff55a: Layer already exists
latest: digest: sha256:9d7f22bb0cbb71bc7bc03cfbda4592c21aedfb461d27d0dbd56fd0b6eb4ba77d size: 505

7. Attempt to pull the image to host B, no image exists

[root@fedora_vm ~]# docker pull rhel-base:5000/busyboxwhom
Using default tag: latest
Trying to pull repository rhel-base:5000/busyboxwhom ...
Pulling repository rhel-base:5000/busyboxwhom
Error: image busyboxwhom not found
[root@fedora_vm ~]#


Actual results:
Push fails.

Expected results:
Should be able to push image that was deleted from registry.


Additional info: