Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1481196

Summary:	skopeo delete + registry gc leads to inconsistent docker v2 registry
Product:	Red Hat Enterprise Linux 7	Reporter:	Marko Myllynen <myllynen>
Component:	docker-distribution	Assignee:	Jindrich Novy <jnovy>
Status:	CLOSED WONTFIX	QA Contact:	atomic-bugs <atomic-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.4	CC:	dornelas, jswensso, rrajaram
Target Milestone:	rc	Keywords:	Extras
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-01-15 07:41:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Marko Myllynen 2017-08-14 10:49:58 UTC

Description of problem:
This seems to be reproducible:

# systemctl stop docker-distribution
# /bin/rm -rf /var/lib/registry/*   
# systemctl start docker-distribution
# sh -x ./t
+ UPSTREAM=registry.access.redhat.com
+ REGISTRY=registry.example.com:5000
+ img=rhel7/etcd
+ tag=latest
+ docker pull registry.access.redhat.com/rhel7/etcd:latest
Trying to pull repository registry.access.redhat.com/rhel7/etcd ... 
sha256:bdc7602a49b66fbbab7517c42b57fa185df653608d3ed3b1ad04266425c842d0: Pulling from registry.access.redhat.com/rhel7/etcd
be1e04c6fbff: Already exists 
449ed11d916a: Already exists 
7bb86ab4ce06: Already exists 
Digest: sha256:bdc7602a49b66fbbab7517c42b57fa185df653608d3ed3b1ad04266425c842d0
Status: Image is up to date for registry.access.redhat.com/rhel7/etcd:latest
+ docker tag registry.access.redhat.com/rhel7/etcd:latest registry.example.com:5000/rhel7/etcd:latest
+ docker push registry.example.com:5000/rhel7/etcd:latest
The push refers to a repository [registry.example.com:5000/rhel7/etcd]
4a4cd6ca7b22: Pushed 
bb4f52dd78f6: Pushed 
f1f88d1c363a: Pushed 
latest: digest: sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d size: 949
+ docker rmi registry.example.com:5000/rhel7/etcd:latest
Untagged: registry.example.com:5000/rhel7/etcd:latest
Untagged: registry.example.com:5000/rhel7/etcd@sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d

At this point on a client this works as expected:

# docker rmi registry.example.com:5000/rhel7/etcd ; docker pull rhel7/etcd

Then:

# skopeo delete docker://registry.example.com:5000/rhel7/etcd:latest
# registry garbage-collect /etc/docker-distribution/registry/config.yml
rhel7/etcd

0 blobs marked, 5 blobs eligible for deletion
blob eligible for deletion: sha256:10f2056a02b666f19c83a2a8cb9591efd8121cb863a8e31124b8c4b3abe3ca6d
blob eligible for deletion: sha256:5b1d750aaf067bda7021df8f39b78f89cd38edb227317640d0e7238f5f491fde
blob eligible for deletion: sha256:7385c0dcabcfe94281ba4c47eb83fe40b51eb4436a2bf243623510dc7173853d
blob eligible for deletion: sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d
blob eligible for deletion: sha256:a7ef92bed2ae829b1a854e57e355e592ff0fab5ee5fdd2885aae48d6edf922ad
# registry garbage-collect /etc/docker-distribution/registry/config.yml
rhel7/etcd

0 blobs marked, 0 blobs eligible for deletion
# ./t                                                                  
Trying to pull repository registry.access.redhat.com/rhel7/etcd ... 
sha256:bdc7602a49b66fbbab7517c42b57fa185df653608d3ed3b1ad04266425c842d0: Pulling from registry.access.redhat.com/rhel7/etcd
be1e04c6fbff: Already exists 
449ed11d916a: Already exists 
7bb86ab4ce06: Already exists 
Digest: sha256:bdc7602a49b66fbbab7517c42b57fa185df653608d3ed3b1ad04266425c842d0
Status: Image is up to date for registry.access.redhat.com/rhel7/etcd:latest
The push refers to a repository [registry.example.com:5000/rhel7/etcd]
4a4cd6ca7b22: Layer already exists 
bb4f52dd78f6: Layer already exists 
f1f88d1c363a: Layer already exists 
latest: digest: sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d size: 949
Untagged: registry.example.com:5000/rhel7/etcd:latest
Untagged: registry.example.com:5000/rhel7/etcd@sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d

At this point it would seem that rhel7/etcd is pushed but the earlier client command fails. Doing delete/pull/push/gc in any order seems not to restore the registry to a working state any more.

Unclear whether this is a skopeo or docker-distribution issue so filing initially against skopeo.

Version-Release number of selected component (if applicable):
docker-1.12.6-48.git0fdc778.el7.x86_64
docker-distribution-2.6.1-1.1.gita25b9ef.el7.x86_64
skopeo-0.1.20-2.1.gite802625.el7.x86_64

Additional information:
# cat /etc/docker-distribution/registry/config.yml 
version: 0.1
log:
  level: warn
  fields:
    service: registry
storage:
    cache:
        layerinfo: inmemory
    delete:
        enabled: true
    filesystem:
        rootdirectory: /var/lib/registry
http:
    addr: :5000
    relativeurls: true
    secret: changeme
    tls:
      certificate: /etc/docker-distribution/registry/cert.cert
      key: /etc/docker-distribution/registry/cert.key
      #clientcas: /etc/docker-distribution/registry/ca.cert

Comment 2 Derrick Ornelas 2017-12-14 21:39:53 UTC

I've always assumed this was expected, and that a restart of the service was needed after the garbage-collect if you wanted to push the same image again

# cat /etc/docker-distribution/registry/config.yml 
version: 0.1
log:
  level: debug
  fields:
    service: registry
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /var/lib/registry
    delete:
        enabled: true
http:
    addr: :5000


# bash -x test.sh 
+ UPSTREAM=registry.access.redhat.com
+ REGISTRY=registry.example.com:5000
+ img=rhel7/etcd
+ tag=latest
+ docker pull registry.access.redhat.com/rhel7/etcd:latest
Trying to pull repository registry.access.redhat.com/rhel7/etcd ... 
latest: Pulling from registry.access.redhat.com/rhel7/etcd
9cadd93b16ff: Pull complete 
4aa565ad8b7a: Pull complete 
fd964436f70c: Pull complete 
Digest: sha256:ad158d2c23050eee1bc09faa291e7177a027670f4913c2cee6182a8a09d041bf
+ docker tag registry.access.redhat.com/rhel7/etcd:latest registry.example.com:5000/rhel7/etcd:latest
+ docker push registry.example.com:5000/rhel7/etcd:latest
The push refers to a repository [registry.example.com:5000/rhel7/etcd]
43f884ff9b52: Pushed 
02404b4d7e5d: Pushed 
e1d829eddb62: Pushed 
latest: digest: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732 size: 949
+ docker rmi registry.access.redhat.com/rhel7/etcd:latest
Untagged: registry.access.redhat.com/rhel7/etcd:latest
Untagged: registry.access.redhat.com/rhel7/etcd@sha256:ad158d2c23050eee1bc09faa291e7177a027670f4913c2cee6182a8a09d041bf


# docker rmi registry.example.com:5000/rhel7/etcd; docker pull registry.example.com:5000/rhel7/etcd
Untagged: registry.example.com:5000/rhel7/etcd:latest
Untagged: registry.example.com:5000/rhel7/etcd@sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732
Deleted: sha256:5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9
Deleted: sha256:f66339b2019f32dc8dbd4546ca88863355c76a30ce068c8fb2aa4fb383b21d43
Deleted: sha256:0619a10062a93ae062e8b0dd68e961105048ca1aa3232de1ce58010701b6469c
Deleted: sha256:e1d829eddb62dc49f1c56dbf8acd0c71299b3996115399de853a9d66d81b822f
Using default tag: latest
Trying to pull repository registry.example.com:5000/rhel7/etcd ... 
latest: Pulling from registry.example.com:5000/rhel7/etcd
9cadd93b16ff: Pull complete 
4aa565ad8b7a: Pull complete 
fd964436f70c: Pull complete 
Digest: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732


# skopeo --debug delete --tls-verify=false docker://registry.example.com:5000/rhel7/etcd:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.example.com:5000 
DEBU[0000] GET https://registry.example.com:5000/v2/ 
DEBU[0000] Ping https://registry.example.com:5000/v2/ err &url.Error{Op:"Get", URL:"https://registry.example.com:5000/v2/", Err:(*errors.errorString)(0xc420166010)} 
DEBU[0000] GET http://registry.example.com:5000/v2/ 
DEBU[0000] Ping http://registry.example.com:5000/v2/ err <nil> 
DEBU[0000] Ping http://registry.example.com:5000/v2/ status 200 
DEBU[0000] GET http://registry.example.com:5000/v2/rhel7/etcd/manifests/latest 
DEBU[0000] DELETE http://registry.example.com:5000/v2/rhel7/etcd/manifests/sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732 
DEBU[0000] Deleting /var/lib/atomic/sigstore/rhel7/etcd@sha256=473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732/signature-1 


# docker rmi registry.example.com:5000/rhel7/etcd; docker pull registry.example.com:5000/rhel7/etcd
Untagged: registry.example.com:5000/rhel7/etcd:latest
Untagged: registry.example.com:5000/rhel7/etcd@sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732
Deleted: sha256:5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9
Deleted: sha256:f66339b2019f32dc8dbd4546ca88863355c76a30ce068c8fb2aa4fb383b21d43
Deleted: sha256:0619a10062a93ae062e8b0dd68e961105048ca1aa3232de1ce58010701b6469c
Deleted: sha256:e1d829eddb62dc49f1c56dbf8acd0c71299b3996115399de853a9d66d81b822f
Using default tag: latest
Trying to pull repository registry.example.com:5000/rhel7/etcd ... 
Pulling repository registry.example.com:5000/rhel7/etcd
Error: image rhel7/etcd:latest not found


# registry garbage-collect /etc/docker-distribution/registry/config.yml
[...]

0 blobs marked, 5 blobs eligible for deletion
blob eligible for deletion: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/47/473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/47/473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732")  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=188.048µs trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=92183615-45fe-4598-bea2-8c434e07c268 trace.line=177
blob eligible for deletion: sha256:4aa565ad8b7a87248163ce7dba1dd3894821aac97e846b932ff6b8ef9a8a508a
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/4a/4aa565ad8b7a87248163ce7dba1dd3894821aac97e846b932ff6b8ef9a8a508a  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/4a/4aa565ad8b7a87248163ce7dba1dd3894821aac97e846b932ff6b8ef9a8a508a")  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=169.397µs trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=62a016d4-59a6-4270-8ad9-01f66a9b7cf8 trace.line=177
blob eligible for deletion: sha256:5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/5a/5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/5a/5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9")  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=182.572µs trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=cfd42666-b39c-45ea-a662-9942e9e620fe trace.line=177
blob eligible for deletion: sha256:9cadd93b16ff2a0c51ac967ea2abfadfac50cfa3af8b5bf983d89b8f8647f3e4
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/9c/9cadd93b16ff2a0c51ac967ea2abfadfac50cfa3af8b5bf983d89b8f8647f3e4  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/9c/9cadd93b16ff2a0c51ac967ea2abfadfac50cfa3af8b5bf983d89b8f8647f3e4")  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=13.956331ms trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=f72c14ba-cfbf-4586-982d-eed220f1000e trace.line=177
blob eligible for deletion: sha256:fd964436f70c99bd7997b0fc574767355089ed4bd42eeb3ee0bd5dce29f4c55f
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/fd/fd964436f70c99bd7997b0fc574767355089ed4bd42eeb3ee0bd5dce29f4c55f  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/fd/fd964436f70c99bd7997b0fc574767355089ed4bd42eeb3ee0bd5dce29f4c55f")  go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=3.228001ms trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=558e6ab5-d99b-42c4-82bd-50bc2b05daa6 trace.line=177


# registry garbage-collect /etc/docker-distribution/registry/config.yml
[...]

0 blobs marked, 0 blobs eligible for deletion



# bash -x test.sh 
+ UPSTREAM=registry.access.redhat.com
+ REGISTRY=registry.example.com:5000
+ img=rhel7/etcd
+ tag=latest
+ docker pull registry.access.redhat.com/rhel7/etcd:latest
Trying to pull repository registry.access.redhat.com/rhel7/etcd ... 
latest: Pulling from registry.access.redhat.com/rhel7/etcd
9cadd93b16ff: Pull complete 
4aa565ad8b7a: Pull complete 
fd964436f70c: Pull complete 
Digest: sha256:ad158d2c23050eee1bc09faa291e7177a027670f4913c2cee6182a8a09d041bf
+ docker tag registry.access.redhat.com/rhel7/etcd:latest registry.example.com:5000/rhel7/etcd:latest
+ docker push registry.example.com:5000/rhel7/etcd:latest
The push refers to a repository [registry.example.com:5000/rhel7/etcd]
43f884ff9b52: Layer already exists 
02404b4d7e5d: Layer already exists 
e1d829eddb62: Layer already exists 
latest: digest: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732 size: 949
+ docker rmi registry.access.redhat.com/rhel7/etcd:latest
Untagged: registry.access.redhat.com/rhel7/etcd:latest
Untagged: registry.access.redhat.com/rhel7/etcd@sha256:ad158d2c23050eee1bc09faa291e7177a027670f4913c2cee6182a8a09d041bf


# systemctl restart docker-distribution

# docker push registry.example.com:5000/rhel7/etcd:latest
The push refers to a repository [registry.example.com:5000/rhel7/etcd]
43f884ff9b52: Pushed 
02404b4d7e5d: Pushed 
e1d829eddb62: Pushed 
latest: digest: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732 size: 949



If this is actually a problem then I'd say it's in docker-distribution

Comment 5 RHEL Program Management 2021-01-15 07:41:03 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.