Bug 1481196
| Summary: | skopeo delete + registry gc leads to inconsistent docker v2 registry | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Marko Myllynen <myllynen> |
| Component: | docker-distribution | Assignee: | Jindrich Novy <jnovy> |
| Status: | CLOSED WONTFIX | QA Contact: | atomic-bugs <atomic-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.4 | CC: | dornelas, jswensso, rrajaram |
| Target Milestone: | rc | Keywords: | Extras |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-01-15 07:41:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I've always assumed this was expected, and that a restart of the service was needed after the garbage-collect if you wanted to push the same image again
# cat /etc/docker-distribution/registry/config.yml
version: 0.1
log:
level: debug
fields:
service: registry
storage:
cache:
layerinfo: inmemory
filesystem:
rootdirectory: /var/lib/registry
delete:
enabled: true
http:
addr: :5000
# bash -x test.sh
+ UPSTREAM=registry.access.redhat.com
+ REGISTRY=registry.example.com:5000
+ img=rhel7/etcd
+ tag=latest
+ docker pull registry.access.redhat.com/rhel7/etcd:latest
Trying to pull repository registry.access.redhat.com/rhel7/etcd ...
latest: Pulling from registry.access.redhat.com/rhel7/etcd
9cadd93b16ff: Pull complete
4aa565ad8b7a: Pull complete
fd964436f70c: Pull complete
Digest: sha256:ad158d2c23050eee1bc09faa291e7177a027670f4913c2cee6182a8a09d041bf
+ docker tag registry.access.redhat.com/rhel7/etcd:latest registry.example.com:5000/rhel7/etcd:latest
+ docker push registry.example.com:5000/rhel7/etcd:latest
The push refers to a repository [registry.example.com:5000/rhel7/etcd]
43f884ff9b52: Pushed
02404b4d7e5d: Pushed
e1d829eddb62: Pushed
latest: digest: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732 size: 949
+ docker rmi registry.access.redhat.com/rhel7/etcd:latest
Untagged: registry.access.redhat.com/rhel7/etcd:latest
Untagged: registry.access.redhat.com/rhel7/etcd@sha256:ad158d2c23050eee1bc09faa291e7177a027670f4913c2cee6182a8a09d041bf
# docker rmi registry.example.com:5000/rhel7/etcd; docker pull registry.example.com:5000/rhel7/etcd
Untagged: registry.example.com:5000/rhel7/etcd:latest
Untagged: registry.example.com:5000/rhel7/etcd@sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732
Deleted: sha256:5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9
Deleted: sha256:f66339b2019f32dc8dbd4546ca88863355c76a30ce068c8fb2aa4fb383b21d43
Deleted: sha256:0619a10062a93ae062e8b0dd68e961105048ca1aa3232de1ce58010701b6469c
Deleted: sha256:e1d829eddb62dc49f1c56dbf8acd0c71299b3996115399de853a9d66d81b822f
Using default tag: latest
Trying to pull repository registry.example.com:5000/rhel7/etcd ...
latest: Pulling from registry.example.com:5000/rhel7/etcd
9cadd93b16ff: Pull complete
4aa565ad8b7a: Pull complete
fd964436f70c: Pull complete
Digest: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732
# skopeo --debug delete --tls-verify=false docker://registry.example.com:5000/rhel7/etcd:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000] Using "default-docker" configuration
DEBU[0000] Using file:///var/lib/atomic/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.example.com:5000
DEBU[0000] GET https://registry.example.com:5000/v2/
DEBU[0000] Ping https://registry.example.com:5000/v2/ err &url.Error{Op:"Get", URL:"https://registry.example.com:5000/v2/", Err:(*errors.errorString)(0xc420166010)}
DEBU[0000] GET http://registry.example.com:5000/v2/
DEBU[0000] Ping http://registry.example.com:5000/v2/ err <nil>
DEBU[0000] Ping http://registry.example.com:5000/v2/ status 200
DEBU[0000] GET http://registry.example.com:5000/v2/rhel7/etcd/manifests/latest
DEBU[0000] DELETE http://registry.example.com:5000/v2/rhel7/etcd/manifests/sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732
DEBU[0000] Deleting /var/lib/atomic/sigstore/rhel7/etcd@sha256=473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732/signature-1
# docker rmi registry.example.com:5000/rhel7/etcd; docker pull registry.example.com:5000/rhel7/etcd
Untagged: registry.example.com:5000/rhel7/etcd:latest
Untagged: registry.example.com:5000/rhel7/etcd@sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732
Deleted: sha256:5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9
Deleted: sha256:f66339b2019f32dc8dbd4546ca88863355c76a30ce068c8fb2aa4fb383b21d43
Deleted: sha256:0619a10062a93ae062e8b0dd68e961105048ca1aa3232de1ce58010701b6469c
Deleted: sha256:e1d829eddb62dc49f1c56dbf8acd0c71299b3996115399de853a9d66d81b822f
Using default tag: latest
Trying to pull repository registry.example.com:5000/rhel7/etcd ...
Pulling repository registry.example.com:5000/rhel7/etcd
Error: image rhel7/etcd:latest not found
# registry garbage-collect /etc/docker-distribution/registry/config.yml
[...]
0 blobs marked, 5 blobs eligible for deletion
blob eligible for deletion: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/47/473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732 go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/47/473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732") go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=188.048µs trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=92183615-45fe-4598-bea2-8c434e07c268 trace.line=177
blob eligible for deletion: sha256:4aa565ad8b7a87248163ce7dba1dd3894821aac97e846b932ff6b8ef9a8a508a
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/4a/4aa565ad8b7a87248163ce7dba1dd3894821aac97e846b932ff6b8ef9a8a508a go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/4a/4aa565ad8b7a87248163ce7dba1dd3894821aac97e846b932ff6b8ef9a8a508a") go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=169.397µs trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=62a016d4-59a6-4270-8ad9-01f66a9b7cf8 trace.line=177
blob eligible for deletion: sha256:5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/5a/5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9 go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/5a/5a98b74c0806eb6e94ed009b3d792c8d6411ce62c608b04caef8eea6c887bce9") go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=182.572µs trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=cfd42666-b39c-45ea-a662-9942e9e620fe trace.line=177
blob eligible for deletion: sha256:9cadd93b16ff2a0c51ac967ea2abfadfac50cfa3af8b5bf983d89b8f8647f3e4
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/9c/9cadd93b16ff2a0c51ac967ea2abfadfac50cfa3af8b5bf983d89b8f8647f3e4 go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/9c/9cadd93b16ff2a0c51ac967ea2abfadfac50cfa3af8b5bf983d89b8f8647f3e4") go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=13.956331ms trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=f72c14ba-cfbf-4586-982d-eed220f1000e trace.line=177
blob eligible for deletion: sha256:fd964436f70c99bd7997b0fc574767355089ed4bd42eeb3ee0bd5dce29f4c55f
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/fd/fd964436f70c99bd7997b0fc574767355089ed4bd42eeb3ee0bd5dce29f4c55f go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry
DEBU[0000] filesystem.Delete("/docker/registry/v2/blobs/sha256/fd/fd964436f70c99bd7997b0fc574767355089ed4bd42eeb3ee0bd5dce29f4c55f") go.version=go1.8.3 instance.id=8cfdb137-79d7-407f-bde6-0430295369c8 service=registry trace.duration=3.228001ms trace.file=/builddir/build/BUILD/distribution-48294d928ced5dd9b378f7fd7c6f5da3ff3f2c89/_build/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete trace.id=558e6ab5-d99b-42c4-82bd-50bc2b05daa6 trace.line=177
# registry garbage-collect /etc/docker-distribution/registry/config.yml
[...]
0 blobs marked, 0 blobs eligible for deletion
# bash -x test.sh
+ UPSTREAM=registry.access.redhat.com
+ REGISTRY=registry.example.com:5000
+ img=rhel7/etcd
+ tag=latest
+ docker pull registry.access.redhat.com/rhel7/etcd:latest
Trying to pull repository registry.access.redhat.com/rhel7/etcd ...
latest: Pulling from registry.access.redhat.com/rhel7/etcd
9cadd93b16ff: Pull complete
4aa565ad8b7a: Pull complete
fd964436f70c: Pull complete
Digest: sha256:ad158d2c23050eee1bc09faa291e7177a027670f4913c2cee6182a8a09d041bf
+ docker tag registry.access.redhat.com/rhel7/etcd:latest registry.example.com:5000/rhel7/etcd:latest
+ docker push registry.example.com:5000/rhel7/etcd:latest
The push refers to a repository [registry.example.com:5000/rhel7/etcd]
43f884ff9b52: Layer already exists
02404b4d7e5d: Layer already exists
e1d829eddb62: Layer already exists
latest: digest: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732 size: 949
+ docker rmi registry.access.redhat.com/rhel7/etcd:latest
Untagged: registry.access.redhat.com/rhel7/etcd:latest
Untagged: registry.access.redhat.com/rhel7/etcd@sha256:ad158d2c23050eee1bc09faa291e7177a027670f4913c2cee6182a8a09d041bf
# systemctl restart docker-distribution
# docker push registry.example.com:5000/rhel7/etcd:latest
The push refers to a repository [registry.example.com:5000/rhel7/etcd]
43f884ff9b52: Pushed
02404b4d7e5d: Pushed
e1d829eddb62: Pushed
latest: digest: sha256:473e59e77c4a8edb6bcb58089e467a4439cdcaf24151b5ba3c6869858f1e7732 size: 949
If this is actually a problem then I'd say it's in docker-distribution
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. |
Description of problem: This seems to be reproducible: # systemctl stop docker-distribution # /bin/rm -rf /var/lib/registry/* # systemctl start docker-distribution # sh -x ./t + UPSTREAM=registry.access.redhat.com + REGISTRY=registry.example.com:5000 + img=rhel7/etcd + tag=latest + docker pull registry.access.redhat.com/rhel7/etcd:latest Trying to pull repository registry.access.redhat.com/rhel7/etcd ... sha256:bdc7602a49b66fbbab7517c42b57fa185df653608d3ed3b1ad04266425c842d0: Pulling from registry.access.redhat.com/rhel7/etcd be1e04c6fbff: Already exists 449ed11d916a: Already exists 7bb86ab4ce06: Already exists Digest: sha256:bdc7602a49b66fbbab7517c42b57fa185df653608d3ed3b1ad04266425c842d0 Status: Image is up to date for registry.access.redhat.com/rhel7/etcd:latest + docker tag registry.access.redhat.com/rhel7/etcd:latest registry.example.com:5000/rhel7/etcd:latest + docker push registry.example.com:5000/rhel7/etcd:latest The push refers to a repository [registry.example.com:5000/rhel7/etcd] 4a4cd6ca7b22: Pushed bb4f52dd78f6: Pushed f1f88d1c363a: Pushed latest: digest: sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d size: 949 + docker rmi registry.example.com:5000/rhel7/etcd:latest Untagged: registry.example.com:5000/rhel7/etcd:latest Untagged: registry.example.com:5000/rhel7/etcd@sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d At this point on a client this works as expected: # docker rmi registry.example.com:5000/rhel7/etcd ; docker pull rhel7/etcd Then: # skopeo delete docker://registry.example.com:5000/rhel7/etcd:latest # registry garbage-collect /etc/docker-distribution/registry/config.yml rhel7/etcd 0 blobs marked, 5 blobs eligible for deletion blob eligible for deletion: sha256:10f2056a02b666f19c83a2a8cb9591efd8121cb863a8e31124b8c4b3abe3ca6d blob eligible for deletion: sha256:5b1d750aaf067bda7021df8f39b78f89cd38edb227317640d0e7238f5f491fde blob eligible for deletion: sha256:7385c0dcabcfe94281ba4c47eb83fe40b51eb4436a2bf243623510dc7173853d blob eligible for deletion: sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d blob eligible for deletion: sha256:a7ef92bed2ae829b1a854e57e355e592ff0fab5ee5fdd2885aae48d6edf922ad # registry garbage-collect /etc/docker-distribution/registry/config.yml rhel7/etcd 0 blobs marked, 0 blobs eligible for deletion # ./t Trying to pull repository registry.access.redhat.com/rhel7/etcd ... sha256:bdc7602a49b66fbbab7517c42b57fa185df653608d3ed3b1ad04266425c842d0: Pulling from registry.access.redhat.com/rhel7/etcd be1e04c6fbff: Already exists 449ed11d916a: Already exists 7bb86ab4ce06: Already exists Digest: sha256:bdc7602a49b66fbbab7517c42b57fa185df653608d3ed3b1ad04266425c842d0 Status: Image is up to date for registry.access.redhat.com/rhel7/etcd:latest The push refers to a repository [registry.example.com:5000/rhel7/etcd] 4a4cd6ca7b22: Layer already exists bb4f52dd78f6: Layer already exists f1f88d1c363a: Layer already exists latest: digest: sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d size: 949 Untagged: registry.example.com:5000/rhel7/etcd:latest Untagged: registry.example.com:5000/rhel7/etcd@sha256:79b01cb9feff6a28cdb40e23c07229fb95f5d8d3830d3d9b8203c10b802daa9d At this point it would seem that rhel7/etcd is pushed but the earlier client command fails. Doing delete/pull/push/gc in any order seems not to restore the registry to a working state any more. Unclear whether this is a skopeo or docker-distribution issue so filing initially against skopeo. Version-Release number of selected component (if applicable): docker-1.12.6-48.git0fdc778.el7.x86_64 docker-distribution-2.6.1-1.1.gita25b9ef.el7.x86_64 skopeo-0.1.20-2.1.gite802625.el7.x86_64 Additional information: # cat /etc/docker-distribution/registry/config.yml version: 0.1 log: level: warn fields: service: registry storage: cache: layerinfo: inmemory delete: enabled: true filesystem: rootdirectory: /var/lib/registry http: addr: :5000 relativeurls: true secret: changeme tls: certificate: /etc/docker-distribution/registry/cert.cert key: /etc/docker-distribution/registry/cert.key #clientcas: /etc/docker-distribution/registry/ca.cert