Hide Forgot
Description of problem: image garbage collection fails to remove an image which is no longer used, error occurs when the image garbage collection gets executed and one or several images exist which have one tag but two remote repositories linked to it. There is PR in upstream https://github.com/kubernetes/kubernetes/pull/70647 is this available with OCP? or will it be backported with ocp 3.9? Version-Release number of selected component (if applicable): Actual results: Expected results: Additional info:
garbage collection is owned by the pod/node team. transferring.
*** Bug 1688339 has been marked as a duplicate of this bug. ***
@mmariyan, @Seth Jennings Can you confirm the reproduced steps using docker of which version? I can not reproduce this bug with the steps in PR: https://github.com/kubernetes/kubernetes/pull/70647 , which mean there is no image exist which have one tag but two repositories digest. version: oc v3.11.98 docker 1.13.1 (docker-1.13.1-94.gitb2f74b2.el7.x86_64) steps: docker pull centos:7 docker tag centos:7 example.com:1234/centos:latest docker push example.com:1234/centos:latest docker rmi example.com:1234/centos:latest docker pull example.com:1234/centos:latest result: #docker image inspect centos:7 [ { "Id": "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1", "RepoTags": [ "docker.io/centos:7", "example.com:1234/centos:latest" ], "RepoDigests": [ "docker.io/centos@sha256:8d487d68857f5bc9595793279b33d082b03713341ddec91054382641d14db861", "example.com:1234/centos@sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66" ], "Parent": "", "Comment": "", "Created": "2019-03-14T21:19:53.361167852Z", "Container": "958baf5225f586da9c70a21e911a0a875402dd22d83133d78b3b3aa6130e7892", ... Note: this result show one image has two tags and two RepoDigests!
@Ge Liu, Sunlic will follow up minmli's work
@Weinan Liu, Great, thx @ Sunil Choudhary ， is there any update for this bug? it block the 3.11 z-steram release, thanks.
Ok, the recreate is even simpler than upstream $ docker pull centos:7 $ docker tag centos:7 centos:7.6 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE centos 7.6 9f38484d220f 2 weeks ago 202 MB docker.io/centos 7 9f38484d220f 2 weeks ago 202 MB $ docker rmi 9f38484d220f Error response from daemon: conflict: unable to delete 9f38484d220f (must be forced) - image is referenced in multiple repositories The issue arises when you try to delete an image by id that has tags/digests from more that one repo referencing it.
However, in order to verify the fix, you'll need to do this through the kubelet since the fix to the dockershim now untags all tags/digests from the image before trying to delete it by id.
I also don't see a 3.11 build tag that contains the 3.11 fix yet. Moving back to MODIFIED. $ git fetch origin --tags $ git tag --contains b28850a9a92b8570a815297a74404aab6cad3fee <nothing>
Gah my bad. Was looking at origin not ose. $ git tag --contains b28850a9a9 v3.11.100-1 v3.11.101-1 v3.11.102-1 v3.11.103-1 v3.11.96-1 v3.11.97-1 v3.11.98-1 v3.11.99-1
Verified on v3.11.100. atomic-openshift-node service is able to delete images with multiple RepoTags & RepoDigest. # oc version oc v3.11.100 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO # rpm -qa | grep -i atomic-openshift-node atomic-openshift-node-3.11.100-1.git.0.f303dac.el7.x86_64 # cat docker-image-inspect-2bcb04bdb83f [ { "Id": "sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c", "RepoTags": [ "docker-registry-default.apps.0328-8ex.example.com/sunilc/nginx:1", "docker.io/nginx:latest", "nginx:1" ], "RepoDigests": [ "docker-registry-default.apps.0328-8ex.example.com/sunilc/nginx@sha256:dabecc7dece2fff98fb00add2f0b525b7cd4a2cacddcc27ea4a15a7922ea47ea", "docker.io/nginx@sha256:c8a861b8a1eeef6d48955a6c6d5dff8e2580f13ff4d0f549e082e7c82a8617a2" ... # cat docker-image-inspect-9f38484d220f [ { "Id": "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1", "RepoTags": [ "centos:7", "centos:7.6", "docker-registry-default.apps.0328-8ex.example.com/sunilc/centos:7", "docker.io/centos:7", "docker.io/centos:centos7" ], "RepoDigests": [ "docker-registry-default.apps.0328-8ex.example.com/sunilc/centos@sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66", "docker.io/centos@sha256:8d487d68857f5bc9595793279b33d082b03713341ddec91054382641d14db861" ... From atomic-openshift-node logs: [...] Apr 02 00:34:00 node1.example.com atomic-openshift-node[59238]: I0402 00:34:00.255486 59238 image_gc_manager.go:355] Evaluating image ID sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1 for possible garbage collection Apr 02 00:34:00 node1.example.com atomic-openshift-node[59238]: I0402 00:34:00.255500 59238 image_gc_manager.go:371] [imageGCManager]: Removing image "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1" to free 201782942 bytes Apr 02 00:34:06 node1.example.com atomic-openshift-node[59238]: I0402 00:34:06.195529 59238 image_gc_manager.go:355] Evaluating image ID sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c for possible garbage collection Apr 02 00:34:06 node1.example.com atomic-openshift-node[59238]: I0402 00:34:06.195558 59238 image_gc_manager.go:371] [imageGCManager]: Removing image "sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c" to free 109294563 bytes [...]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0636