Bug 1647348

garbage collection is owned by the pod/node team.  transferring.

*** Bug 1688339 has been marked as a duplicate of this bug. ***

@mmariyan,  @Seth Jennings  
Can you confirm the reproduced steps using docker of which version? 
I can not reproduce this bug with the steps in PR:  https://github.com/kubernetes/kubernetes/pull/70647 , which mean there is no image exist which have one tag but two repositories digest. 

version:
oc v3.11.98
docker 1.13.1 (docker-1.13.1-94.gitb2f74b2.el7.x86_64)

steps:
docker pull centos:7
docker tag centos:7 example.com:1234/centos:latest
docker push example.com:1234/centos:latest
docker rmi example.com:1234/centos:latest
docker pull example.com:1234/centos:latest

result:
#docker image inspect centos:7 
[
    {
        "Id": "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1",
        "RepoTags": [
            "docker.io/centos:7",
            "example.com:1234/centos:latest"
        ],
        "RepoDigests": [
            "docker.io/centos@sha256:8d487d68857f5bc9595793279b33d082b03713341ddec91054382641d14db861",
            "example.com:1234/centos@sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2019-03-14T21:19:53.361167852Z",
        "Container": "958baf5225f586da9c70a21e911a0a875402dd22d83133d78b3b3aa6130e7892",
...

Note: this result show one image has two tags and two RepoDigests!

@Ge Liu,  Sunlic will follow up minmli's work

@Weinan Liu, Great, thx
@ Sunil Choudhary ， is there any update for this bug? it block the 3.11 z-steram release, thanks.

Ok, the recreate is even simpler than upstream

$ docker pull centos:7

$ docker tag centos:7 centos:7.6

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              7.6                 9f38484d220f        2 weeks ago         202 MB
docker.io/centos    7                   9f38484d220f        2 weeks ago         202 MB

$ docker rmi 9f38484d220f
Error response from daemon: conflict: unable to delete 9f38484d220f (must be forced) - image is referenced in multiple repositories

The issue arises when you try to delete an image by id that has tags/digests from more that one repo referencing it.

However, in order to verify the fix, you'll need to do this through the kubelet since the fix to the dockershim now untags all tags/digests from the image before trying to delete it by id.

I also don't see a 3.11 build tag that contains the 3.11 fix yet.  Moving back to MODIFIED.

$ git fetch origin --tags

$ git tag --contains b28850a9a92b8570a815297a74404aab6cad3fee
<nothing>

Gah my bad.  Was looking at origin not ose.
$ git tag --contains b28850a9a9
v3.11.100-1
v3.11.101-1
v3.11.102-1
v3.11.103-1
v3.11.96-1
v3.11.97-1
v3.11.98-1
v3.11.99-1

Verified on v3.11.100. atomic-openshift-node service is able to delete images with multiple RepoTags & RepoDigest.

# oc version
oc v3.11.100
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

# rpm -qa | grep -i atomic-openshift-node
atomic-openshift-node-3.11.100-1.git.0.f303dac.el7.x86_64

# cat docker-image-inspect-2bcb04bdb83f
[
    {
        "Id": "sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c",
        "RepoTags": [
            "docker-registry-default.apps.0328-8ex.example.com/sunilc/nginx:1",
            "docker.io/nginx:latest",
            "nginx:1"
        ],
        "RepoDigests": [
            "docker-registry-default.apps.0328-8ex.example.com/sunilc/nginx@sha256:dabecc7dece2fff98fb00add2f0b525b7cd4a2cacddcc27ea4a15a7922ea47ea",
            "docker.io/nginx@sha256:c8a861b8a1eeef6d48955a6c6d5dff8e2580f13ff4d0f549e082e7c82a8617a2"
...

# cat docker-image-inspect-9f38484d220f
[
    {
        "Id": "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1",
        "RepoTags": [
            "centos:7",
            "centos:7.6",
            "docker-registry-default.apps.0328-8ex.example.com/sunilc/centos:7",
            "docker.io/centos:7",
            "docker.io/centos:centos7"
        ],
        "RepoDigests": [
            "docker-registry-default.apps.0328-8ex.example.com/sunilc/centos@sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66",
            "docker.io/centos@sha256:8d487d68857f5bc9595793279b33d082b03713341ddec91054382641d14db861"
...

From atomic-openshift-node logs:
[...]
Apr 02 00:34:00 node1.example.com atomic-openshift-node[59238]: I0402 00:34:00.255486   59238 image_gc_manager.go:355] Evaluating image ID sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1 for possible garbage collection
Apr 02 00:34:00 node1.example.com atomic-openshift-node[59238]: I0402 00:34:00.255500   59238 image_gc_manager.go:371] [imageGCManager]: Removing image "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1" to free 201782942 bytes
Apr 02 00:34:06 node1.example.com atomic-openshift-node[59238]: I0402 00:34:06.195529   59238 image_gc_manager.go:355] Evaluating image ID sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c for possible garbage collection
Apr 02 00:34:06 node1.example.com atomic-openshift-node[59238]: I0402 00:34:06.195558   59238 image_gc_manager.go:371] [imageGCManager]: Removing image "sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c" to free 109294563 bytes
[...]

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0636