Bug 1647348 - image garbage collection fails to remove an image
Summary: image garbage collection fails to remove an image
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 3.11.z
Assignee: Seth Jennings
QA Contact: Sunil Choudhary
URL:
Whiteboard:
: 1688339 (view as bug list)
Depends On:
Blocks: 1691333
TreeView+ depends on / blocked
 
Reported: 2018-11-07 09:16 UTC by mmariyan
Modified: 2020-01-08 15:15 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Image garbage collection no longer fails for images with only one tag but more than one repository associated
Clone Of:
: 1691333 1749930 (view as bug list)
Environment:
Last Closed: 2019-04-11 05:38:23 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0636 None None None 2019-04-11 05:38:38 UTC

Description mmariyan 2018-11-07 09:16:40 UTC
Description of problem:

image garbage collection fails to remove an image which is no longer used, error occurs when the image garbage collection gets executed and one or several images exist which have one tag but two remote repositories linked to it.

There is PR in upstream https://github.com/kubernetes/kubernetes/pull/70647
is this available with OCP? or will it be backported with ocp 3.9?

Version-Release number of selected component (if applicable):


Actual results:


Expected results:


Additional info:

Comment 1 Ben Parees 2018-11-07 14:51:38 UTC
garbage collection is owned by the pod/node team.  transferring.

Comment 10 Seth Jennings 2019-03-13 19:23:05 UTC
*** Bug 1688339 has been marked as a duplicate of this bug. ***

Comment 13 MinLi 2019-03-22 06:34:16 UTC
@mmariyan,  @Seth Jennings  
Can you confirm the reproduced steps using docker of which version? 
I can not reproduce this bug with the steps in PR:  https://github.com/kubernetes/kubernetes/pull/70647 , which mean there is no image exist which have one tag but two repositories digest. 

version:
oc v3.11.98
docker 1.13.1 (docker-1.13.1-94.gitb2f74b2.el7.x86_64)

steps:
docker pull centos:7
docker tag centos:7 example.com:1234/centos:latest
docker push example.com:1234/centos:latest
docker rmi example.com:1234/centos:latest
docker pull example.com:1234/centos:latest

result:
#docker image inspect centos:7 
[
    {
        "Id": "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1",
        "RepoTags": [
            "docker.io/centos:7",
            "example.com:1234/centos:latest"
        ],
        "RepoDigests": [
            "docker.io/centos@sha256:8d487d68857f5bc9595793279b33d082b03713341ddec91054382641d14db861",
            "example.com:1234/centos@sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2019-03-14T21:19:53.361167852Z",
        "Container": "958baf5225f586da9c70a21e911a0a875402dd22d83133d78b3b3aa6130e7892",
...

Note: this result show one image has two tags and two RepoDigests!

Comment 15 Weinan Liu 2019-03-28 08:12:25 UTC
@Ge Liu,  Sunlic will follow up minmli's work

Comment 16 ge liu 2019-04-01 02:36:58 UTC
@Weinan Liu, Great, thx
@ Sunil Choudhary , is there any update for this bug? it block the 3.11 z-steram release, thanks.

Comment 17 Seth Jennings 2019-04-01 15:33:11 UTC
Ok, the recreate is even simpler than upstream

$ docker pull centos:7

$ docker tag centos:7 centos:7.6

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              7.6                 9f38484d220f        2 weeks ago         202 MB
docker.io/centos    7                   9f38484d220f        2 weeks ago         202 MB

$ docker rmi 9f38484d220f
Error response from daemon: conflict: unable to delete 9f38484d220f (must be forced) - image is referenced in multiple repositories

The issue arises when you try to delete an image by id that has tags/digests from more that one repo referencing it.

Comment 18 Seth Jennings 2019-04-01 15:34:34 UTC
However, in order to verify the fix, you'll need to do this through the kubelet since the fix to the dockershim now untags all tags/digests from the image before trying to delete it by id.

Comment 19 Seth Jennings 2019-04-01 16:41:34 UTC
I also don't see a 3.11 build tag that contains the 3.11 fix yet.  Moving back to MODIFIED.

$ git fetch origin --tags

$ git tag --contains b28850a9a92b8570a815297a74404aab6cad3fee
<nothing>

Comment 20 Seth Jennings 2019-04-01 17:08:50 UTC
Gah my bad.  Was looking at origin not ose.
$ git tag --contains b28850a9a9
v3.11.100-1
v3.11.101-1
v3.11.102-1
v3.11.103-1
v3.11.96-1
v3.11.97-1
v3.11.98-1
v3.11.99-1

Comment 22 Sunil Choudhary 2019-04-02 04:56:08 UTC
Verified on v3.11.100. atomic-openshift-node service is able to delete images with multiple RepoTags & RepoDigest.

# oc version
oc v3.11.100
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

# rpm -qa | grep -i atomic-openshift-node
atomic-openshift-node-3.11.100-1.git.0.f303dac.el7.x86_64

# cat docker-image-inspect-2bcb04bdb83f
[
    {
        "Id": "sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c",
        "RepoTags": [
            "docker-registry-default.apps.0328-8ex.example.com/sunilc/nginx:1",
            "docker.io/nginx:latest",
            "nginx:1"
        ],
        "RepoDigests": [
            "docker-registry-default.apps.0328-8ex.example.com/sunilc/nginx@sha256:dabecc7dece2fff98fb00add2f0b525b7cd4a2cacddcc27ea4a15a7922ea47ea",
            "docker.io/nginx@sha256:c8a861b8a1eeef6d48955a6c6d5dff8e2580f13ff4d0f549e082e7c82a8617a2"
...

# cat docker-image-inspect-9f38484d220f
[
    {
        "Id": "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1",
        "RepoTags": [
            "centos:7",
            "centos:7.6",
            "docker-registry-default.apps.0328-8ex.example.com/sunilc/centos:7",
            "docker.io/centos:7",
            "docker.io/centos:centos7"
        ],
        "RepoDigests": [
            "docker-registry-default.apps.0328-8ex.example.com/sunilc/centos@sha256:ca58fe458b8d94bc6e3072f1cfbd334855858e05e1fd633aa07cf7f82b048e66",
            "docker.io/centos@sha256:8d487d68857f5bc9595793279b33d082b03713341ddec91054382641d14db861"
...

From atomic-openshift-node logs:
[...]
Apr 02 00:34:00 node1.example.com atomic-openshift-node[59238]: I0402 00:34:00.255486   59238 image_gc_manager.go:355] Evaluating image ID sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1 for possible garbage collection
Apr 02 00:34:00 node1.example.com atomic-openshift-node[59238]: I0402 00:34:00.255500   59238 image_gc_manager.go:371] [imageGCManager]: Removing image "sha256:9f38484d220fa527b1fb19747638497179500a1bed8bf0498eb788229229e6e1" to free 201782942 bytes
Apr 02 00:34:06 node1.example.com atomic-openshift-node[59238]: I0402 00:34:06.195529   59238 image_gc_manager.go:355] Evaluating image ID sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c for possible garbage collection
Apr 02 00:34:06 node1.example.com atomic-openshift-node[59238]: I0402 00:34:06.195558   59238 image_gc_manager.go:371] [imageGCManager]: Removing image "sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c" to free 109294563 bytes
[...]

Comment 24 errata-xmlrpc 2019-04-11 05:38:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0636


Note You need to log in before you can comment on or make changes to this bug.