Bug 1577739 - Image Garbage collection trying to delete images in use by stopped containers
Summary: Image Garbage collection trying to delete images in use by stopped containers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.10.0
Assignee: Seth Jennings
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On:
Blocks: 1580554 1580551 1580552 1580555 1619477
TreeView+ depends on / blocked
 
Reported: 2018-05-14 05:07 UTC by Venkata Tadimarri
Modified: 2018-08-21 01:44 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Prevents image garbage collection from attempting to remove images in use by containers
Clone Of:
: 1580551 1580552 1580554 1580555 (view as bug list)
Environment:
Last Closed: 2018-07-30 19:15:23 UTC


Attachments (Terms of Use)
node.log (2.11 MB, application/x-gzip)
2018-06-05 10:09 UTC, DeShuai Ma
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1816 None None None 2018-07-30 19:15:53 UTC

Description Venkata Tadimarri 2018-05-14 05:07:12 UTC
Description of problem:

Image Garbage collection is trying to delete images in use by stopped containers. 

journalctl_--no-pager_--unit_docker:Apr 30 15:25:14 drlosn05.example.com dockerd-current[1521]: time="2018-04-30T15:25:14.524633778+12:00" level=error msg="Handler for DELETE /v1.24/images/sha256:41ad666f
e1b52e4a5ee47042bb3e6f53a3ae9cec6edf86960b1337de8e07d651 returned error: conflict: unable to delete 41ad666fe1b5 (must be forced) - image is being used by stopped container 7ea6cb10acce"
journalctl_--no-pager_--unit_docker:Apr 30 15:25:14 drlosn05.example.com dockerd-current[1521]: time="2018-04-30T15:25:14.524780645+12:00" level=error msg="Handler for DELETE /v1.24/images/sha256:41ad666f
e1b52e4a5ee47042bb3e6f53a3ae9cec6edf86960b1337de8e07d651 returned error: conflict: unable to delete 41ad666fe1b5 (must be forced) - image is being used by stopped container 7ea6cb10acce"
journalctl_--no-pager_--unit_docker:Apr 30 15:25:14 drlosn05.example.com dockerd-current[1521]: time="2018-04-30T15:25:14.545644810+12:00" level=error msg="Handler for DELETE /v1.24/images/sha256:d58c2b65
f4d51dbe67ce04a247f38d2169b078306469ed64ead077a1d6df031a returned error: conflict: unable to delete d58c2b65f4d5 (must be forced) - image is being used by stopped container 0aaec7ec9df2"
journalctl_--no-pager_--unit_docker:Apr 30 15:25:14 drlosn05.example.com dockerd-current[1521]: time="2018-04-30T15:25:14.545735698+12:00" level=error msg="Handler for DELETE /v1.24/images/sha256:d58c2b65
f4d51dbe67ce04a247f38d2169b078306469ed64ead077a1d6df031a returned error: conflict: unable to delete d58c2b65f4d5 (must be forced) - image is being used by stopped container 0aaec7ec9df2"



Version-Release number of selected component (if applicable):
ansible-2.4.2.0-2.el7.noarch 
openshift-ansible-3.4.95-1.git.0.3c99da3.el7.noarch
atomic-openshift-3.6.173.0.113-1.git.0.65fb9fb.el7.x86_64



Actual results:

Images in use by the stopped containers are being considered by GC for deletion. 

Expected results:

Images in use should not be included in the images evaluated for deletion.

Additional info:

https://github.com/kubernetes/kubernetes/pull/57020 


Issue reported here: 

https://github.com/kubernetes/kubernetes/issues/54440

Comment 1 Venkata Tadimarri 2018-05-14 22:34:09 UTC
Attached the node-config.yaml file as well for GC config and other settings.

Comment 10 DeShuai Ma 2018-06-05 10:09:31 UTC
Created attachment 1447795 [details]
node.log

Comment 11 DeShuai Ma 2018-06-05 10:10:16 UTC
Verify on v3.10.0-0.58.0
detail log see the attached log.

Comment 14 errata-xmlrpc 2018-07-30 19:15:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816


Note You need to log in before you can comment on or make changes to this bug.