Bug 1577739

Summary: Image Garbage collection trying to delete images in use by stopped containers
Product: OpenShift Container Platform Reporter: Venkata Tadimarri <ktadimar>
Component: NodeAssignee: Seth Jennings <sjenning>
Status: CLOSED ERRATA QA Contact: DeShuai Ma <dma>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.6.0CC: aos-bugs, bleanhar, cstark, jokerman, mmccomas
Target Milestone: ---Keywords: Reopened
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Prevents image garbage collection from attempting to remove images in use by containers
 Story Points: ---
Clone Of:
: 1580551 1580552 1580554 1580555 (view as bug list) Environment:
Last Closed: 2018-07-30 19:15:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1580551, 1580552, 1580554, 1580555, 1619477    
Attachments:
Description Flags
node.log none

Description Venkata Tadimarri 2018-05-14 05:07:12 UTC 
Description of problem:

Image Garbage collection is trying to delete images in use by stopped containers. 

journalctl_--no-pager_--unit_docker:Apr 30 15:25:14 drlosn05.example.com dockerd-current[1521]: time="2018-04-30T15:25:14.524633778+12:00" level=error msg="Handler for DELETE /v1.24/images/sha256:41ad666f
e1b52e4a5ee47042bb3e6f53a3ae9cec6edf86960b1337de8e07d651 returned error: conflict: unable to delete 41ad666fe1b5 (must be forced) - image is being used by stopped container 7ea6cb10acce"
journalctl_--no-pager_--unit_docker:Apr 30 15:25:14 drlosn05.example.com dockerd-current[1521]: time="2018-04-30T15:25:14.524780645+12:00" level=error msg="Handler for DELETE /v1.24/images/sha256:41ad666f
e1b52e4a5ee47042bb3e6f53a3ae9cec6edf86960b1337de8e07d651 returned error: conflict: unable to delete 41ad666fe1b5 (must be forced) - image is being used by stopped container 7ea6cb10acce"
journalctl_--no-pager_--unit_docker:Apr 30 15:25:14 drlosn05.example.com dockerd-current[1521]: time="2018-04-30T15:25:14.545644810+12:00" level=error msg="Handler for DELETE /v1.24/images/sha256:d58c2b65
f4d51dbe67ce04a247f38d2169b078306469ed64ead077a1d6df031a returned error: conflict: unable to delete d58c2b65f4d5 (must be forced) - image is being used by stopped container 0aaec7ec9df2"
journalctl_--no-pager_--unit_docker:Apr 30 15:25:14 drlosn05.example.com dockerd-current[1521]: time="2018-04-30T15:25:14.545735698+12:00" level=error msg="Handler for DELETE /v1.24/images/sha256:d58c2b65
f4d51dbe67ce04a247f38d2169b078306469ed64ead077a1d6df031a returned error: conflict: unable to delete d58c2b65f4d5 (must be forced) - image is being used by stopped container 0aaec7ec9df2"



Version-Release number of selected component (if applicable):
ansible-2.4.2.0-2.el7.noarch 
openshift-ansible-3.4.95-1.git.0.3c99da3.el7.noarch
atomic-openshift-3.6.173.0.113-1.git.0.65fb9fb.el7.x86_64



Actual results:

Images in use by the stopped containers are being considered by GC for deletion. 

Expected results:

Images in use should not be included in the images evaluated for deletion.

Additional info:

https://github.com/kubernetes/kubernetes/pull/57020 


Issue reported here: 

https://github.com/kubernetes/kubernetes/issues/54440
Comment 1 Venkata Tadimarri 2018-05-14 22:34:09 UTC 
Attached the node-config.yaml file as well for GC config and other settings.
Comment 10 DeShuai Ma 2018-06-05 10:09:31 UTC 
Created attachment 1447795 [details]
node.log
Comment 11 DeShuai Ma 2018-06-05 10:10:16 UTC 
Verify on v3.10.0-0.58.0
detail log see the attached log.
Comment 14 errata-xmlrpc 2018-07-30 19:15:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816