Bug 1300570
Summary: | Image garbage collection setting should be more specific for different disk configuration. | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Johnny Liu <jialiu> |
Component: | Node | Assignee: | Derek Carr <decarr> |
Status: | CLOSED ERRATA | QA Contact: | DeShuai Ma <dma> |
Severity: | medium | Docs Contact: | |
Priority: | medium | ||
Version: | 3.1.0 | CC: | agoldste, aos-bugs, decarr, haowang, jdyson, jokerman, lmeyer, mmccomas, mmilata, pbabinca, pep, tdawson, xxia |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2016-05-12 16:27:08 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1267746 |
Description
Johnny Liu
2016-01-21 07:44:09 UTC
Note: the registry has no impact on Kubelet image garbage collection. Kubelet image garbage collection only targets images stored in the Docker daemon's graph storage. You don't need to have a registry deployed to test image garbage collection, and if you do have a registry deployed, its storage configuration (emptyDir vs hostPath vs NFS vs PV) does not affect node image storage or garbage collection. Possible fix here: https://github.com/google/cadvisor/issues/944#issuecomment-173665149. Waiting on more discussion with upstream before I proceed further. cadvisor PR: https://github.com/google/cadvisor/pull/1070 Once this is merged, we'll need PRs for Kubernetes and Origin to pull in the updated cadvisor. Or we may cherry-pick the fix into Origin in the short term if we aren't comfortable bumping all of cadvisor for this fix. Kube PRs: https://github.com/kubernetes/kubernetes/pull/19354, https://github.com/kubernetes/kubernetes/pull/20395 https://github.com/kubernetes/kubernetes/pull/19354 - merged 1/29 https://github.com/kubernetes/kubernetes/pull/20395 - not yet merged, but tagged lgtm Taking bug in Andy's absence and will look to cherry-pick. https://github.com/kubernetes/kubernetes/pull/20395 just merged upstream. This will be picked up in the next rebase into Origin. Will be in next puddle Verify on openshift v3.1.1.904 [root@openshift-135 ~]# openshift version openshift v3.1.1.904 kubernetes v1.2.0-alpha.7-703-gbc4550d etcd 2.2.5 [root@openshift-129 ~]# cat /etc/sysconfig/docker-storage DOCKER_STORAGE_OPTIONS=--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/rhel72-docker--pool --storage-opt dm.use_deferred_removal=true [root@openshift-129 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool rhel72 twi-aot--- 17.44g 20.99 5.33 root rhel72 -wi-ao---- 10.00g swap rhel72 -wi-ao---- 2.00g atomic-openshift-node logs: I0222 15:01:45.411470 27751 image_manager.go:230] [ImageManager]: Disk usage on "rhel72-docker--pool" () is at 70% which is over the high threshold (20%). Trying to free 11319902208 bytes I0222 15:01:45.508647 27751 docker.go:357] Docker Container: /atomic-openshift-node is not managed by kubelet. I0222 15:01:45.508679 27751 docker.go:357] Docker Container: /openvswitch is not managed by kubelet. I0222 15:01:45.508687 27751 docker.go:357] Docker Container: /small_wozniak is not managed by kubelet. I0222 15:01:45.508914 27751 image_manager.go:287] [ImageManager]: Removing image "0192cfcebeb04ff778cf44aa7f6d336e43ee9fdc6cfb02de091364248a700cfc" to free 490397531 bytes I0222 15:01:45.651197 27751 docker.go:357] Docker Container: /atomic-openshift-node is not managed by kubelet. I0222 15:01:45.651227 27751 docker.go:357] Docker Container: /openvswitch is not managed by kubelet. I0222 15:01:45.651235 27751 docker.go:357] Docker Container: /small_wozniak is not managed by kubelet. I0222 15:01:45.655037 27751 kubelet.go:2409] SyncLoop (housekeeping) I0222 15:01:45.668928 27751 image_manager.go:287] [ImageManager]: Removing image "0bbc57b809f12a1a21c8105fa428f714bee4c588e9d47beb4b053bccffb68416" to free 603628030 bytes Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2016:1064 |