Description of problem: The dm.min_free_space defaults to 10%, which "specifies the min free space percent in a thin pool require for new device creation to succeed....Whenever a new a thin pool device is created (during docker pull or during container creation), the Engine checks if the minimum free space is available. If sufficient space is unavailable, then device creation fails and any relevant docker operation fails." [1] This setting is preventing the storage usage to cross the 90% limit. However, image GC is expected to kick in only beyond image-gc-high-threshold. The image-gc-high-threshold has a default value of 90%, and hence GC never triggers. If image-gc-high-threshold is set to a value lower than (100 - dm.min_free_space)%, GC triggers. https://github.com/docker/docker/blob/master/docs/reference/commandline/dockerd.md#storage-driver-options How reproducible: Everytime. Steps to Reproduce: Allow images on disk consumption % to exceed image-gc-high-threshold, with Kubernetes and docker defaults. Actual results: GC does not trigger due to conflicting defaults between Kubernetes (image-gc-high-threshold) and Docker (dm.min_free_space) Expected results: Same as actual. Suggestion is to set out-of-the-box image-gc-high-threshold default to a value lower than 90%, to prevent conflicting defaults.
Upstream PR: https://github.com/kubernetes/kubernetes/pull/40432
Origin PR: https://github.com/openshift/origin/pull/12762
This has been merged into ocp and is in OCP v3.5.0.17 or newer.
Verified. openshift v3.5.0.18+9a5d1aa kubernetes v1.5.2+43a9be4 1. Check default value of node-config.yaml image-gc-high-threshold: - '85' 2. Docker pull images in node, let Disk usage on "rhel-docker--pool" more than 85%, and less than 90%. 3. Check images count and node log [root@qe-chezhang-node-registry-router-2 ~]# docker images | wc -l 15 [root@qe-chezhang-node-registry-router-2 ~]# journalctl -u atomic-openshift-node | grep "over the high threshold" Feb 10 01:10:12 qe-chezhang-node-registry-router-2 atomic-openshift-node[26506]: I0210 01:10:12.321159 26570 image_gc_manager.go:270] [imageGCManager]: Disk usage on "rhel-docker--pool" () is at 88% which is over the high threshold (85%). Trying to free 1523580928 bytes 4. Check images count again, some (oldest)images were removed. [root@qe-chezhang-node-registry-router-2 ~]# docker images | wc -l 12
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0884