Red Hat Bugzilla – Bug 1292964
OpenShift doesn't notice that Docker Storage is, or is reaching that state of being, full
Last modified: 2017-03-08 13:43 EST
Description of problem:
OpenShift doesn't notice that Docker Storage is, or is reaching that state of being, full. This allows the storage to fill up and no one be the wiser, and then the docker service can, and will likely, fail, preventing further use of the node.
Version-Release number of selected component (if applicable):
Found on OSE 3.0, but cannot find evidence that it is fixed in 3.1
Steps to Reproduce:
1. Spin up normal OSE 3.0 environment (infrastructure of the env does not appears to impact it)
2. Use like normal, pull images and fill up storage
Either the docker service completely fails or it pretends to work but will not properly pull images into pods leaving pods stuck in a pending state
As you pull images, you get an update on storage levels.
This is only so useful if you can clear up the docker storage, hence blocks on bug 1292845.
GH issue: https://github.com/openshift/origin/issues/6350
So I reported this originally, and as pointed out it blocks on the image clean up. I had to manually run clean up steps (specifically the image ones) from the OSE 3 docs in order to get it to actually clean up after itself. I have not tried running the ansible playbooks, but overall it seems like the clean up process should be happening automatically at least semi scheduled. Which in my case did not seem to be the case
Has this been fixed in 3.1 with this PR?
No, we have no fix yet.
This work is being done upstream. According to a proposal , everything we need (volume accounting) shall be covered. As of now, only the volume interface  is in place. Unfortunatelly, accounting for the host_path volumes has been recently disabled  due to high CPU load. Neither NFS nor AWS nor GCE is supported yet.
This is a new feature in Kubernetes 1.4 that just got merged.
You can read the feature description here:
Users will be able to set thresholds for both the rootfs (i.e. nodefs) and the imagefs (i.e. docker storage). If those thresholds are met, the node will report disk pressure, perform image gc, and evict pods on the node to reduce disk pressure to a stable state. While the node reports disk pressure, no additional pods are admitted to the node for execution.
Marking this upcoming release.
I am assuming these are going to be exposed in some way in the node configs?
Boris - correct, in 3.4, users will be able to configure the values in node-config.
OCP 3.4 has added support to handle disk pressure based on the work we did in upstream Kubernetes 1.4.
I am moving this to ON_QA as a result.
Test on openshift v18.104.22.168+9c963ec, disk pressure works as expected.
detail in the card. https://trello.com/c/3LvGAHr3/371-5-kubelet-evicts-pods-when-low-on-disk-node-reliability
Verify this bug.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.