Description of problem: IHAC where the alert "KubePersistentVolumeFullInFourDays" is triggered for the logging-es-0 pvc will be filled within 4 days. However, running the command in the Prometheus shows that the storage is enough. The graph in the attachments shows that there is little spike when the value is below 0 (as in the alert), however, it stabilize on the correct value after couple of minutes. Labels alertname = KubePersistentVolumeFullInFourDays cluster = <cluster> endpoint = https-metrics instance = <node-ip>:10250 job = kubelet namespace = openshift-logging persistentvolumeclaim = logging-es-0 prometheus = openshift-monitoring/k8s service = kubelet severity = critical Annotations message = Based on recent sampling, the persistent volume claimed by logging-es-0 in namespace openshift-logging is expected to fill up within four days. Currently 4.838066176e+10 bytes are available. Version-Release number of selected component (if applicable): OpenShift Container Platform 3.11 How reproducible: n/a Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: the problem could be related to bug - https://bugzilla.redhat.com/1809375 where the prometheus pods are constantly restarted.
This is a prediction based alert that should be of a warning type. It doesn't have any direct impact on running cluster and there is `KubePersistentVolumeUsageCritical` alert which if firing needs fast reaction. KubePersistentVolumeFullInFourDays is greatly improved in the latest OpenShift versions and currently, we don't have plans for backport.