Bug 1651583 - Alert KubePersistentVolumeFullInFourDays should not be triggered on namespaces other than openshift.*, kube.*, default or logging
Summary: Alert KubePersistentVolumeFullInFourDays should not be triggered on namespace...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 3.11.z
Assignee: Frederic Branczyk
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-20 11:52 UTC by hgomes
Modified: 2019-10-18 01:34 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-18 01:34:36 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3139 None None None 2019-10-18 01:34:58 UTC

Description hgomes 2018-11-20 11:52:03 UTC
Description of problem:

Rule :
kubelet_volume_stats_available_bytes{job="kubelet",namespace=~"(openshift.*|kube.*|default|logging)"} and predict_linear(kubelet_volume_stats_available_bytes{job="kubelet",namespace=~"(openshift.*|kube.*|default|logging)"}[6h], 4 * 24 * 3600) < 0

Alert :
 alertname="KubePersistentVolumeFullInFourDays"
15:46:20, 2018-11-09
Labels :
severity="critical"
service="kubelet"
prometheus="openshift-monitoring/k8s"
persistentvolumeclaim="datadir-mysql-0"
namespace="kube-system"
job="kubelet"
instance="194.13.15.207:10250"
exported_namespace="gams-sanofi"
endpoint="https-metrics"
cluster="scale-ocp-prod"
message:	Based on recent sampling, the persistent volume claimed by datadir-mysql-0 in namespace kube-system is expected to fill up within four days. Currently 1.3514752e+08 bytes are available.

Also note that disk space may be better no be exprimed in Mo or Go that bytes.

Version-Release number of selected component (if applicable):

Openshift 3.11

Steps to Reproduce:

- create a new projet 
- create a small PV
- Put a/some big files on it
- Wait for the alert to be triggered

Actual results:



Expected results:

To be triggered only at the described namespaces.

Additional info:

Comment 1 Frederic Branczyk 2018-11-21 08:58:22 UTC
Sorry for the inconvenience, this has already been merged to be fixed in a 3.11.z release: https://github.com/openshift/cluster-monitoring-operator/pull/127

Comment 2 hgomes 2018-11-26 10:51:50 UTC
Hi Frederic,
Thank you for letting me know.
Do we have an ETA for the next ERRATA where the fix is implemented?

Comment 3 Frederic Branczyk 2018-11-26 12:53:39 UTC
I'm not aware of the schedule for z-stream releases, that question would need to go to the respective team taking care of that.

Comment 10 errata-xmlrpc 2019-10-18 01:34:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3139


Note You need to log in before you can comment on or make changes to this bug.