Bug 2074807

Summary:	NodeFilesystemSpaceFillingUp alert fires even before kubelet GC kicks in
Product:	OpenShift Container Platform	Reporter:	Arunprasad Rajkumar <arajkuma>
Component:	Monitoring	Assignee:	Simon Pasquier <spasquie>
Status:	CLOSED ERRATA	QA Contact:	hongyan li <hongyli>
Severity:	low	Docs Contact:
Priority:	medium
Version:	4.10	CC:	amuller, anpicker, aos-bugs, hongyli, spasquie, wking
Target Milestone:	---
Target Release:	4.11.0
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	2077722 (view as bug list)		Environment:
Last Closed:	2022-08-10 11:07:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2077722

Description Arunprasad Rajkumar 2022-04-13 06:59:49 UTC

Description of problem:

Previously[1] we attempted to do the same, but there was a
misunderstanding about the GC behavior and it caused the alert to be
fired even before GC comes into play.

According to[2][3] kubelet GC kicks in only when `imageGCHighThresholdPercent` is hit which is set to 85% by default. However `NodeFilesystemSpaceFillingUp` is set to fire as soon as 80% usage is hit.

[1] https://github.com/prometheus-operator/kube-prometheus/pull/1357
[2] https://docs.openshift.com/container-platform/4.10/nodes/nodes/nodes-nodes-garbage-collection.html#nodes-nodes-garbage-collection-images_nodes-nodes-configuring
[3] https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ 

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
NodeFilesystemSpaceFillingUp fires before kubelet GC kicks in

Expected results:

NodeFilesystemSpaceFillingUp shouldn't fire before kubelet GC kicks in


Additional info:

Comment 2 hongyan li 2022-04-22 08:19:57 UTC

Wait for pr is in payload

Comment 4 hongyan li 2022-04-24 02:17:34 UTC

Test with payload  4.11.0-0.nightly-2022-04-23-153426
% host=$(oc -n openshift-monitoring get route thanos-querier -ojsonpath={.spec.host})
% token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
% curl -H "Authorization: Bearer $token" -k "https://$host/api/v1/rules" | jq |grep -A10 NodeFilesystemSpaceFillingUp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  252k    0  252k    0     0   140k      0 --:--:--  0:00:01 --:--:--  140k
            "name": "NodeFilesystemSpaceFillingUp",
            "query": "(node_filesystem_avail_bytes{fstype!=\"\",job=\"node-exporter\"} / node_filesystem_size_bytes{fstype!=\"\",job=\"node-exporter\"} * 100 < 10 and predict_linear(node_filesystem_avail_bytes{fstype!=\"\",job=\"node-exporter\"}[6h], 4 * 60 * 60) < 0 and node_filesystem_readonly{fstype!=\"\",job=\"node-exporter\"} == 0)",
            "duration": 3600,
            "labels": {
              "prometheus": "openshift-monitoring/k8s",
              "severity": "critical"
            },
            "annotations": {
              "description": "Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available space left and is filling up fast.",
              "runbook_url": "https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemSpaceFillingUp.md",
              "summary": "Filesystem is predicted to run out of space within the next 4 hours."
            },
            "alerts": [],
            "health": "ok",
            "evaluationTime": 0.00215177,
            "lastEvaluation": "2022-04-24T02:15:36.216317682Z",
            "type": "alerting"
          },
          {
            "state": "inactive",
            "name": "NodeFilesystemSpaceFillingUp",
            "query": "(node_filesystem_avail_bytes{fstype!=\"\",job=\"node-exporter\"} / node_filesystem_size_bytes{fstype!=\"\",job=\"node-exporter\"} * 100 < 15 and predict_linear(node_filesystem_avail_bytes{fstype!=\"\",job=\"node-exporter\"}[6h], 24 * 60 * 60) < 0 and node_filesystem_readonly{fstype!=\"\",job=\"node-exporter\"} == 0)",
            "duration": 3600,
            "labels": {
              "prometheus": "openshift-monitoring/k8s",
              "severity": "warning"
            },
            "annotations": {
              "description": "Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available space left and is filling up.",
              "runbook_url": "https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/NodeFilesystemSpaceFillingUp.md",
              "summary": "Filesystem is predicted to run out of space within the next 24 hours."
            },
            "alerts": [],
            "health": "ok",
            "evaluationTime": 0.002492956,
            "lastEvaluation": "2022-04-24T02:15:36.21382262Z",
            "type": "alerting"
          },
          {
            "state": "inactive",

Comment 9 errata-xmlrpc 2022-08-10 11:07:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069