Hide Forgot
Created attachment 1839720 [details] Screenshot of grafana showing negative number for memory utilization Description of problem: Viewing memory utilization on the built in grafana is not showing a percentage used but instead a very large negative number. Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2021-10-28-211203 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: 4.10 recording rule for instance:node_memory_utilisation:ratio - expr: | 1 - ( node_memory_MemAvailable_bytes{job="node-exporter"} or ( node_memory_Buffers_bytes{job="node-exporter"} + node_memory_Cached_bytes{job="node-exporter"} + node_memory_MemFree_bytes{job="node-exporter"} + node_memory_Slab_bytes{job="node-exporter"} ) / node_memory_MemTotal_bytes{job="node-exporter"} ) record: instance:node_memory_utilisation:ratio by comparison 4.9 recording rule for instance:node_memory_utilisation:ratio - expr: | 1 - ( node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"} ) record: instance:node_memory_utilisation:ratio Looks like if we add additional parenthesis we can fix the recording rule: 1 - ( (node_memory_MemAvailable_bytes{job="node-exporter"} or ( node_memory_Buffers_bytes{job="node-exporter"} + node_memory_Cached_bytes{job="node-exporter"} + node_memory_MemFree_bytes{job="node-exporter"} + node_memory_Slab_bytes{job="node-exporter"} )) / node_memory_MemTotal_bytes{job="node-exporter"} )
There is already a bugfix in node_exporter: https://github.com/prometheus/node_exporter/pull/2170 We just need to take it into openshift/node_exporter and update the dependency in Cluster Monitoring Operator.
tested with the PR # oc -n openshift-monitoring logs -c node-exporter node-exporter-gr7qw level=info ts=2021-11-10T07:14:23.182Z caller=node_exporter.go:182 msg="Starting node_exporter" version="(version=1.2.2, branch=master, revision=48647cc40c4f7b09d07231b1effeed6d5106f44f)" instance:node_memory_utilisation:ratio expression is: - expr: | 1 - ( ( node_memory_MemAvailable_bytes{job="node-exporter"} or ( node_memory_Buffers_bytes{job="node-exporter"} + node_memory_Cached_bytes{job="node-exporter"} + node_memory_MemFree_bytes{job="node-exporter"} + node_memory_Slab_bytes{job="node-exporter"} ) ) / node_memory_MemTotal_bytes{job="node-exporter"} ) record: instance:node_memory_utilisation:ratio the result is correct in grafana page, it shows a percentage used and not a very large negative number
fix is in 4.10.0-0.nightly-2021-11-11-170956, based on Comment 2 and 3, set to VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056