Bug 2019992
| Summary: | instance:node_memory_utilisation:ratio metric is incorrect | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Alex Krzos <akrzos> | ||||
| Component: | Monitoring | Assignee: | Arunprasad Rajkumar <arajkuma> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Junqi Zhao <juzhao> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 4.10 | CC: | amuller, anpicker, aos-bugs, erooth, hasun, spasquie | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.10.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-03-10 16:24:56 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
There is already a bugfix in node_exporter: https://github.com/prometheus/node_exporter/pull/2170 We just need to take it into openshift/node_exporter and update the dependency in Cluster Monitoring Operator. tested with the PR
# oc -n openshift-monitoring logs -c node-exporter node-exporter-gr7qw
level=info ts=2021-11-10T07:14:23.182Z caller=node_exporter.go:182 msg="Starting node_exporter" version="(version=1.2.2, branch=master, revision=48647cc40c4f7b09d07231b1effeed6d5106f44f)"
instance:node_memory_utilisation:ratio expression is:
- expr: |
1 - (
(
node_memory_MemAvailable_bytes{job="node-exporter"}
or
(
node_memory_Buffers_bytes{job="node-exporter"}
+
node_memory_Cached_bytes{job="node-exporter"}
+
node_memory_MemFree_bytes{job="node-exporter"}
+
node_memory_Slab_bytes{job="node-exporter"}
)
)
/
node_memory_MemTotal_bytes{job="node-exporter"}
)
record: instance:node_memory_utilisation:ratio
the result is correct in grafana page, it shows a percentage used and not a very large negative number
fix is in 4.10.0-0.nightly-2021-11-11-170956, based on Comment 2 and 3, set to VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056 |
Created attachment 1839720 [details] Screenshot of grafana showing negative number for memory utilization Description of problem: Viewing memory utilization on the built in grafana is not showing a percentage used but instead a very large negative number. Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2021-10-28-211203 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: 4.10 recording rule for instance:node_memory_utilisation:ratio - expr: | 1 - ( node_memory_MemAvailable_bytes{job="node-exporter"} or ( node_memory_Buffers_bytes{job="node-exporter"} + node_memory_Cached_bytes{job="node-exporter"} + node_memory_MemFree_bytes{job="node-exporter"} + node_memory_Slab_bytes{job="node-exporter"} ) / node_memory_MemTotal_bytes{job="node-exporter"} ) record: instance:node_memory_utilisation:ratio by comparison 4.9 recording rule for instance:node_memory_utilisation:ratio - expr: | 1 - ( node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"} ) record: instance:node_memory_utilisation:ratio Looks like if we add additional parenthesis we can fix the recording rule: 1 - ( (node_memory_MemAvailable_bytes{job="node-exporter"} or ( node_memory_Buffers_bytes{job="node-exporter"} + node_memory_Cached_bytes{job="node-exporter"} + node_memory_MemFree_bytes{job="node-exporter"} + node_memory_Slab_bytes{job="node-exporter"} )) / node_memory_MemTotal_bytes{job="node-exporter"} )