Bug 2019992 - instance:node_memory_utilisation:ratio metric is incorrect
Summary: instance:node_memory_utilisation:ratio metric is incorrect
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.10
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.10.0
Assignee: Arunprasad Rajkumar
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-03 19:24 UTC by Alex Krzos
Modified: 2022-03-10 16:25 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:24:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Screenshot of grafana showing negative number for memory utilization (103.20 KB, image/png)
2021-11-03 19:24 UTC, Alex Krzos
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1455 0 None open Bug 2013920: Bump jsonnet dependencies to latest 2021-11-08 10:30:36 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:25:11 UTC

Description Alex Krzos 2021-11-03 19:24:12 UTC
Created attachment 1839720 [details]
Screenshot of grafana showing negative number for memory utilization

Description of problem:
Viewing memory utilization on the built in grafana is not showing a percentage used but instead a very large negative number.

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2021-10-28-211203

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:


4.10 recording rule for instance:node_memory_utilisation:ratio

      - expr: |
          1 - (
            node_memory_MemAvailable_bytes{job="node-exporter"}
            or
            (
              node_memory_Buffers_bytes{job="node-exporter"}
              +
              node_memory_Cached_bytes{job="node-exporter"}
              +
              node_memory_MemFree_bytes{job="node-exporter"}
              +
              node_memory_Slab_bytes{job="node-exporter"}
            )
          /
            node_memory_MemTotal_bytes{job="node-exporter"}
          )
        record: instance:node_memory_utilisation:ratio


by comparison 4.9 recording rule for instance:node_memory_utilisation:ratio

      - expr: |
          1 - (
            node_memory_MemAvailable_bytes{job="node-exporter"}
          /
            node_memory_MemTotal_bytes{job="node-exporter"}
          )
        record: instance:node_memory_utilisation:ratio


Looks like if we add additional parenthesis we can fix the recording rule:
          1 - (
            (node_memory_MemAvailable_bytes{job="node-exporter"}
            or
            (
              node_memory_Buffers_bytes{job="node-exporter"}
              +
              node_memory_Cached_bytes{job="node-exporter"}
              +
              node_memory_MemFree_bytes{job="node-exporter"}
              +
              node_memory_Slab_bytes{job="node-exporter"}
            ))
          /
            node_memory_MemTotal_bytes{job="node-exporter"}
          )

Comment 1 Haoyu Sun 2021-11-04 10:10:08 UTC
There is already a bugfix in node_exporter: https://github.com/prometheus/node_exporter/pull/2170
We just need to take it into openshift/node_exporter and update the dependency in Cluster Monitoring Operator.

Comment 2 Junqi Zhao 2021-11-10 10:09:09 UTC
tested with the PR
# oc -n openshift-monitoring logs -c node-exporter node-exporter-gr7qw
level=info ts=2021-11-10T07:14:23.182Z caller=node_exporter.go:182 msg="Starting node_exporter" version="(version=1.2.2, branch=master, revision=48647cc40c4f7b09d07231b1effeed6d5106f44f)"

instance:node_memory_utilisation:ratio expression is:
        - expr: |
            1 - (
              (
                node_memory_MemAvailable_bytes{job="node-exporter"}
                or
                (
                  node_memory_Buffers_bytes{job="node-exporter"}
                  +
                  node_memory_Cached_bytes{job="node-exporter"}
                  +
                  node_memory_MemFree_bytes{job="node-exporter"}
                  +
                  node_memory_Slab_bytes{job="node-exporter"}
                )
              )
            /
              node_memory_MemTotal_bytes{job="node-exporter"}
            )
          record: instance:node_memory_utilisation:ratio

the result is correct in grafana page, it shows a percentage used and not a very large negative number

Comment 5 Junqi Zhao 2021-11-12 08:48:26 UTC
fix is in 4.10.0-0.nightly-2021-11-11-170956, based on Comment 2 and 3, set to VERIFIED

Comment 8 errata-xmlrpc 2022-03-10 16:24:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.