Bug 1712293 - "Memory Usage" under "Workloads -> Pods" of cluster console has large difference with "oc adm top pod"
Summary: "Memory Usage" under "Workloads -> Pods" of cluster console has large differe...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.5.0
Assignee: Sergiusz Urbaniak
QA Contact: Junqi Zhao
URL:
Whiteboard:
: 1709176 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-21 09:27 UTC by Junqi Zhao
Modified: 2020-05-05 07:49 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-05 07:49:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
console screenshot (297.76 KB, image/png)
2019-09-04 06:54 UTC, Sergiusz Urbaniak
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github coreos kube-prometheus pull 215 0 'None' closed jsonnet/prometheus-adapter: use pod cgroup for for cpu/memory metrics. 2021-02-10 07:07:39 UTC
Github openshift console pull 2486 0 'None' closed Bug 1712293: frontend/public: use working set bytes for memory metrics 2021-02-10 07:07:40 UTC

Description Junqi Zhao 2019-05-21 09:27:30 UTC
Description of problem:
Login cluster console with cluster admin, click "Workloads -> Pods", and select one project from drop-down list, eg: openshift-kube-apiserver,
then select one pod, in the "Pod Overview" section, the "Memory Usage" uses 
"pod_name:container_memory_usage_bytes:sum{pod_name='kube-apiserver-ip-10-0-139-94.us-east-2.compute.internal',namespace='openshift-kube-apiserver'}" to calculate "Memory Usage"

search in prometheus UI
****************************************************************
pod_name:container_memory_usage_bytes:sum{pod_name='kube-apiserver-ip-10-0-139-94.us-east-2.compute.internal',namespace='openshift-kube-apiserver'} / 1024 /1024

Element	Value
{namespace="openshift-kube-apiserver",pod_name="kube-apiserver-ip-10-0-139-94.us-east-2.compute.internal"}	1833.5234375
****************************************************************

whereas
****************************************************************
# oc -n openshift-kube-apiserver adm top pod kube-apiserver-ip-10-0-139-94.us-east-2.compute.internal
NAME                                                       CPU(cores)   MEMORY(bytes)   
kube-apiserver-ip-10-0-139-94.us-east-2.compute.internal   206m         867Mi  
****************************************************************

Although we know `oc -n openshift-kube-apiserver adm top pod kube-apiserver-ip-10-0-139-94.us-east-2.compute.internal` uses the following expression
****************************************************************
sum(container_memory_working_set_bytes{container_name!="POD",container_name!="",pod_name!="", pod_name="kube-apiserver-ip-10-0-139-94.us-east-2.compute.internal"}) / 1024 /1024
Element	Value
{}	867.28515625
****************************************************************

I think "Memory Usage" under "Workloads -> Pods" should be the same with "oc adm top pod"

Version-Release number of selected component (if applicable):
4.1.0-0.nightly-2019-05-18-050636

How reproducible:
Always

Steps to Reproduce:
1. See the description part
2.
3.

Actual results:
"Memory Usage" under "Workloads -> Pods" is not the same with "oc adm top pod"

Expected results:
"Memory Usage" under "Workloads -> Pods" should be the same with "oc adm top pod"

Additional info:

Comment 2 Pawel Krupa 2019-05-21 12:20:41 UTC
Similar issue with CPU metrics: https://bugzilla.redhat.com/show_bug.cgi?id=1709176

Comment 9 Sergiusz Urbaniak 2019-09-04 06:54:32 UTC
Created attachment 1611369 [details]
console screenshot

Comment 12 Sergiusz Urbaniak 2019-09-04 11:15:19 UTC
*** Bug 1709176 has been marked as a duplicate of this bug. ***

Comment 13 Junqi Zhao 2019-09-05 11:25:25 UTC
fix is not in 4.2.0-0.nightly-2019-09-04-142146


Note You need to log in before you can comment on or make changes to this bug.