Bug 1633812

Description of problem:

I'm trying to find out what memory my pod is consuming and I see different values from the OCP GUI, oc adm top pods and inside the pod.
What is the correct value to use and why are these differently reported?


For the OCP GUI see attached snippet - 3.3 GiB
oc adm top pod  3000M
[qcambre@ocplogs ~]$ oc adm top pods
NAME            CPU(cores)   MEMORY(bytes)
websc-1-qchhj   1431m        3011Mi

And inside the POD.
sh-4.2$ cat /sys/fs/cgroup/memory/memory.stat
cache 838934528
rss 2664419328
rss_huge 2554331136
mapped_file 26480640
swap 0
pgpgin 7097370
pgpgout 6864457
pgfault 618247
pgmajfault 674
inactive_anon 0
active_anon 2664353792
inactive_file 355774464
active_file 483160064
unevictable 0
hierarchical_memory_limit 50460135424
hierarchical_memsw_limit 9223372036854771712
total_cache 838934528
total_rss 2664419328
total_rss_huge 2554331136
total_mapped_file 26480640
total_swap 0
total_pgpgin 7097370
total_pgpgout 6864457
total_pgfault 618247
total_pgmajfault 674
total_inactive_anon 0
total_active_anon 2664353792
total_inactive_file 355774464
total_active_file 483160064
total_unevictable 0

Can you please explain the differences? And what should we use to monitor the memory of pods properly?

Some research:
This issue explains in detail how metrics are calculated on the webconsole :
https://github.com/openshift/origin-web-console/issues/1315

Here is script reproduces what kubelet/cAdvisor does[1], you can consider memory_usage_in_bytes and convert to megabytes.

[1]

#!/bin/bash
#!/usr/bin/env bash

# This script reproduces what kubelet/cAdvisor does
# to calculate memory.available relative to root cgroup.
# The major change is that it excludes total_inactive_file memory.

# current memory usage
memory_capacity_in_kb=$(cat /proc/meminfo | grep MemTotal | awk '{print $2}')
memory_capacity_in_bytes=$((memory_capacity_in_kb * 1024))
memory_usage_in_bytes=$(cat /sys/fs/cgroup/memory/memory.usage_in_bytes)
memory_total_inactive_file=$(cat /sys/fs/cgroup/memory/memory.stat | grep total_inactive_file | awk '{print $2}')

memory_working_set=$memory_usage_in_bytes
if [ "$memory_working_set" -lt "$memory_total_inactive_file" ];
then
    memory_working_set=0
else
    memory_working_set=$((memory_usage_in_bytes - memory_total_inactive_file))
fi

memory_available_in_bytes=$((memory_capacity_in_bytes - memory_working_set))
memory_available_in_kb=$((memory_available_in_bytes / 1024))
memory_available_in_mb=$((memory_available_in_kb / 1024))

echo "memory.capacity_in_bytes $memory_capacity_in_bytes"
echo "memory.usage_in_bytes $memory_usage_in_bytes"
echo "memory.total_inactive_file $memory_total_inactive_file"
echo "memory.working_set $memory_working_set"
echo "memory.available_in_bytes $memory_available_in_bytes"
echo "memory.available_in_kb $memory_available_in_kb"
echo "memory.available_in_mb $memory_available_in_mb"

~
Regards.

This can be related with https://bugzilla.redhat.com/show_bug.cgi?id=1431667.

Version-Release number of selected component (if applicable):
OCP 3.9