Description of problem:

The way memory usage is shown in different dashboards is causing some confusion for customers.

The first instance is from the 'grafana-dashboard-k8s-resources-cluster' config and uses a sum of container_memory_rss by namespace to show memory in the 'K8s / Compute Resource / Cluster' view:

                          "targets": [
                              {
                                  "expr": "sum(container_memory_rss{cluster=\"$cluster\", container_name!=\"\"}) by (namespace)",
                                  "format": "time_series",
                                  "intervalFactor": 2,
                                  "legendFormat": "{{namespace}}",
                                  "legendLink": null,
                                  "step": 10
                              }
                          ],

Code: https://github.com/openshift/cluster-monitoring-operator/blob/release-3.11/assets/grafana/dashboard-definitions.yaml#L4033

The first instance is from the grafana-dashboard-k8s-resources-namespace config and used a sum of container_memory_usage_bytes by pod in the 'K8s / Compute Resource / NAmespaces' view:

                          "targets": [
                              {
                                  "expr": "sum(container_memory_usage_bytes{namespace=\"$namespace\", container_name!=\"\"}) by (pod_name)",
                                  "format": "time_series",
                                  "intervalFactor": 2,
                                  "legendFormat": "{{pod_name}}",
                                  "legendLink": null,
                                  "step": 10
                              }
                          ],

Code: https://github.com/openshift/cluster-monitoring-operator/blob/release-3.11/assets/grafana/dashboard-definitions.yaml#L4845

Between these 2 metrics, container_memory_usage_bytes and container_memory_rss are 2 independent metrics generated from cAdvisor:

container_memory_usage_bytes is generated from the cgroup memory.usage_in_bytes which includes rss, cache and swap where as container_memory_rss is based on the rss value from memory.stat.


Within Grafana, the cluster view memory pane shows "Memory Usage w/o Cache" where as the Namespace pane is showing "Memory Usage", but I don't think that is a clear enough distinction to end users that they're reporting 2 different values from 2 different metric sources.


Version-Release number of selected component (if applicable):

3.11


How reproducible:

View the resources in grafana

Actual results:

Memory reporter is different and can confuse end users

Expected results:

Memory is aligned or we better explain what it is we are reporting on


Additional info: