Created attachment 1784879 [details]
memory request during upgrade e2e test

Description of problem:

With the recent changes from: https://github.com/openshift/cluster-monitoring-operator/pull/1158, the memory request of some of the containers of the monitoring stack were adjusted, but according to some recent tests on CI clusters, there seems to still be room for improvements.

I attached to this bugzilla, the results of two queries ran against the metrics from a e2e-agnostic-operator and e2e-agnostic-upgrade run in CI. Both of these were including the recent changes made in PR 1158. A negative result means that the memory was requested but not used, and a positive result means that more memory was used than the one that was requested. Results are in MiB per pod per container in the openshift-monitoring namespace.

From a first look, we should be able to:
- reduce memory request of thanos-sidecar to 25MiB
- align all kube-rbac-proxy memory requests to ~12MiB
- increase memory request of prometheus-operator, kube-state-metrics and prometheus-adapter

How reproducible:
The memory usage can be queried from Prometheus in a live cluster and can be easily seen.

Steps to Reproduce:
1. Create a cluster with cluster bot
2. Execute a query against prometheus to get the memory usage of pods in the openshift-monitoring space vs the actual memory usage of those pods. The exact query can be found in the attachments.