Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1818806

Summary:	Default openshift install requests too many CPU resources to install all components, requests of components on cluster are wrong
Product:	OpenShift Container Platform	Reporter:	Simon Pasquier <spasquie>
Component:	Monitoring	Assignee:	Simon Pasquier <spasquie>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.3.z	CC:	adahiya, alegrand, anpicker, bparees, ccoleman, erooth, esimard, jialiu, juzhao, kakkoyun, lcosic, mleonard, mloibl, pkrupa, rphillips, surbania, syangsao, wking, wsun
Target Milestone:	---
Target Release:	4.3.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:	1813221	Environment:
Last Closed:	2020-08-05 10:54:06 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1813221
Bug Blocks:

Comment 1 Simon Pasquier 2020-03-30 12:46:00 UTC

*** Bug 1816803 has been marked as a duplicate of this bug. ***

Comment 3 Sergiusz Urbaniak 2020-05-05 07:45:33 UTC

*** Bug 1822768 has been marked as a duplicate of this bug. ***

Comment 4 W. Trevor King 2020-05-28 09:13:14 UTC

*** Bug 1840885 has been marked as a duplicate of this bug. ***

Comment 10 Junqi Zhao 2020-06-03 06:37:09 UTC

The dependent Bug 1813221 is not fixed, move it to Assigned

Comment 17 Simon Pasquier 2020-07-03 09:14:10 UTC

*** Bug 1853088 has been marked as a duplicate of this bug. ***

Comment 20 Junqi Zhao 2020-07-10 10:04:48 UTC

Tested with 4.3.0-0.nightly-2020-07-10-031338, and compared with 44 settings, see from https://bugzilla.redhat.com/show_bug.cgi?id=1813221#c41, there are a few problems
1. alertmanager/config-reloader container don't have resources.requests.cpu setting, and we should reduce the cpu request for alertmanager-proxy
2. should reduce resources.requests.cpu setting for grafana/grafana-proxy containers
3. should reduce resources.requests.cpu setting for kube-rbac-proxy-main/kube-rbac-proxy-self/kube-state-metrics containers
4. should reduce resources.requests.cpu setting for node-exporter/kube-rbac-proxy containers
5. should reduce resources.requests.cpu setting for openshift-state-metrics container
6. should reduce resources.requests.cpu setting for prometheus-adapter container
7. should reduce resources.requests.cpu setting for prometheus/thanos-sidecar/prometheus-proxy/prom-label-proxy containers, don't have resources.requests.cpu setting for prometheus-config-reloader/rules-configmap-reloader containers
8. should reduce resources.requests.cpu setting for prometheus-operator container
9. don't have resources.requests.cpu setting for telemeter-client/reload containers
10. should reduce resources.requests.cpu setting for thanos-querier/oauth-proxy containers
# for i in $(kubectl -n openshift-monitoring get pod --no-headers | awk '{print $1}'); do echo $i; kubectl -n openshift-monitoring get pod $i -o go-template='{{range.spec.containers}}{{"Container Name: "}}{{.name}}{{"\r\nresources: "}}{{.resources}}{{"\n"}}{{end}}'; echo -e "\n"; done
alertmanager-main-0
Container Name: alertmanager
resources: map[requests:map[memory:200Mi]]
Container Name: config-reloader
resources: map[]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


alertmanager-main-1
Container Name: alertmanager
resources: map[requests:map[memory:200Mi]]
Container Name: config-reloader
resources: map[]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


alertmanager-main-2
Container Name: alertmanager
resources: map[requests:map[memory:200Mi]]
Container Name: config-reloader
resources: map[]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


cluster-monitoring-operator-bcff5c786-b5c9d
Container Name: cluster-monitoring-operator
resources: map[requests:map[cpu:10m memory:50Mi]]


grafana-65cc69bb4b-bf6sd
Container Name: grafana
resources: map[requests:map[cpu:100m memory:100Mi]]
Container Name: grafana-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


kube-state-metrics-76f5c5b69-w5b5c
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:10m memory:40Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:10m memory:40Mi]]
Container Name: kube-state-metrics
resources: map[requests:map[cpu:10m memory:40Mi]]


node-exporter-42p2r
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-6srcd
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-78kd6
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-crpgl
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-gdng4
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-psg2q
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


openshift-state-metrics-d7f77b7c4-8lp5h
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: openshift-state-metrics
resources: map[requests:map[cpu:100m memory:150Mi]]


prometheus-adapter-6bc6f5fdbd-rnz72
Container Name: prometheus-adapter
resources: map[requests:map[cpu:10m memory:20Mi]]


prometheus-adapter-6bc6f5fdbd-rtv64
Container Name: prometheus-adapter
resources: map[requests:map[cpu:10m memory:20Mi]]


prometheus-k8s-0
Container Name: prometheus
resources: map[requests:map[cpu:200m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[]
Container Name: rules-configmap-reloader
resources: map[]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:50m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


prometheus-k8s-1
Container Name: prometheus
resources: map[requests:map[cpu:200m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[]
Container Name: rules-configmap-reloader
resources: map[]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:50m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


prometheus-operator-5cbf6b9ff4-6zlbl
Container Name: prometheus-operator
resources: map[requests:map[cpu:10m memory:60Mi]]


telemeter-client-7b5b466948-8m67j
Container Name: telemeter-client
resources: map[]
Container Name: reload
resources: map[]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


thanos-querier-788d89f746-kn44n
Container Name: thanos-querier
resources: map[requests:map[cpu:10m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


thanos-querier-788d89f746-sjkr8
Container Name: thanos-querier
resources: map[requests:map[cpu:10m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]

Comment 25 Junqi Zhao 2020-07-27 03:32:11 UTC

Tested with 4.3.0-0.nightly-2020-07-25-094223, issue is fixed
# for i in $(kubectl -n openshift-monitoring get pod --no-headers | awk '{print $1}'); do echo $i; kubectl -n openshift-monitoring get pod $i -o go-template='{{range.spec.containers}}{{"Container Name: "}}{{.name}}{{"\r\nresources: "}}{{.resources}}{{"\n"}}{{end}}'; echo -e "\n"; done
alertmanager-main-0
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-1
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-2
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


cluster-monitoring-operator-7c46c4dd7c-t9x9c
Container Name: cluster-monitoring-operator
resources: map[requests:map[cpu:10m memory:50Mi]]


grafana-bbdfb8846-xr44n
Container Name: grafana
resources: map[requests:map[cpu:4m memory:100Mi]]
Container Name: grafana-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


kube-state-metrics-6b78dc8f78-gctsz
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-state-metrics
resources: map[requests:map[cpu:2m memory:40Mi]]


node-exporter-9dj5w
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-mnlwf
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-nt4nz
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-t6jsj
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-ww4lf
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-zf7ch
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


openshift-state-metrics-cc7d4d797-mqs6p
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: openshift-state-metrics
resources: map[requests:map[cpu:1m memory:150Mi]]


prometheus-adapter-756f54898b-hbpq9
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-adapter-756f54898b-n5rng
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-0
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: rules-configmap-reloader
resources: map[requests:map[cpu:1m]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-1
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: rules-configmap-reloader
resources: map[requests:map[cpu:1m]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-operator-7f6b94c5c8-5l2sq
Container Name: prometheus-operator
resources: map[requests:map[cpu:5m memory:60Mi]]


telemeter-client-59455b549d-mcvbb
Container Name: telemeter-client
resources: map[requests:map[cpu:1m]]
Container Name: reload
resources: map[requests:map[cpu:1m]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-6d6b7957f7-fc6mz
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-6d6b7957f7-npkgb
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]

Comment 27 errata-xmlrpc 2020-08-05 10:54:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.3.31 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3180