Bug 1818806 - Default openshift install requests too many CPU resources to install all components, requests of components on cluster are wrong
Summary: Default openshift install requests too many CPU resources to install all comp...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.3.z
Assignee: Simon Pasquier
QA Contact: Junqi Zhao
URL:
Whiteboard:
: 1816803 1822768 1840885 1853088 (view as bug list)
Depends On: 1813221
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-30 12:38 UTC by Simon Pasquier
Modified: 2023-10-06 19:32 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of: 1813221
Environment:
Last Closed: 2020-08-05 10:54:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github coreos prometheus-operator pull 3272 0 None closed pkg/alertmanager: mergepatch alertmanager containers 2021-02-15 05:36:36 UTC
Github openshift cluster-monitoring-operator pull 848 0 None closed Bug 1818806: remove cpu limits for config reloader 2021-02-15 05:36:35 UTC
Github openshift cluster-monitoring-operator pull 852 0 None closed Bug 1818806: jsonnet/*: adapt resource requests 2021-02-15 05:36:36 UTC
Github openshift openshift-state-metrics pull 49 0 None closed Bug 1818806: jsonnet: set cpu requests 2021-02-15 05:36:36 UTC
Github openshift prometheus-operator pull 75 0 None closed Bug 1818806: pkg/alertmanager: mergepatch alertmanager containers 2021-02-15 05:36:36 UTC
Github openshift telemeter pull 346 0 None closed Bug 1818806: jsonnet/telemeter: set cpu requests, unset limits 2021-02-15 05:36:37 UTC
Red Hat Product Errata RHBA-2020:3180 0 None None None 2020-08-05 10:54:34 UTC

Comment 1 Simon Pasquier 2020-03-30 12:46:00 UTC
*** Bug 1816803 has been marked as a duplicate of this bug. ***

Comment 3 Sergiusz Urbaniak 2020-05-05 07:45:33 UTC
*** Bug 1822768 has been marked as a duplicate of this bug. ***

Comment 4 W. Trevor King 2020-05-28 09:13:14 UTC
*** Bug 1840885 has been marked as a duplicate of this bug. ***

Comment 10 Junqi Zhao 2020-06-03 06:37:09 UTC
The dependent Bug 1813221 is not fixed, move it to Assigned

Comment 17 Simon Pasquier 2020-07-03 09:14:10 UTC
*** Bug 1853088 has been marked as a duplicate of this bug. ***

Comment 20 Junqi Zhao 2020-07-10 10:04:48 UTC
Tested with 4.3.0-0.nightly-2020-07-10-031338, and compared with 44 settings, see from https://bugzilla.redhat.com/show_bug.cgi?id=1813221#c41, there are a few problems
1. alertmanager/config-reloader container don't have resources.requests.cpu setting, and we should reduce the cpu request for alertmanager-proxy
2. should reduce resources.requests.cpu setting for grafana/grafana-proxy containers
3. should reduce resources.requests.cpu setting for kube-rbac-proxy-main/kube-rbac-proxy-self/kube-state-metrics containers
4. should reduce resources.requests.cpu setting for node-exporter/kube-rbac-proxy containers
5. should reduce resources.requests.cpu setting for openshift-state-metrics container
6. should reduce resources.requests.cpu setting for prometheus-adapter container
7. should reduce resources.requests.cpu setting for prometheus/thanos-sidecar/prometheus-proxy/prom-label-proxy containers, don't have resources.requests.cpu setting for prometheus-config-reloader/rules-configmap-reloader containers
8. should reduce resources.requests.cpu setting for prometheus-operator container
9. don't have resources.requests.cpu setting for telemeter-client/reload containers
10. should reduce resources.requests.cpu setting for thanos-querier/oauth-proxy containers
# for i in $(kubectl -n openshift-monitoring get pod --no-headers | awk '{print $1}'); do echo $i; kubectl -n openshift-monitoring get pod $i -o go-template='{{range.spec.containers}}{{"Container Name: "}}{{.name}}{{"\r\nresources: "}}{{.resources}}{{"\n"}}{{end}}'; echo -e "\n"; done
alertmanager-main-0
Container Name: alertmanager
resources: map[requests:map[memory:200Mi]]
Container Name: config-reloader
resources: map[]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


alertmanager-main-1
Container Name: alertmanager
resources: map[requests:map[memory:200Mi]]
Container Name: config-reloader
resources: map[]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


alertmanager-main-2
Container Name: alertmanager
resources: map[requests:map[memory:200Mi]]
Container Name: config-reloader
resources: map[]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


cluster-monitoring-operator-bcff5c786-b5c9d
Container Name: cluster-monitoring-operator
resources: map[requests:map[cpu:10m memory:50Mi]]


grafana-65cc69bb4b-bf6sd
Container Name: grafana
resources: map[requests:map[cpu:100m memory:100Mi]]
Container Name: grafana-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


kube-state-metrics-76f5c5b69-w5b5c
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:10m memory:40Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:10m memory:40Mi]]
Container Name: kube-state-metrics
resources: map[requests:map[cpu:10m memory:40Mi]]


node-exporter-42p2r
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-6srcd
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-78kd6
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-crpgl
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-gdng4
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


node-exporter-psg2q
Container Name: node-exporter
resources: map[requests:map[cpu:102m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


openshift-state-metrics-d7f77b7c4-8lp5h
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: openshift-state-metrics
resources: map[requests:map[cpu:100m memory:150Mi]]


prometheus-adapter-6bc6f5fdbd-rnz72
Container Name: prometheus-adapter
resources: map[requests:map[cpu:10m memory:20Mi]]


prometheus-adapter-6bc6f5fdbd-rtv64
Container Name: prometheus-adapter
resources: map[requests:map[cpu:10m memory:20Mi]]


prometheus-k8s-0
Container Name: prometheus
resources: map[requests:map[cpu:200m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[]
Container Name: rules-configmap-reloader
resources: map[]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:50m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


prometheus-k8s-1
Container Name: prometheus
resources: map[requests:map[cpu:200m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[]
Container Name: rules-configmap-reloader
resources: map[]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:50m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


prometheus-operator-5cbf6b9ff4-6zlbl
Container Name: prometheus-operator
resources: map[requests:map[cpu:10m memory:60Mi]]


telemeter-client-7b5b466948-8m67j
Container Name: telemeter-client
resources: map[]
Container Name: reload
resources: map[]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


thanos-querier-788d89f746-kn44n
Container Name: thanos-querier
resources: map[requests:map[cpu:10m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]


thanos-querier-788d89f746-sjkr8
Container Name: thanos-querier
resources: map[requests:map[cpu:10m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:10m memory:20Mi]]

Comment 25 Junqi Zhao 2020-07-27 03:32:11 UTC
Tested with 4.3.0-0.nightly-2020-07-25-094223, issue is fixed
# for i in $(kubectl -n openshift-monitoring get pod --no-headers | awk '{print $1}'); do echo $i; kubectl -n openshift-monitoring get pod $i -o go-template='{{range.spec.containers}}{{"Container Name: "}}{{.name}}{{"\r\nresources: "}}{{.resources}}{{"\n"}}{{end}}'; echo -e "\n"; done
alertmanager-main-0
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-1
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-2
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


cluster-monitoring-operator-7c46c4dd7c-t9x9c
Container Name: cluster-monitoring-operator
resources: map[requests:map[cpu:10m memory:50Mi]]


grafana-bbdfb8846-xr44n
Container Name: grafana
resources: map[requests:map[cpu:4m memory:100Mi]]
Container Name: grafana-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


kube-state-metrics-6b78dc8f78-gctsz
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-state-metrics
resources: map[requests:map[cpu:2m memory:40Mi]]


node-exporter-9dj5w
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-mnlwf
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-nt4nz
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-t6jsj
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-ww4lf
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-zf7ch
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


openshift-state-metrics-cc7d4d797-mqs6p
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: openshift-state-metrics
resources: map[requests:map[cpu:1m memory:150Mi]]


prometheus-adapter-756f54898b-hbpq9
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-adapter-756f54898b-n5rng
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-0
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: rules-configmap-reloader
resources: map[requests:map[cpu:1m]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-1
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: rules-configmap-reloader
resources: map[requests:map[cpu:1m]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-operator-7f6b94c5c8-5l2sq
Container Name: prometheus-operator
resources: map[requests:map[cpu:5m memory:60Mi]]


telemeter-client-59455b549d-mcvbb
Container Name: telemeter-client
resources: map[requests:map[cpu:1m]]
Container Name: reload
resources: map[requests:map[cpu:1m]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-6d6b7957f7-fc6mz
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-6d6b7957f7-npkgb
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]

Comment 27 errata-xmlrpc 2020-08-05 10:54:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.3.31 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3180


Note You need to log in before you can comment on or make changes to this bug.