Bug 1813221 - Default openshift install requests too many CPU resources to install all components, requests of components on cluster are wrong
Summary: Default openshift install requests too many CPU resources to install all comp...
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.4.z
Assignee: Sergiusz Urbaniak
QA Contact: Junqi Zhao
URL:
Whiteboard:
: 1812999 1854002 (view as bug list)
Depends On: 1812719
Blocks: 1818806
TreeView+ depends on / blocked
 
Reported: 2020-03-13 10:24 UTC by Sergiusz Urbaniak
Modified: 2020-07-09 04:39 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1812719
: 1818806 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
alertmanager-main statefulset yaml file (6.85 KB, text/plain)
2020-06-22 02:29 UTC, Junqi Zhao
no flags Details


Links
System ID Priority Status Summary Last Updated
Github coreos prometheus-operator pull 3271 None closed pkg/alertmanager: mergepatch alertmanager containers 2020-07-06 07:31:21 UTC
Github openshift cluster-monitoring-operator pull 706 None closed Bug 1813221: jsonnet/*: adapt resource requests 2020-07-06 07:31:20 UTC
Github openshift cluster-monitoring-operator pull 769 None closed Bug 1813221: set remaining cpu requests 2020-07-06 07:31:20 UTC
Github openshift cluster-monitoring-operator pull 818 None closed Bug 1813221: jsonnet: add missing resource requests for alertmanager sidecar 2020-07-07 12:29:50 UTC
Github openshift cluster-monitoring-operator pull 841 None open Bug 1813221: remove cpu limits for config reloader 2020-07-08 15:25:59 UTC
Github openshift openshift-state-metrics pull 46 None closed Bug 1813221: jsonnet: set cpu requests 2020-07-06 07:31:21 UTC
Github openshift prometheus-operator pull 74 None closed Bug 1813221: pkg/alertmanager: mergepatch alertmanager containers 2020-07-06 07:31:20 UTC
Github openshift telemeter pull 320 None closed Bug 1813221: jsonnet/telemeter: set cpu requests, unset limits 2020-07-06 07:31:20 UTC

Comment 4 Junqi Zhao 2020-04-10 06:51:35 UTC
Tested with 4.4.0-0.nightly-2020-04-09-194724, cpu request for config-reloader container of alertmanager pod is 100m, it is too big, same for openshift-state-metrics container of openshift-state-metrics pod

sum by(container) (rate(container_cpu_usage_seconds_total{pod="alertmanager-main-0", container!="",container!="POD"}[15m])*1000)
Element 	Value
{container="alertmanager"}	2.354869532214438
{container="alertmanager-proxy"}	0.657906244888057
{container="config-reloader"}	0.0022678136128019668

sum by(container) (rate(container_cpu_usage_seconds_total{pod="openshift-state-metrics-84564ddb7b-kdnlv", container!="",container!="POD"}[15m])*1000)
Element 	Value
{container="kube-rbac-proxy-main"}	0.05401827151806968
{container="kube-rbac-proxy-self"}	0.049755666116416825
{container="openshift-state-metrics"}	0.12596162939027375


# for i in $(kubectl -n openshift-monitoring get pod --no-headers | awk '{print $1}'); do echo $i; kubectl -n openshift-monitoring get pod $i -o go-template='{{range.spec.containers}}{{"Container Name: "}}{{.name}}{{"\r\nresources: "}}{{.resources}}{{"\n"}}{{end}}'; echo -e "\n"; done
alertmanager-main-0

Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-1
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-2
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


cluster-monitoring-operator-5df74c575d-xjz5b
Container Name: cluster-monitoring-operator
resources: map[requests:map[cpu:10m memory:50Mi]]


grafana-67dc94bf75-9zggt
Container Name: grafana
resources: map[requests:map[cpu:4m memory:100Mi]]
Container Name: grafana-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


kube-state-metrics-679b84dcff-rvffm
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-state-metrics
resources: map[requests:map[cpu:2m memory:40Mi]]


node-exporter-5l7fh
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-7cw6l
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-7tbwb
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-bnd6b
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-ntj54
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-t9zbk
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


openshift-state-metrics-84564ddb7b-kdnlv
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: openshift-state-metrics
resources: map[requests:map[cpu:100m memory:150Mi]]


prometheus-adapter-5f67bdc747-2xv87
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-adapter-5f67bdc747-q8jr7
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-0
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: rules-configmap-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-1
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: rules-configmap-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-operator-5cc58c7856-9gpf9
Container Name: prometheus-operator
resources: map[requests:map[cpu:5m memory:60Mi]]


telemeter-client-85d96c6496-srfq4
Container Name: telemeter-client
resources: map[requests:map[cpu:1m]]
Container Name: reload
resources: map[requests:map[cpu:1m]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-6fddc75f4d-2d7gt
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-6fddc75f4d-s7wdm
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]

Comment 7 Junqi Zhao 2020-04-17 02:45:51 UTC
Tested with 4.4.0-0.nightly-2020-04-16-205909, the result is the same as Comment 4, https://github.com/openshift/openshift-state-metrics/pull/46/files is already in this build now, since it is merged in https://openshift-release.svc.ci.openshift.org//releasestream/4.4.0-0.nightly/release/4.4.0-0.nightly-2020-04-16-162058 and later build.

"cpu request for config-reloader container of alertmanager pod is 100m, it is too big, same for openshift-state-metrics container of openshift-state-metrics pod" 

-> we also need to fix this issue, I think we should do the same for 4.4 as in 4.5, see 4.5 fix: https://github.com/openshift/cluster-monitoring-operator/pull/755/files

# for i in $(kubectl -n openshift-monitoring get pod --no-headers | awk '{print $1}'); do echo $i; kubectl -n openshift-monitoring get pod $i -o go-template='{{range.spec.containers}}{{"Container Name: "}}{{.name}}{{"\r\nresources: "}}{{.resources}}{{"\n"}}{{end}}'; echo -e "\n"; done
alertmanager-main-0
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-1
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-2
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


cluster-monitoring-operator-5976bcb888-phhz4
Container Name: cluster-monitoring-operator
resources: map[requests:map[cpu:10m memory:50Mi]]


grafana-746fcd5f5d-gwqsq
Container Name: grafana
resources: map[requests:map[cpu:4m memory:100Mi]]
Container Name: grafana-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


kube-state-metrics-749ff87465-rv85m
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-state-metrics
resources: map[requests:map[cpu:2m memory:40Mi]]


node-exporter-gcmz2
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-hfvlm
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-k48pg
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-nn7r2
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-p88s8
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-wcjgl
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


openshift-state-metrics-5b7d864ff9-9qlt8
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:10m memory:20Mi]]
Container Name: openshift-state-metrics
resources: map[requests:map[cpu:100m memory:150Mi]]


prometheus-adapter-864547d858-gnh6l
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-adapter-864547d858-nkz8h
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-0
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: rules-configmap-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-1
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: rules-configmap-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-operator-5b86546448-d7jwg
Container Name: prometheus-operator
resources: map[requests:map[cpu:5m memory:60Mi]]


telemeter-client-88f5d5c5f-pp9th
Container Name: telemeter-client
resources: map[requests:map[cpu:1m]]
Container Name: reload
resources: map[requests:map[cpu:1m]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-6b44f48967-97wq6
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-6b44f48967-z2rgh
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]

Comment 14 Junqi Zhao 2020-05-08 03:14:24 UTC
Tested with 4.4.0-0.nightly-2020-05-07-223059, cpu request for config-reloader container of alertmanager pod is 100m, should be reduced, other settings are fine
# for i in $(kubectl -n openshift-monitoring get pod --no-headers | awk '{print $1}'); do echo $i; kubectl -n openshift-monitoring get pod $i -o go-template='{{range.spec.containers}}{{"Container Name: "}}{{.name}}{{"\r\nresources: "}}{{.resources}}{{"\n"}}{{end}}'; echo -e "\n"; done
alertmanager-main-0
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-1
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-2
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


cluster-monitoring-operator-587677f6dd-gxcf4
Container Name: cluster-monitoring-operator
resources: map[requests:map[cpu:10m memory:50Mi]]


grafana-7f56fbfb44-vjmkd
Container Name: grafana
resources: map[requests:map[cpu:4m memory:100Mi]]
Container Name: grafana-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


kube-state-metrics-56c6d4b486-2dg4x
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-state-metrics
resources: map[requests:map[cpu:2m memory:40Mi]]


node-exporter-6zvft
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-87xzt
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-9nxhf
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-9q66q
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-xcrg2
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-xs8hd
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


openshift-state-metrics-65479646c9-r7cdv
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: openshift-state-metrics
resources: map[requests:map[cpu:1m memory:150Mi]]


prometheus-adapter-754766cb98-2kbpp
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-adapter-754766cb98-vr2kk
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-0
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: rules-configmap-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-1
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: rules-configmap-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:1m memory:25Mi]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-operator-6cf47f8648-njqh6
Container Name: prometheus-operator
resources: map[requests:map[cpu:5m memory:60Mi]]


telemeter-client-5fdbf4f9fd-76f49
Container Name: telemeter-client
resources: map[requests:map[cpu:1m]]
Container Name: reload
resources: map[requests:map[cpu:1m]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-5fdf958d8c-9gt9q
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-5fdf958d8c-f9b4t
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]

Comment 21 W. Trevor King 2020-05-27 03:45:57 UTC
*** Bug 1812999 has been marked as a duplicate of this bug. ***

Comment 25 Junqi Zhao 2020-06-01 03:04:57 UTC
Did not find the fix, move back to MODIFIED

Comment 27 Junqi Zhao 2020-06-03 02:07:47 UTC
Tested with 4.4.0-0.nightly-2020-06-02-202425, cpu request for config-reloader container of alertmanager pod is still 100m

alertmanager-main-0
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]

Comment 29 Scott Dodson 2020-06-03 14:31:19 UTC
Please don't move this back to MODIFIED until additional fixes have been merged.

Comment 33 Junqi Zhao 2020-06-22 02:16:28 UTC
https://github.com/openshift/prometheus-operator/pull/74
is in https://openshift-release.svc.ci.openshift.org/releasestream/4.4.0-0.nightly/release/4.4.0-0.nightly-2020-06-19-230820 and later build
tested with 4.4.0-0.nightly-2020-06-21-210301, 

cpu request for config-reloader container of alertmanager pod is still 100m
alertmanager-main-0
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[limits:map[cpu:100m memory:25Mi] requests:map[cpu:100m memory:25Mi]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]

Comment 34 Junqi Zhao 2020-06-22 02:27:55 UTC
The reason is that if we only specify the resources.limits, the resources.requests would use the same value as resources.limits
# oc -n openshift-monitoring get sts/alertmanager-main -oyaml
...
        name: config-reloader
        resources:
          limits:
            cpu: 100m
            memory: 25Mi
...

Comment 35 Junqi Zhao 2020-06-22 02:29:26 UTC
Created attachment 1698244 [details]
alertmanager-main statefulset yaml file

Comment 38 Pawel Krupa 2020-07-06 07:32:53 UTC
*** Bug 1854002 has been marked as a duplicate of this bug. ***

Comment 41 Junqi Zhao 2020-07-09 04:39:14 UTC
issue is fixed with 4.4.0-0.nightly-2020-07-08-233114
# for i in $(kubectl -n openshift-monitoring get pod --no-headers | awk '{print $1}'); do echo $i; kubectl -n openshift-monitoring get pod $i -o go-template='{{range.spec.containers}}{{"Container Name: "}}{{.name}}{{"\r\nresources: "}}{{.resources}}{{"\n"}}{{end}}'; echo -e "\n"; done
alertmanager-main-0
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-1
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


alertmanager-main-2
Container Name: alertmanager
resources: map[requests:map[cpu:4m memory:200Mi]]
Container Name: config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: alertmanager-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


cluster-monitoring-operator-6f5d96f5bb-zkj8l
Container Name: cluster-monitoring-operator
resources: map[requests:map[cpu:10m memory:50Mi]]


grafana-85d57fb957-cxfvj
Container Name: grafana
resources: map[requests:map[cpu:4m memory:100Mi]]
Container Name: grafana-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


kube-state-metrics-654fb85db9-cxrg2
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:40Mi]]
Container Name: kube-state-metrics
resources: map[requests:map[cpu:2m memory:40Mi]]


node-exporter-74gxn
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-7t6bx
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-9sqfg
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-qwpk6
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-r8c7p
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


node-exporter-x4z98
Container Name: node-exporter
resources: map[requests:map[cpu:8m memory:180Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:30Mi]]


openshift-state-metrics-6776b7d69c-tzxlw
Container Name: kube-rbac-proxy-main
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy-self
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: openshift-state-metrics
resources: map[requests:map[cpu:1m memory:150Mi]]


prometheus-adapter-578dfd9cd4-fvccl
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-adapter-578dfd9cd4-wn649
Container Name: prometheus-adapter
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-0
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: rules-configmap-reloader
resources: map[requests:map[cpu:1m]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-k8s-1
Container Name: prometheus
resources: map[requests:map[cpu:70m memory:1Gi]]
Container Name: prometheus-config-reloader
resources: map[requests:map[cpu:1m]]
Container Name: rules-configmap-reloader
resources: map[requests:map[cpu:1m]]
Container Name: thanos-sidecar
resources: map[requests:map[cpu:1m memory:100Mi]]
Container Name: prometheus-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


prometheus-operator-b664c969-zkwt7
Container Name: prometheus-operator
resources: map[requests:map[cpu:5m memory:60Mi]]


telemeter-client-8f694888-qvh7r
Container Name: telemeter-client
resources: map[requests:map[cpu:1m]]
Container Name: reload
resources: map[requests:map[cpu:1m]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-f87789589-mjkp6
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


thanos-querier-f87789589-r7xtv
Container Name: thanos-querier
resources: map[requests:map[cpu:5m memory:12Mi]]
Container Name: oauth-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: kube-rbac-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]
Container Name: prom-label-proxy
resources: map[requests:map[cpu:1m memory:20Mi]]


Note You need to log in before you can comment on or make changes to this bug.