Bug 1772440 - [OSP] Infra pods consume too much resources [NEEDINFO]
Summary: [OSP] Infra pods consume too much resources
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.4.0
Assignee: Mike Fedosin
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-14 11:07 UTC by Mike Fedosin
Modified: 2020-05-04 11:16 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-04 11:15:35 UTC
Target Upstream Version:
amurdaca: needinfo? (mfedosin)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift machine-config-operator pull 1268 'None' closed Bug 1772440: OpenStack: decrease infra pods resource consumption 2020-04-30 10:37:46 UTC
Github openshift machine-config-operator pull 1281 'None' closed Bug 1772440: OpenStack: decrease haproxy container resource consumption 2020-04-30 10:37:46 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:16:06 UTC

Description Mike Fedosin 2019-11-14 11:07:19 UTC
Description of problem:

All infra pods have "requests" section with 1Gb of RAM and 150m of CPU, but they do not consume more than 200mb and 100m respectively. This leads to irrational use of resources and prevents the launch of workloads on workers.

Additional info:
Customer's request

Comment 2 weiwei jiang 2019-11-20 06:24:16 UTC
Checked with 4.3.0-0.nightly-2019-11-19-122017, that haproxy pods still consume 1Gi memory.

➜  ~ oc get pods -n openshift-openstack-infra -o custom-columns=Name:.metadata.name,Requests:.spec.containers[*].resources.requests,Containers:.spec.containers[*].name
Name                                          Requests                                              Containers
coredns-qewj43osp-nm7nw-master-0              map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-master-1              map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-master-2              map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-worker-c4qnw          map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-worker-kv7b6          map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-worker-q9k2j          map[cpu:100m memory:200Mi]                            coredns
haproxy-qewj43osp-nm7nw-master-0              map[cpu:150m memory:1Gi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-qewj43osp-nm7nw-master-1              map[cpu:150m memory:1Gi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-qewj43osp-nm7nw-master-2              map[cpu:150m memory:1Gi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
keepalived-qewj43osp-nm7nw-master-0           map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-master-1           map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-master-2           map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-worker-c4qnw       map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-worker-kv7b6       map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-worker-q9k2j       map[cpu:100m memory:200Mi]                            keepalived
mdns-publisher-qewj43osp-nm7nw-master-0       map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-master-1       map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-master-2       map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-worker-c4qnw   map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-worker-kv7b6   map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-worker-q9k2j   map[cpu:100m memory:200Mi]                            mdns-publisher
➜  ~ oc version 
Client Version: v4.3.0
Server Version: 4.3.0-0.nightly-2019-11-19-122017
Kubernetes Version: v1.16.2

Comment 3 Antonio Murdaca 2019-11-21 11:11:00 UTC
Mike can you recheck this one pls?

Comment 5 Antonio Murdaca 2019-12-12 16:39:36 UTC
Since freeze is tomorrow, I'm tentatively moving this to 4.4 but feel free to push it back to 4.3 if it's really needed for 4.3

Comment 8 Martin André 2020-01-09 15:58:36 UTC
Moving to ON_QA as it appears all the patches have merged.

Comment 9 weiwei jiang 2020-01-13 10:01:33 UTC
Checked with 4.3.0-0.nightly-2020-01-11-070223, it's already fixed.

$ oc get pods -n openshift-openstack-infra -o custom-columns=Name:.metadata.name,Requests:.spec.containers[*].resources.requests,Containers:.spec.containers[*].name                                                                  130 ↵
Name                                         Requests                                                Containers
coredns-wj43shd1-ks5tw-master-0              map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-master-1              map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-master-2              map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-worker-9r7cs          map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-worker-kgm6x          map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-worker-nhhcj          map[cpu:100m memory:200Mi]                              coredns
haproxy-wj43shd1-ks5tw-master-0              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-wj43shd1-ks5tw-master-1              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-wj43shd1-ks5tw-master-2              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
keepalived-wj43shd1-ks5tw-master-0           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-master-1           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-master-2           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-worker-9r7cs       map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-worker-kgm6x       map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-worker-nhhcj       map[cpu:100m memory:200Mi]                              keepalived
mdns-publisher-wj43shd1-ks5tw-master-0       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-master-1       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-master-2       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-worker-9r7cs   map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-worker-kgm6x   map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-worker-nhhcj   map[cpu:100m memory:200Mi]                              mdns-publisher


And also for OCP 4.4 
╰─$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-01-13-010304   True        False         12m     Cluster version is 4.4.0-0.nightly-2020-01-13-010304
╭─wjiang@wjiang ~ 
╰─$ oc get pods -n openshift-openstack-infra -o custom-columns=Name:.metadata.name,Requests:.spec.containers[*].resources.requests,Containers:.spec.containers[*].name 
Name                                        Requests                                                Containers
coredns-wj44ios-42qmn-master-0              map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-master-1              map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-master-2              map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-worker-8rqs8          map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-worker-g4lzs          map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-worker-r4x22          map[cpu:100m memory:200Mi]                              coredns
haproxy-wj44ios-42qmn-master-0              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-wj44ios-42qmn-master-1              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-wj44ios-42qmn-master-2              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
keepalived-wj44ios-42qmn-master-0           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-master-1           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-master-2           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-worker-8rqs8       map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-worker-g4lzs       map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-worker-r4x22       map[cpu:100m memory:200Mi]                              keepalived
mdns-publisher-wj44ios-42qmn-master-0       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-master-1       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-master-2       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-worker-8rqs8   map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-worker-g4lzs   map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-worker-r4x22   map[cpu:100m memory:200Mi]                              mdns-publisher

Comment 11 errata-xmlrpc 2020-05-04 11:15:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.