Bug 1772440

Summary: [OSP] Infra pods consume too much resources
Product: OpenShift Container Platform Reporter: Mike Fedosin <mfedosin>
Component: Machine Config OperatorAssignee: Mike Fedosin <mfedosin>
Status: CLOSED ERRATA QA Contact: weiwei jiang <wjiang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: amurdaca, m.andre, wjiang, wsun
Target Milestone: ---Flags: amurdaca: needinfo? (mfedosin)
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-04 11:15:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Mike Fedosin 2019-11-14 11:07:19 UTC
Description of problem:

All infra pods have "requests" section with 1Gb of RAM and 150m of CPU, but they do not consume more than 200mb and 100m respectively. This leads to irrational use of resources and prevents the launch of workloads on workers.

Additional info:
Customer's request

Comment 2 weiwei jiang 2019-11-20 06:24:16 UTC
Checked with 4.3.0-0.nightly-2019-11-19-122017, that haproxy pods still consume 1Gi memory.

➜  ~ oc get pods -n openshift-openstack-infra -o custom-columns=Name:.metadata.name,Requests:.spec.containers[*].resources.requests,Containers:.spec.containers[*].name
Name                                          Requests                                              Containers
coredns-qewj43osp-nm7nw-master-0              map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-master-1              map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-master-2              map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-worker-c4qnw          map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-worker-kv7b6          map[cpu:100m memory:200Mi]                            coredns
coredns-qewj43osp-nm7nw-worker-q9k2j          map[cpu:100m memory:200Mi]                            coredns
haproxy-qewj43osp-nm7nw-master-0              map[cpu:150m memory:1Gi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-qewj43osp-nm7nw-master-1              map[cpu:150m memory:1Gi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-qewj43osp-nm7nw-master-2              map[cpu:150m memory:1Gi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
keepalived-qewj43osp-nm7nw-master-0           map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-master-1           map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-master-2           map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-worker-c4qnw       map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-worker-kv7b6       map[cpu:100m memory:200Mi]                            keepalived
keepalived-qewj43osp-nm7nw-worker-q9k2j       map[cpu:100m memory:200Mi]                            keepalived
mdns-publisher-qewj43osp-nm7nw-master-0       map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-master-1       map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-master-2       map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-worker-c4qnw   map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-worker-kv7b6   map[cpu:100m memory:200Mi]                            mdns-publisher
mdns-publisher-qewj43osp-nm7nw-worker-q9k2j   map[cpu:100m memory:200Mi]                            mdns-publisher
➜  ~ oc version 
Client Version: v4.3.0
Server Version: 4.3.0-0.nightly-2019-11-19-122017
Kubernetes Version: v1.16.2

Comment 3 Antonio Murdaca 2019-11-21 11:11:00 UTC
Mike can you recheck this one pls?

Comment 5 Antonio Murdaca 2019-12-12 16:39:36 UTC
Since freeze is tomorrow, I'm tentatively moving this to 4.4 but feel free to push it back to 4.3 if it's really needed for 4.3

Comment 8 Martin André 2020-01-09 15:58:36 UTC
Moving to ON_QA as it appears all the patches have merged.

Comment 9 weiwei jiang 2020-01-13 10:01:33 UTC
Checked with 4.3.0-0.nightly-2020-01-11-070223, it's already fixed.

$ oc get pods -n openshift-openstack-infra -o custom-columns=Name:.metadata.name,Requests:.spec.containers[*].resources.requests,Containers:.spec.containers[*].name                                                                  130 ↵
Name                                         Requests                                                Containers
coredns-wj43shd1-ks5tw-master-0              map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-master-1              map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-master-2              map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-worker-9r7cs          map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-worker-kgm6x          map[cpu:100m memory:200Mi]                              coredns
coredns-wj43shd1-ks5tw-worker-nhhcj          map[cpu:100m memory:200Mi]                              coredns
haproxy-wj43shd1-ks5tw-master-0              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-wj43shd1-ks5tw-master-1              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-wj43shd1-ks5tw-master-2              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
keepalived-wj43shd1-ks5tw-master-0           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-master-1           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-master-2           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-worker-9r7cs       map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-worker-kgm6x       map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj43shd1-ks5tw-worker-nhhcj       map[cpu:100m memory:200Mi]                              keepalived
mdns-publisher-wj43shd1-ks5tw-master-0       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-master-1       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-master-2       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-worker-9r7cs   map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-worker-kgm6x   map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj43shd1-ks5tw-worker-nhhcj   map[cpu:100m memory:200Mi]                              mdns-publisher


And also for OCP 4.4 
╰─$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-01-13-010304   True        False         12m     Cluster version is 4.4.0-0.nightly-2020-01-13-010304
╭─wjiang@wjiang ~ 
╰─$ oc get pods -n openshift-openstack-infra -o custom-columns=Name:.metadata.name,Requests:.spec.containers[*].resources.requests,Containers:.spec.containers[*].name 
Name                                        Requests                                                Containers
coredns-wj44ios-42qmn-master-0              map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-master-1              map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-master-2              map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-worker-8rqs8          map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-worker-g4lzs          map[cpu:100m memory:200Mi]                              coredns
coredns-wj44ios-42qmn-worker-r4x22          map[cpu:100m memory:200Mi]                              coredns
haproxy-wj44ios-42qmn-master-0              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-wj44ios-42qmn-master-1              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
haproxy-wj44ios-42qmn-master-2              map[cpu:100m memory:200Mi],map[cpu:100m memory:200Mi]   haproxy,haproxy-monitor
keepalived-wj44ios-42qmn-master-0           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-master-1           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-master-2           map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-worker-8rqs8       map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-worker-g4lzs       map[cpu:100m memory:200Mi]                              keepalived
keepalived-wj44ios-42qmn-worker-r4x22       map[cpu:100m memory:200Mi]                              keepalived
mdns-publisher-wj44ios-42qmn-master-0       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-master-1       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-master-2       map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-worker-8rqs8   map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-worker-g4lzs   map[cpu:100m memory:200Mi]                              mdns-publisher
mdns-publisher-wj44ios-42qmn-worker-r4x22   map[cpu:100m memory:200Mi]                              mdns-publisher

Comment 11 errata-xmlrpc 2020-05-04 11:15:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581