Bug 1946232 - Openshift console can consume very high CPU
Summary: Openshift console can consume very high CPU
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Console Kubevirt Plugin
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.11.0
Assignee: Gilad Lekner
QA Contact: Guohua Ouyang
URL:
Whiteboard: Scrubbed
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-05 13:17 UTC by mchebbi@redhat.com
Modified: 2023-09-15 01:33 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-16 06:11:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
load virtualization tab over 600 VMs (4.55 MB, application/octet-stream)
2021-06-02 09:24 UTC, Guohua Ouyang
no flags Details
navigate with 385 running VMs (3.29 MB, image/gif)
2021-11-02 09:00 UTC, Guohua Ouyang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift console pull 9696 0 None Merged Bug 1946232: VM List can consume very high CPU 2022-01-26 07:46:50 UTC
Github openshift console pull 9761 0 None Merged Bug 1946232: VM List can consume very high CPU 2022-01-26 07:46:52 UTC
Github openshift console pull 9978 0 None Merged Bug 1946232: VM List can consume very high CPU 2022-01-26 07:46:53 UTC

Description mchebbi@redhat.com 2021-04-05 13:17:55 UTC
gathred data url: shorturl.at/bnBI7
Hello,
customer is experiencing performance problems in console access as the installed components are  increased on the platform.  When he access the console , their pc's are consuming very high cpu and a searching process are taking long time.
cutomer also noticed that he has only the performance issue on virtualization tab.  While he browse and search the number of 600 virtual machines, he encounter the problem.

I Have checked the sos-report and found that issue could be caused by the high number of duplicated process :


|-conmon(15467)-+-console(15510)-+-{console}(15672)
           |               |                |-{console}(15673)
           |               |                |-{console}(15674)
           |               |                |-{console}(15675)
           |               |                |-{console}(15676)
           |               |                |-{console}(15686)
           |               |                |-{console}(15688)
           |               |                |-{console}(15689)
           |               |                |-{console}(15690)
           |               |                |-{console}(15697)
           |               |                |-{console}(15698)
           |               |                |-{console}(15699)
           |               |                |-{console}(15700)
           |               |                |-{console}(15705)
           |               |                |-{console}(15710)
           |               |                |-{console}(15721)
           |               |                |-{console}(15722)
           |               |                |-{console}(15723)
           |               |                |-{console}(15724)
           |               |                |-{console}(15744)
           |               |                |-{console}(15745)
           |               |                |-{console}(15746)
           |               |                |-{console}(15770)
           |               |                |-{console}(15783)
           |               |                |-{console}(15784)
           |               |                |-{console}(15785)
           |               |                |-{console}(15786)
           |               |                |-{console}(16006)
           |               |                |-{console}(16022)
           |               |                |-{console}(25314)
           |               |                |-{console}(25315)
           |               |                |-{console}(26018)
           |               |                |-{console}(26019)
           |               |                |-{console}(26020)
           |               |                |-{console}(26021)
           |               |                |-{console}(26022)
           |               |                |-{console}(26043)
           |               |                |-{console}(26044)
           |               |                |-{console}(26045)
           |               |                |-{console}(26129)
           |               |                |-{console}(26992)
           |               |                |-{console}(26993)
           |               |                |-{console}(29493)
           |               |                |-{console}(32223)
           |               |                |-{console}(32224)
           |               |                |-{console}(32225)
           |               |                |-{console}(42257)
           |               |                |-{console}(42258)
           |               |                |-{console}(42259)
           |               |                |-{console}(511816)
           |               |                |-{console}(1502049)
           |               |                |-{console}(2081398)
           |               |                |-{console}(2086878)
           |               |                |-{console}(2088461)
           |               |                `-{console}(2088462)
           |               `-{conmon}(15469)

==========================================================================================


[mchebbi@fedora process]$ grep -i 'console operator' ps_-elfL
F S UID          PID    PPID     LWP  C NLWP PRI  NI ADDR SZ WCHAN  STIME TTY      TIME     CMD
4 S 1000570+   15510   15467   15510  0   56  80   0 - 1210397 -    Mar28 ?        00:00:00 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15672  0   56  80   0 - 1210397 -    Mar28 ?        00:01:37 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15673  0   56  80   0 - 1210397 -    Mar28 ?        00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15674  0   56  80   0 - 1210397 -    Mar28 ?        00:00:15 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
5 S 1000570+   15510   15467   15675  0   56  80   0 - 1210397 -    Mar28 ?        00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15676  0   56  80   0 - 1210397 -    Mar28 ?        00:00:00 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15686  0   56  80   0 - 1210397 -    Mar28 ?        00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15688  0   56  80   0 - 1210397 -    Mar28 ?        00:00:21 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15689  0   56  80   0 - 1210397 -    Mar28 ?        00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15690  0   56  80   0 - 1210397 do_epo Mar28 ?      00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15697  0   56  80   0 - 1210397 -    Mar28 ?        00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15698  0   56  80   0 - 1210397 -    Mar28 ?        00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
5 S 1000570+   15510   15467   15699  0   56  80   0 - 1210397 -    Mar28 ?        00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15700  0   56  80   0 - 1210397 -    Mar28 ?        00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15705  0   56  80   0 - 1210397 -    Mar28 ?        00:00:16 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15710  0   56  80   0 - 1210397 -    Mar28 ?        00:00:23 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15721  0   56  80   0 - 1210397 -    Mar28 ?        00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15722  0   56  80   0 - 1210397 -    Mar28 ?        00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15723  0   56  80   0 - 1210397 -    Mar28 ?        00:00:21 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15724  0   56  80   0 - 1210397 -    Mar28 ?        00:00:00 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15744  0   56  80   0 - 1210397 -    Mar28 ?        00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15745  0   56  80   0 - 1210397 -    Mar28 ?        00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15746  0   56  80   0 - 1210397 -    Mar28 ?        00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15770  0   56  80   0 - 1210397 -    Mar28 ?        00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15783  0   56  80   0 - 1210397 -    Mar28 ?        00:00:23 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15784  0   56  80   0 - 1210397 -    Mar28 ?        00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15785  0   56  80   0 - 1210397 -    Mar28 ?        00:00:21 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   15786  0   56  80   0 - 1210397 -    Mar28 ?        00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   16006  0   56  80   0 - 1210397 -    Mar28 ?        00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   16022  0   56  80   0 - 1210397 -    Mar28 ?        00:00:09 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   25314  0   56  80   0 - 1210397 -    Mar28 ?        00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   25315  0   56  80   0 - 1210397 -    Mar28 ?        00:00:15 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26018  0   56  80   0 - 1210397 -    Mar28 ?        00:00:21 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26019  0   56  80   0 - 1210397 -    Mar28 ?        00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26020  0   56  80   0 - 1210397 -    Mar28 ?        00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26021  0   56  80   0 - 1210397 -    Mar28 ?        00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26022  0   56  80   0 - 1210397 -    Mar28 ?        00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26043  0   56  80   0 - 1210397 -    Mar28 ?        00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26044  0   56  80   0 - 1210397 -    Mar28 ?        00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26045  0   56  80   0 - 1210397 -    Mar28 ?        00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26129  0   56  80   0 - 1210397 -    Mar28 ?        00:00:15 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26992  0   56  80   0 - 1210397 -    Mar28 ?        00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   26993  0   56  80   0 - 1210397 -    Mar28 ?        00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   29493  0   56  80   0 - 1210397 -    Mar28 ?        00:00:07 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   32223  0   56  80   0 - 1210397 -    Mar28 ?        00:00:14 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   32224  0   56  80   0 - 1210397 -    Mar28 ?        00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   32225  0   56  80   0 - 1210397 -    Mar28 ?        00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   42257  0   56  80   0 - 1210397 -    Mar28 ?        00:00:23 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   42258  0   56  80   0 - 1210397 -    Mar28 ?        00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467   42259  0   56  80   0 - 1210397 -    Mar28 ?        00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467  511816  0   56  80   0 - 1210397 -    Mar28 ?        00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467 1502049  0   56  80   0 - 1210397 -    Mar29 ?        00:00:13 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467 2081398  0   56  80   0 - 1210397 -    Mar29 ?        00:00:12 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467 2086878  0   56  80   0 - 1210397 -    Mar29 ?        00:00:14 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467 2088461  0   56  80   0 - 1210397 -    Mar29 ?        00:00:09 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
1 S 1000570+   15510   15467 2088462  0   56  80   0 - 1210397 -    Mar29 ?        00:00:11 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml
[mchebbi@fedora process]$ 
------------------
thanks in advance for your help.

Comment 1 Jakub Hadvig 2021-04-09 17:36:24 UTC
Started the review process but dint have time to finish due to work on stories and bugs.

Comment 3 Gilad Lekner 2021-04-18 14:31:44 UTC
Hey Sam, @mchebbi
while investigating the issue, I created 600 VMs, and 600 PVCs to measure the performance between the views.
I did notice a lower performance on the VMs page while making a search query.
this could happen for couple of reasons:

1. The table has more columns which are more complex to render
2. The VMs page has another tab for VM Templates which has a table of it self, could cause a more complex rendering aswell.
3. All the rows are rendered, IMO we should introduce infinite scrolling or pagination in the future, which will massively improve performance

thoughts?

Comment 4 mchebbi@redhat.com 2021-04-19 09:05:40 UTC
(In reply to Gilad Lekner from comment #3)
> Hey Sam, @mchebbi
> while investigating the issue, I created 600 VMs, and 600 PVCs to measure
> the performance between the views.
> I did notice a lower performance on the VMs page while making a search query.
> this could happen for couple of reasons:
> 
> 1. The table has more columns which are more complex to render
> 2. The VMs page has another tab for VM Templates which has a table of it
> self, could cause a more complex rendering aswell.
> 3. All the rows are rendered, IMO we should introduce infinite scrolling or
> pagination in the future, which will massively improve performance
> 
> thoughts?

@Gilad Lekner: I totally agree with you, infinite scrolling or pagination could improve performance.

Comment 5 Guohua Ouyang 2021-06-02 09:24:49 UTC
Created attachment 1788642 [details]
load virtualization tab over 600 VMs

Create 600 VMs(status is off) on OCP 4.8, the virtualization tab is able to load and the PC which used to launch the web console is not in high CPU.

Comment 6 Gilad Lekner 2021-06-08 10:01:59 UTC
@mchebbi hey, https://github.com/openshift/console/pull/8821 is merged.
can you please check to see if the issue is fixed?

Comment 7 Ying Cui 2021-06-23 07:46:22 UTC
In comment #5, we could not reproduce the issue, we will move to next release to debug in the future.

Comment 15 Gilad Lekner 2021-08-04 08:52:55 UTC
pr is open, commenting for redundant needinfo

Comment 18 Guohua Ouyang 2021-09-01 01:57:18 UTC
Hi,
Is there any chance to ask the customer to verify this bug?
I see good results on 4.9.0-fc.0 by loading 134 running VMIs, less than 1s.
Since I could not reproduce the bug, I'm not sure whether my tests are enough.

Thanks,
Guohua

Comment 19 Gilad Lekner 2021-09-02 15:10:12 UTC
moving back to assigned for an additional optimization

Comment 23 Guohua Ouyang 2021-11-02 09:00:22 UTC
Created attachment 1839110 [details]
navigate with 385 running VMs

Tested in a cluster which has 385 running VMs and 200+ off VMs, navigate in the virtualization and template page has no issues.

Move the bug to verified based on the test results, feel free to reopen it when later bugs is found.

Comment 24 Guohua Ouyang 2021-11-22 01:30:41 UTC
Could still see the issue on the cluster with the latest console(build from master): https://console-openshift-console.apps.ocp-virt.prod.psi.redhat.com/dashboards.
1. CPU consumer is high when viewing dashboards.
2. CUP consumer is high when viewing VM list page.
3. CPU consumer is also high when viewing Workloads -> Pods.

Based on 1 and 3, I guess this is a common OCP or web interface issue because it requests too many resources from the backend at once.

Move the bug back to assigned for further investigation.

Comment 25 Gilad Lekner 2021-11-29 11:41:26 UTC
@gouyang @yzamir 
Created another PR https://github.com/openshift/console/pull/10572
In Attempt to further optimize the VM list status rendering.
The Lazy VM Status won't be computed (before it was lazy computed) if the printable status is one of the following:

      printableStatus === VMStatusSimpleLabel.Paused ||
      printableStatus === VMStatusSimpleLabel.Unknown ||
      printableStatus === VMStatusSimpleLabel.Stopping ||
      printableStatus === VMStatusSimpleLabel.Stopped ||
      printableStatus === VMStatusSimpleLabel.Running

Comment 29 Yaacov Zamir 2022-05-16 04:11:20 UTC
Customer closed the customer support,
https://access.redhat.com/support/cases/#/case/02902867/discussion?commentId=a0a2K00000ck2y9QAA

I prefere not to backport more fixes to 4.10,

Guohua, hi can we verify this bug based on customer closing the support case as resolved ?

Comment 30 Guohua Ouyang 2022-05-16 06:11:01 UTC
I agree we close the bug as customer close the case already.
Set the resolution to NextRelease as it has good results on 4.10

Comment 31 Red Hat Bugzilla 2023-09-15 01:33:33 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days


Note You need to log in before you can comment on or make changes to this bug.