gathred data url: shorturl.at/bnBI7 Hello, customer is experiencing performance problems in console access as the installed components are increased on the platform. When he access the console , their pc's are consuming very high cpu and a searching process are taking long time. cutomer also noticed that he has only the performance issue on virtualization tab. While he browse and search the number of 600 virtual machines, he encounter the problem. I Have checked the sos-report and found that issue could be caused by the high number of duplicated process : |-conmon(15467)-+-console(15510)-+-{console}(15672) | | |-{console}(15673) | | |-{console}(15674) | | |-{console}(15675) | | |-{console}(15676) | | |-{console}(15686) | | |-{console}(15688) | | |-{console}(15689) | | |-{console}(15690) | | |-{console}(15697) | | |-{console}(15698) | | |-{console}(15699) | | |-{console}(15700) | | |-{console}(15705) | | |-{console}(15710) | | |-{console}(15721) | | |-{console}(15722) | | |-{console}(15723) | | |-{console}(15724) | | |-{console}(15744) | | |-{console}(15745) | | |-{console}(15746) | | |-{console}(15770) | | |-{console}(15783) | | |-{console}(15784) | | |-{console}(15785) | | |-{console}(15786) | | |-{console}(16006) | | |-{console}(16022) | | |-{console}(25314) | | |-{console}(25315) | | |-{console}(26018) | | |-{console}(26019) | | |-{console}(26020) | | |-{console}(26021) | | |-{console}(26022) | | |-{console}(26043) | | |-{console}(26044) | | |-{console}(26045) | | |-{console}(26129) | | |-{console}(26992) | | |-{console}(26993) | | |-{console}(29493) | | |-{console}(32223) | | |-{console}(32224) | | |-{console}(32225) | | |-{console}(42257) | | |-{console}(42258) | | |-{console}(42259) | | |-{console}(511816) | | |-{console}(1502049) | | |-{console}(2081398) | | |-{console}(2086878) | | |-{console}(2088461) | | `-{console}(2088462) | `-{conmon}(15469) ========================================================================================== [mchebbi@fedora process]$ grep -i 'console operator' ps_-elfL F S UID PID PPID LWP C NLWP PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 4 S 1000570+ 15510 15467 15510 0 56 80 0 - 1210397 - Mar28 ? 00:00:00 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15672 0 56 80 0 - 1210397 - Mar28 ? 00:01:37 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15673 0 56 80 0 - 1210397 - Mar28 ? 00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15674 0 56 80 0 - 1210397 - Mar28 ? 00:00:15 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 5 S 1000570+ 15510 15467 15675 0 56 80 0 - 1210397 - Mar28 ? 00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15676 0 56 80 0 - 1210397 - Mar28 ? 00:00:00 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15686 0 56 80 0 - 1210397 - Mar28 ? 00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15688 0 56 80 0 - 1210397 - Mar28 ? 00:00:21 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15689 0 56 80 0 - 1210397 - Mar28 ? 00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15690 0 56 80 0 - 1210397 do_epo Mar28 ? 00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15697 0 56 80 0 - 1210397 - Mar28 ? 00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15698 0 56 80 0 - 1210397 - Mar28 ? 00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 5 S 1000570+ 15510 15467 15699 0 56 80 0 - 1210397 - Mar28 ? 00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15700 0 56 80 0 - 1210397 - Mar28 ? 00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15705 0 56 80 0 - 1210397 - Mar28 ? 00:00:16 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15710 0 56 80 0 - 1210397 - Mar28 ? 00:00:23 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15721 0 56 80 0 - 1210397 - Mar28 ? 00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15722 0 56 80 0 - 1210397 - Mar28 ? 00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15723 0 56 80 0 - 1210397 - Mar28 ? 00:00:21 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15724 0 56 80 0 - 1210397 - Mar28 ? 00:00:00 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15744 0 56 80 0 - 1210397 - Mar28 ? 00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15745 0 56 80 0 - 1210397 - Mar28 ? 00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15746 0 56 80 0 - 1210397 - Mar28 ? 00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15770 0 56 80 0 - 1210397 - Mar28 ? 00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15783 0 56 80 0 - 1210397 - Mar28 ? 00:00:23 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15784 0 56 80 0 - 1210397 - Mar28 ? 00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15785 0 56 80 0 - 1210397 - Mar28 ? 00:00:21 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 15786 0 56 80 0 - 1210397 - Mar28 ? 00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 16006 0 56 80 0 - 1210397 - Mar28 ? 00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 16022 0 56 80 0 - 1210397 - Mar28 ? 00:00:09 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 25314 0 56 80 0 - 1210397 - Mar28 ? 00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 25315 0 56 80 0 - 1210397 - Mar28 ? 00:00:15 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26018 0 56 80 0 - 1210397 - Mar28 ? 00:00:21 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26019 0 56 80 0 - 1210397 - Mar28 ? 00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26020 0 56 80 0 - 1210397 - Mar28 ? 00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26021 0 56 80 0 - 1210397 - Mar28 ? 00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26022 0 56 80 0 - 1210397 - Mar28 ? 00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26043 0 56 80 0 - 1210397 - Mar28 ? 00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26044 0 56 80 0 - 1210397 - Mar28 ? 00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26045 0 56 80 0 - 1210397 - Mar28 ? 00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26129 0 56 80 0 - 1210397 - Mar28 ? 00:00:15 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26992 0 56 80 0 - 1210397 - Mar28 ? 00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 26993 0 56 80 0 - 1210397 - Mar28 ? 00:00:19 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 29493 0 56 80 0 - 1210397 - Mar28 ? 00:00:07 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 32223 0 56 80 0 - 1210397 - Mar28 ? 00:00:14 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 32224 0 56 80 0 - 1210397 - Mar28 ? 00:00:20 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 32225 0 56 80 0 - 1210397 - Mar28 ? 00:00:17 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 42257 0 56 80 0 - 1210397 - Mar28 ? 00:00:23 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 42258 0 56 80 0 - 1210397 - Mar28 ? 00:00:22 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 42259 0 56 80 0 - 1210397 - Mar28 ? 00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 511816 0 56 80 0 - 1210397 - Mar28 ? 00:00:18 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 1502049 0 56 80 0 - 1210397 - Mar29 ? 00:00:13 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 2081398 0 56 80 0 - 1210397 - Mar29 ? 00:00:12 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 2086878 0 56 80 0 - 1210397 - Mar29 ? 00:00:14 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 2088461 0 56 80 0 - 1210397 - Mar29 ? 00:00:09 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml 1 S 1000570+ 15510 15467 2088462 0 56 80 0 - 1210397 - Mar29 ? 00:00:11 console operator -v=2 --config=/var/run/configmaps/config/controller-config.yaml [mchebbi@fedora process]$ ------------------ thanks in advance for your help.
Started the review process but dint have time to finish due to work on stories and bugs.
Hey Sam, @mchebbi while investigating the issue, I created 600 VMs, and 600 PVCs to measure the performance between the views. I did notice a lower performance on the VMs page while making a search query. this could happen for couple of reasons: 1. The table has more columns which are more complex to render 2. The VMs page has another tab for VM Templates which has a table of it self, could cause a more complex rendering aswell. 3. All the rows are rendered, IMO we should introduce infinite scrolling or pagination in the future, which will massively improve performance thoughts?
(In reply to Gilad Lekner from comment #3) > Hey Sam, @mchebbi > while investigating the issue, I created 600 VMs, and 600 PVCs to measure > the performance between the views. > I did notice a lower performance on the VMs page while making a search query. > this could happen for couple of reasons: > > 1. The table has more columns which are more complex to render > 2. The VMs page has another tab for VM Templates which has a table of it > self, could cause a more complex rendering aswell. > 3. All the rows are rendered, IMO we should introduce infinite scrolling or > pagination in the future, which will massively improve performance > > thoughts? @Gilad Lekner: I totally agree with you, infinite scrolling or pagination could improve performance.
Created attachment 1788642 [details] load virtualization tab over 600 VMs Create 600 VMs(status is off) on OCP 4.8, the virtualization tab is able to load and the PC which used to launch the web console is not in high CPU.
@mchebbi hey, https://github.com/openshift/console/pull/8821 is merged. can you please check to see if the issue is fixed?
In comment #5, we could not reproduce the issue, we will move to next release to debug in the future.
pr is open, commenting for redundant needinfo
Hi, Is there any chance to ask the customer to verify this bug? I see good results on 4.9.0-fc.0 by loading 134 running VMIs, less than 1s. Since I could not reproduce the bug, I'm not sure whether my tests are enough. Thanks, Guohua
moving back to assigned for an additional optimization
Created attachment 1839110 [details] navigate with 385 running VMs Tested in a cluster which has 385 running VMs and 200+ off VMs, navigate in the virtualization and template page has no issues. Move the bug to verified based on the test results, feel free to reopen it when later bugs is found.
Could still see the issue on the cluster with the latest console(build from master): https://console-openshift-console.apps.ocp-virt.prod.psi.redhat.com/dashboards. 1. CPU consumer is high when viewing dashboards. 2. CUP consumer is high when viewing VM list page. 3. CPU consumer is also high when viewing Workloads -> Pods. Based on 1 and 3, I guess this is a common OCP or web interface issue because it requests too many resources from the backend at once. Move the bug back to assigned for further investigation.
@gouyang @yzamir Created another PR https://github.com/openshift/console/pull/10572 In Attempt to further optimize the VM list status rendering. The Lazy VM Status won't be computed (before it was lazy computed) if the printable status is one of the following: printableStatus === VMStatusSimpleLabel.Paused || printableStatus === VMStatusSimpleLabel.Unknown || printableStatus === VMStatusSimpleLabel.Stopping || printableStatus === VMStatusSimpleLabel.Stopped || printableStatus === VMStatusSimpleLabel.Running
Customer closed the customer support, https://access.redhat.com/support/cases/#/case/02902867/discussion?commentId=a0a2K00000ck2y9QAA I prefere not to backport more fixes to 4.10, Guohua, hi can we verify this bug based on customer closing the support case as resolved ?
I agree we close the bug as customer close the case already. Set the resolution to NextRelease as it has good results on 4.10
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days