Bug 1398546 - [z-stream clone - 4.0.6] RHV-M Web UI performance degrades over time
Summary: [z-stream clone - 4.0.6] RHV-M Web UI performance degrades over time
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.6.7
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ovirt-4.0.6
: ---
Assignee: Alexander Wels
QA Contact: Pavel Novotny
URL:
Whiteboard:
Depends On: 1368101
Blocks: 1399489
TreeView+ depends on / blocked
 
Reported: 2016-11-25 08:18 UTC by rhev-integ
Modified: 2021-03-11 15:15 UTC (History)
23 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1368101
: 1399489 (view as bug list)
Environment:
Last Closed: 2017-01-10 17:01:12 UTC
oVirt Team: UX
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
IE11 webadmin: VMs list gets blank with high RAM usage (181.48 KB, image/png)
2016-12-15 18:02 UTC, Pavel Novotny
no flags Details
RAM usage chart: 4.0.5 vs. 4.0.6 (37.46 KB, image/png)
2016-12-20 18:27 UTC, Pavel Novotny
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2639361 0 None None None 2016-11-25 08:29:18 UTC
Red Hat Product Errata RHBA-2017:0043 0 normal SHIPPED_LIVE Red Hat Virtualization Manager 4.0.6 2017-01-10 21:52:43 UTC
oVirt gerrit 54186 0 master MERGED webadmin: Improve column control menu 2020-02-26 12:05:34 UTC
oVirt gerrit 63708 0 master MERGED webadmin,userportal: Improve UiCommon model cleanup 2020-02-26 12:05:34 UTC
oVirt gerrit 64470 0 master MERGED webadmin: VM dialog mem leak 2020-02-26 12:05:33 UTC
oVirt gerrit 64539 0 master MERGED webadmin: Fix memory leak in Permission popup 2020-02-26 12:05:33 UTC
oVirt gerrit 65092 0 master MERGED webadmin: Add TODO for future fixing 2020-02-26 12:05:35 UTC
oVirt gerrit 65566 0 None MERGED webadmin,userportal: Improve Bootstrap/jQuery tooltip infra 2020-02-26 12:05:33 UTC
oVirt gerrit 65733 0 None MERGED webadmin,userportal: Overhaul UiCommonEditorDriver cleanup() impl. 2020-02-26 12:05:33 UTC

Comment 1 rhev-integ 2016-11-25 08:19:09 UTC
Description of problem:
The web admin portal is slow in IE 11. The same client behaves much better with chrome browser on the same client system. Although the chrome browser behaves better, the performance degradates over the time.

Version-Release number of selected component (if applicable):
    rhevm-3.6.6.2-0.1.el6

How reproducible:
    100% with IE11
    after a while with other browsers

Steps to Reproduce:
1. Start IE11 and connect to the webAmin
2. Work with that for a while

Actual results:
    The memory grows rapidly 1GRAM in 10 min. The average CPU load is ~60% and 99% in peaks. even though the browser is not used. This is most visible in the Virtual Machine tab.

Expected results:
    The usage is fluent. No delay between tab switching. (not counting the data loading)

This comment was originaly posted by rhodain

Comment 57 rhev-integ 2016-11-25 08:26:53 UTC
*** Bug 1348150 has been marked as a duplicate of this bug. ***

This comment was originaly posted by gshereme

Comment 64 rhev-integ 2016-11-25 08:27:51 UTC
I have good news on our progress. After applying several patches to fix memory leaks (and one more not-yet-merged patch), I can say we are now very stable on 4.1 master. I've done several performance tests to verify. The most important one was an endurance test I did that shows webadmin still very stable after 6.25 hours of heavy use under webdriver.

See attachment '6 hour webdriver results'.

This shows that we're still leaking some memory over time (graph on right), and while we have more work to do on that, the relatively flat response time line (graph on left) means the app stayed performant over the 6 hour test.

[Ignore the spike in the left graph. That was a one-time webdriver hiccup.]

This comment was originaly posted by gshereme

Comment 65 rhev-integ 2016-11-25 08:27:58 UTC
Created attachment 1220554 [details]
6 hour webdriver results

This comment was originaly posted by gshereme

Comment 69 rhev-integ 2016-11-25 08:28:30 UTC
(In reply to Greg Sheremeta from comment #62)
> I have good news on our progress. After applying several patches to fix
> memory leaks (and one more not-yet-merged patch), I can say we are now very
> stable on 4.1 master. I've done several performance tests to verify. The
> most important one was an endurance test I did that shows webadmin still
> very stable after 6.25 hours of heavy use under webdriver.

This is great news.

All memory leak fixes should land in 4.0.6 eventually.

Once that happens, we can decide how to proceed regarding 3.6.z.

> 
> See attachment '6 hour webdriver results'.
> 
> This shows that we're still leaking some memory over time (graph on right),
> and while we have more work to do on that, the relatively flat response time
> line (graph on left) means the app stayed performant over the 6 hour test.
> 
> [Ignore the spike in the left graph. That was a one-time webdriver hiccup.]

Given Greg's performance analysis results, I'd conclude this BZ once the "not-yet-merged patch" lands in 4.0.6.

Any further improvements should be done as part of tracker bug 1378935.

This comment was originaly posted by vszocs

Comment 70 rhev-integ 2016-11-25 08:28:38 UTC
Attaching another test result that shows the dramatic improvements made by Vojtech's awesome tooltips fix.

See 'tooltip leak fix results.png'

The flat lines are with tooltips fix onboard. The mountainous slope is without it. The degradation was severe, as you can see.

That patch alone shaved 40 minutes off of an hour run, so we are now 66% faster.

:)

This comment was originaly posted by gshereme

Comment 71 rhev-integ 2016-11-25 08:28:46 UTC
Created attachment 1220952 [details]
tooltip leak fix results

This comment was originaly posted by gshereme

Comment 72 rhev-integ 2016-11-25 08:28:55 UTC
*** Bug 1395911 has been marked as a duplicate of this bug. ***

This comment was originaly posted by oourfali

Comment 77 Pavel Novotny 2016-12-15 18:01:24 UTC
Interim results for ovirt-engine-webadmin-portal-4.0.6.3-0.1.el7ev.noarch:

engine evironment: 1 host, 1 nfs storage, 41 VMs (blank, w/o OS), 4 of them running
Client machine (VM): Windows 8.1 Ent., 8 GB RAM, dual-core (virtualized from baremetal host has Intel Xeon E5-2420 v2 @ 2.20GHz); IE 11

The manual scenario was to go through all main tabs and in each select the first row to load the sub-tab panel.
Then I just opened and closed following dialogs: New VM dialog + New VM Disk, Make Template and New VM Pool dialog.
Then I left the browser opened on the VMs tab and monitored the IE RAM consumption.

Take #1:
time elapsed: IE RAM usage
--------------------------
00m: 190 MB (right after login to webadmin)
10m: 610 MB
45m: 1 653 MB (the VM list gets blank & UI gets completely frozen, page must be reloaded to fix it)
See the screenshot atached for details.

Take #2:
00m: 187 MB (right after login)
10m: 623 MB
45m: 1 667 MB (blank VM list, fromzen UI, reload needed)

So basically after 45 minutes IE gets fried even with a small RHV environment.
However, I need to compare these numbers with 4.0.5 to see if it differs
(basically if 4.0.6 numbers are an improvement or not).

Comment 78 Pavel Novotny 2016-12-15 18:02:32 UTC
Created attachment 1232294 [details]
IE11 webadmin: VMs list gets blank with high RAM usage

Comment 84 Pavel Novotny 2016-12-20 18:26:04 UTC
Verified in 
rhevm-4.0.6.3-0.1.el7ev.noarch
ovirt-engine-webadmin-portal-4.0.6.3-0.1.el7ev.noarch

Browser: Firefox 45.6 ESR @ RHEL 6.8

I ran a test which (in a loop) went through the main tabs and then opened & closed the New VM dialog. 
I used 100 passes and for each measured the RAM usage via the browser built-in tool at the about:memory page.

Results:
4.0.5: RAM usage started around 100 MB and continuously grew up to ~850 MB.
4.0.6: Started at ~140 MB and stayed around this value the whole time.
(chart attached)

Timewise, 100 passes took:
4.0.5: 3h55m
4.0.6: 2h6m

The improvements in RAM usage and in UI performance are obvious.
Please see the RAM usage chart attached.

Comment 85 Pavel Novotny 2016-12-20 18:27:17 UTC
Created attachment 1234007 [details]
RAM usage chart: 4.0.5 vs. 4.0.6

Comment 87 Vojtech Szocs 2017-01-03 12:00:24 UTC
Thanks Pavel for the verification!

> Browser: Firefox 45.6 ESR @ RHEL 6.8
> 4.0.6: Started at ~140 MB and stayed around this value the whole time. (chart attached)
> Test took 2h6m [100 passes]

This proves that memory usage of (mem-leak-patched) WebAdmin in Firefox is generally consistent.

> Windows 8.1 Ent., IE 11
> After 45m: 1 653 MB (the VM list gets blank & UI gets completely frozen, page must be reloaded to fix it)

This proves that IE 11, given practically the same application [*], still suffers from the inability to deal with complex JavaScript code.

[*] WebAdmin GWT permutation for IE 11 vs. Firefox diff == bridging gaps (APIs, standard compliance) between browsers only; actual application logic is the same

Any customer using IE 11 should be aware of the above.

Comment 89 errata-xmlrpc 2017-01-10 17:01:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0043.html


Note You need to log in before you can comment on or make changes to this bug.