Bug 1368101 - RHV-M Web UI performance degrades over time
Summary: RHV-M Web UI performance degrades over time
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.6.7
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ovirt-4.1.0-alpha
: ---
Assignee: Alexander Wels
QA Contact: Pavel Novotny
URL:
Whiteboard:
Keywords: Performance, ZStream
: 1348150 1395911 (view as bug list)
Depends On:
Blocks: 1417161 1293920 1388462 1398546
TreeView+ depends on / blocked
 
Reported: 2016-08-18 12:15 UTC by Roman Hodain
Modified: 2019-04-28 09:20 UTC (History)
29 users (show)

(edit)
Clone Of:
: 1398546 (view as bug list)
(edit)
Last Closed: 2017-04-25 00:47:57 UTC
bmcclain: priority_rfe_tracking+


Attachments (Terms of Use)
6 hour webdriver results (93.89 KB, image/png)
2016-11-14 23:00 UTC, Greg Sheremeta
no flags Details
tooltip leak fix results (86.08 KB, image/png)
2016-11-15 20:22 UTC, Greg Sheremeta
no flags Details
RAM usage chart: 4.0.6 vs. 4.1.0 (43.60 KB, image/png)
2017-02-08 16:14 UTC, Pavel Novotny
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:0997 normal SHIPPED_LIVE Red Hat Virtualization Manager (ovirt-engine) 4.1 GA 2017-04-18 20:11:26 UTC
oVirt gerrit 54186 master MERGED webadmin: Improve column control menu 2016-10-06 17:29 UTC
oVirt gerrit 63708 master MERGED webadmin,userportal: Improve UiCommon model cleanup 2016-09-23 12:37 UTC
oVirt gerrit 64470 master MERGED webadmin: VM dialog mem leak 2016-10-03 13:09 UTC
oVirt gerrit 64539 master MERGED webadmin: Fix memory leak in Permission popup 2016-10-07 14:20 UTC
oVirt gerrit 65092 master MERGED webadmin: Add TODO for future fixing 2016-10-04 13:59 UTC
oVirt gerrit 65566 None None None 2016-10-18 15:15 UTC
oVirt gerrit 65733 None None None 2016-10-25 16:29 UTC
Red Hat Knowledge Base (Solution) 2639361 None None None 2016-09-19 11:28 UTC

Description Roman Hodain 2016-08-18 12:15:55 UTC
Description of problem:
The web admin portal is slow in IE 11. The same client behaves much better with chrome browser on the same client system. Although the chrome browser behaves better, the performance degradates over the time.

Version-Release number of selected component (if applicable):
    rhevm-3.6.6.2-0.1.el6

How reproducible:
    100% with IE11
    after a while with other browsers

Steps to Reproduce:
1. Start IE11 and connect to the webAmin
2. Work with that for a while

Actual results:
    The memory grows rapidly 1GRAM in 10 min. The average CPU load is ~60% and 99% in peaks. even though the browser is not used. This is most visible in the Virtual Machine tab.

Expected results:
    The usage is fluent. No delay between tab switching. (not counting the data loading)

Comment 55 Greg Sheremeta 2016-11-09 14:04:59 UTC
*** Bug 1348150 has been marked as a duplicate of this bug. ***

Comment 62 Greg Sheremeta 2016-11-14 22:59:11 UTC
I have good news on our progress. After applying several patches to fix memory leaks (and one more not-yet-merged patch), I can say we are now very stable on 4.1 master. I've done several performance tests to verify. The most important one was an endurance test I did that shows webadmin still very stable after 6.25 hours of heavy use under webdriver.

See attachment '6 hour webdriver results'.

This shows that we're still leaking some memory over time (graph on right), and while we have more work to do on that, the relatively flat response time line (graph on left) means the app stayed performant over the 6 hour test.

[Ignore the spike in the left graph. That was a one-time webdriver hiccup.]

Comment 63 Greg Sheremeta 2016-11-14 23:00 UTC
Created attachment 1220554 [details]
6 hour webdriver results

Comment 67 Vojtech Szocs 2016-11-15 16:14:14 UTC
(In reply to Greg Sheremeta from comment #62)
> I have good news on our progress. After applying several patches to fix
> memory leaks (and one more not-yet-merged patch), I can say we are now very
> stable on 4.1 master. I've done several performance tests to verify. The
> most important one was an endurance test I did that shows webadmin still
> very stable after 6.25 hours of heavy use under webdriver.

This is great news.

All memory leak fixes should land in 4.0.6 eventually.

Once that happens, we can decide how to proceed regarding 3.6.z.

> 
> See attachment '6 hour webdriver results'.
> 
> This shows that we're still leaking some memory over time (graph on right),
> and while we have more work to do on that, the relatively flat response time
> line (graph on left) means the app stayed performant over the 6 hour test.
> 
> [Ignore the spike in the left graph. That was a one-time webdriver hiccup.]

Given Greg's performance analysis results, I'd conclude this BZ once the "not-yet-merged patch" lands in 4.0.6.

Any further improvements should be done as part of tracker bug 1378935.

Comment 68 Greg Sheremeta 2016-11-15 20:22:07 UTC
Attaching another test result that shows the dramatic improvements made by Vojtech's awesome tooltips fix.

See 'tooltip leak fix results.png'

The flat lines are with tooltips fix onboard. The mountainous slope is without it. The degradation was severe, as you can see.

That patch alone shaved 40 minutes off of an hour run, so we are now 66% faster.

:)

Comment 69 Greg Sheremeta 2016-11-15 20:22 UTC
Created attachment 1220952 [details]
tooltip leak fix results

Comment 70 Oved Ourfali 2016-11-23 11:48:34 UTC
*** Bug 1395911 has been marked as a duplicate of this bug. ***

Comment 73 Vojtech Szocs 2016-11-25 16:35:22 UTC
(In reply to Greg Sheremeta from comment #68)
> Attaching another test result that shows the dramatic improvements made by
> Vojtech's awesome tooltips fix.

It's the combination of all memory leak fixes, not only the tooltip one :) but at least we've somewhat stabilized the memory growth, so users don't have to reload WebAdmin UI that often.

Comment 79 Greg Sheremeta 2017-02-08 14:25:20 UTC
Hi Jack,

Since you have both UI and API slowness, that makes me wonder if engine slowness is the cause of both. But in order to test that theory, you would need to upgrade your engine to 3.6.10 as Oved said in Comment 78, so you get our UI improvements.

Is it possible to upgrade to 3.6.10?

Comment 80 Pavel Novotny 2017-02-08 16:13:20 UTC
Verified in 
rhevm-4.1.0.4-0.1.el7.noarch
ovirt-engine-webadmin-portal-4.1.0.4-0.1.el7.noarch

Browser: Firefox 45.7 ESR @ RHEL 6.8

I ran a test which went in a loop through the main tabs and then opened & closed the New VM dialog. 
I used 100 passes and for each one I measured the RAM usage via the browser built-in tool at the about:memory page.
I compared the data against rhevm-4.0.6.3-0.1.el7ev, which already contains
some memory usage improvements (see bug 1398546).

Results:
4.0.6: Started at ~140 MB and stayed around this value. The average was 160 MB.
4.1.0: Started at ~120 MB or less and stayed around this value. The average was 119 MB.
(chart attached)

Timewise, 100 passes took:
4.0.6: 2h6m
4.1.0: 1h34m (25 % faster)

Comment 81 Pavel Novotny 2017-02-08 16:14 UTC
Created attachment 1248637 [details]
RAM usage chart: 4.0.6 vs. 4.1.0

Comment 82 Yaniv Kaul 2017-02-08 17:38:18 UTC
(In reply to Pavel Novotny from comment #80)
> Verified in 
> rhevm-4.1.0.4-0.1.el7.noarch
> ovirt-engine-webadmin-portal-4.1.0.4-0.1.el7.noarch
> 
> Browser: Firefox 45.7 ESR @ RHEL 6.8
> 
> I ran a test which went in a loop through the main tabs and then opened &
> closed the New VM dialog. 
> I used 100 passes and for each one I measured the RAM usage via the browser
> built-in tool at the about:memory page.

How many objects (hosts, VMs) did you have in the system?

> I compared the data against rhevm-4.0.6.3-0.1.el7ev, which already contains
> some memory usage improvements (see bug 1398546).
> 
> Results:
> 4.0.6: Started at ~140 MB and stayed around this value. The average was 160
> MB.
> 4.1.0: Started at ~120 MB or less and stayed around this value. The average
> was 119 MB.
> (chart attached)
> 
> Timewise, 100 passes took:
> 4.0.6: 2h6m
> 4.1.0: 1h34m (25 % faster)

Nice. Any measurements of network usage? average latency, etc.?

Comment 83 Pavel Novotny 2017-02-08 17:51:02 UTC
(In reply to Yaniv Kaul from comment #82)
[snip]
> > I ran a test which went in a loop through the main tabs and then opened &
> > closed the New VM dialog. 
> > I used 100 passes and for each one I measured the RAM usage via the browser
> > built-in tool at the about:memory page.
> 
> How many objects (hosts, VMs) did you have in the system?

Relatively small environment:
1 host, 1 NFS storage, 41 VMs (blank, w/o OS installed), 4 of them Up

> 
[snip]
> > Timewise, 100 passes took:
> > 4.0.6: 2h6m
> > 4.1.0: 1h34m (25 % faster)
> 
> Nice. Any measurements of network usage? average latency, etc.?

No, I just measured the RAM usage and overall elapsed time.

Comment 84 Yaniv Kaul 2017-02-08 18:23:45 UTC
(In reply to Pavel Novotny from comment #83)
> (In reply to Yaniv Kaul from comment #82)
> [snip]
> > > I ran a test which went in a loop through the main tabs and then opened &
> > > closed the New VM dialog. 
> > > I used 100 passes and for each one I measured the RAM usage via the browser
> > > built-in tool at the about:memory page.
> > 
> > How many objects (hosts, VMs) did you have in the system?
> 
> Relatively small environment:
> 1 host, 1 NFS storage, 41 VMs (blank, w/o OS installed), 4 of them Up

Quite a small environment - if you have a bigger one to test on, that'd be great (with nested virt you can have many more, I reckon).

> 
> > 
> [snip]
> > > Timewise, 100 passes took:
> > > 4.0.6: 2h6m
> > > 4.1.0: 1h34m (25 % faster)
> > 
> > Nice. Any measurements of network usage? average latency, etc.?
> 
> No, I just measured the RAM usage and overall elapsed time.

OK, thanks.


Note You need to log in before you can comment on or make changes to this bug.