Bug 1344428 - [scale] The Dashboard takes a long time to load on a medium scale system (39 Hosts/3K VMs)
Summary: [scale] The Dashboard takes a long time to load on a medium scale system (39 ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: General
Version: 4.0.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-4.1.0-alpha
: 4.1.0
Assignee: Scott Dickerson
QA Contact: Eldad Marciano
URL:
Whiteboard:
: 1376428 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-09 16:10 UTC by Eldad Marciano
Modified: 2017-04-04 12:02 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-01 14:35:25 UTC
oVirt Team: UX
Embargoed:
rule-engine: ovirt-4.1+
rule-engine: blocker+
mgoldboi: planning_ack+
rule-engine: devel_ack+
pstehlik: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 59089 0 master MERGED dashboard: Optimize VM status query 2016-06-21 19:07:29 UTC
oVirt gerrit 59570 0 ovirt-engine-4.0 MERGED dashboard: Optimize VM status query 2016-06-22 06:40:52 UTC
oVirt gerrit 67709 0 master MERGED webadmin: add background cache update to dashboard data servlet 2016-12-07 18:39:31 UTC

Description Eldad Marciano 2016-06-09 16:10:11 UTC
Description of problem:
4.0.7 not working while using FF 42.0 and lower.
for chrome it works prefect.


Version-Release number of selected component (if applicable):
4.0.7

How reproducible:
100%

Steps to Reproduce:
1. open webadmin dashboard tab in FF
2.
3.

Actual results:
dashboard loading indicator running forever.

Expected results:
nice & fast dashboard.

Additional info:

Comment 2 Moran Goldboim 2016-06-14 06:26:56 UTC
does it happen only on FF42? is it working with FF ESR (45.2 currently)?

Comment 3 Yaniv Kaul 2016-06-14 11:09:28 UTC
Since it's 'just' slow, I've reduced severity to high.
I'm also wondering what are the result on the latest FF ESR (as Moran asked), which hopefully improved some performance.

Comment 4 Vojtech Szocs 2016-06-14 15:57:35 UTC
(In reply to Eldad Marciano from comment #0)
> Actual results:
> dashboard loading indicator running forever.

This seems more related to fetching data from Engine, rather than rendering stuff in the browser, is that correct?

Comment 5 Oved Ourfali 2016-06-15 07:49:17 UTC
Moving back to the engine, as the query is the issue.

Comment 6 Gil Klein 2016-06-15 09:31:01 UTC
We can confirm it has nothing to do with the browser. It is slow with many other browsers when running at scale of:

- 2 DCs 
- 3 Clusters 
- 39 Hosts -
- 48 SDs 
- 3000 VMs

Browser that were tested:

Firefox 45.2.0-1.el6_8.x86_64 (ESR): 04m17s
Firefox 46.0.1 x86_64 OpenBSD 6.0 Beta: 04m01s
Firefox 47.0+build3-0ubuntu0.16.amd64: 04m00s
Chromium 50.0.2661.102 Ubuntu 16.04 (64-bit): 00m24s (maybe cache, cannot verify again ATM because of 503 Service Unavailable)

Comment 7 Pavel Novotny 2016-06-15 14:55:26 UTC
(In reply to Gil Klein from comment #6)
> [snip]
> 
> Chromium 50.0.2661.102 Ubuntu 16.04 (64-bit): 00m24s (maybe cache, cannot
> verify again ATM because of 503 Service Unavailable)

After trying again with Chromium 50, the avg. time to load was 4.4 minutes, which is close to the Firefox times. So browser type really doesn't matter in this case.

Comment 12 Eldad Marciano 2016-08-09 11:36:46 UTC
there is some performance improvements but it still takes more than 30 sec to be fully loaded.

seems like we have another query which takes some long duration which fixed in patch set 6 - https://gerrit.ovirt.org/#/c/59089/
https://bugzilla.redhat.com/show_bug.cgi?id=1365443


verified failed qa

Comment 13 Red Hat Bugzilla Rules Engine 2016-08-09 11:36:50 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 14 Oved Ourfali 2016-08-10 12:30:48 UTC
Are you running with the original number of objects?

Comment 15 Eldad Marciano 2016-08-10 19:56:47 UTC
I ran it with the same active vms number, we should double test it with the same existing vms number.

Comment 16 Eldad Marciano 2016-09-04 10:39:54 UTC
I used to refresh my FF cache and verified it on top of 4.0.3-1
and it works pretty nice.

FF version: 47.01

Comment 17 Oved Ourfali 2016-11-23 10:07:21 UTC
The caching will fix it.

Comment 18 Scott Dickerson 2016-12-02 21:12:44 UTC
Periodically updating query caching has been added.  Under normal circumstances each dashboard data fetch will be served from cached data.  The cache is initially populated when the engine starts so the first user to request dashboard data shouldn't incur the full cache warm-up penalty.

Dashboard data is queried in 2 parts: 
  - utilization trends from data warehouse (DWH) tables, and
  - inventory data from standard engine tables.

By default, the utilization cache is updated once every 5 minutes and the inventory cache is updated once every 15 seconds.  Both of these values are configurable with the ovirt-engine.conf mechanism.

Comment 19 Oved Ourfali 2016-12-07 15:07:18 UTC
*** Bug 1376428 has been marked as a duplicate of this bug. ***

Comment 20 Sandro Bonazzola 2016-12-12 13:55:03 UTC
The fix for this issue should be included in oVirt 4.1.0 beta 1 released on December 1st. If not included please move back to modified.

Comment 21 Yaniv Kaul 2016-12-19 12:26:54 UTC
To QE: when verifying the bug, please notice the following and test:
Configuration options have been added to ovirt-engine.conf to:
  - enable/disable the background cache update
  - control the interval between updates for the inventory cache
  - control the interval between updates for the utilization cache

Comment 22 Eldad Marciano 2017-01-19 13:56:32 UTC
(In reply to Yaniv Kaul from comment #21)
> To QE: when verifying the bug, please notice the following and test:
> Configuration options have been added to ovirt-engine.conf to:
>   - enable/disable the background cache update
>   - control the interval between updates for the inventory cache
>   - control the interval between updates for the utilization cache

Ok,
is that ok to verify with defaults?
currently the weak point was with lots of assets a specially vms and disks.

we plan to verify that bug single SD (NFS) with as much as possible 3K vms.

Comment 23 Eldad Marciano 2017-02-01 11:41:59 UTC
didn't reproduced for firefox 45 and chrome 55.0

seems like we can shift this bug to other team, to investigate with lower firefox version.

Comment 25 Eldad Marciano 2017-02-01 12:58:26 UTC
Moving to verify,
currently this bug not reproduced on firefox 45 or chrome 55.0 
engine scale out profile:
- 250 hosts 
- 3.7K vms
- 1 SD
- 1 DC


Note You need to log in before you can comment on or make changes to this bug.