Bug 1438962 - DB query cache auto-eviction could be longer then the query run time in a large environment
Summary: DB query cache auto-eviction could be longer then the query run time in a lar...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: Frontend.WebAdmin
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ovirt-4.1.2
: 4.1.2
Assignee: Scott Dickerson
QA Contact: Pavel Novotny
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-04 21:52 UTC by Scott Dickerson
Modified: 2017-05-23 08:21 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-23 08:21:34 UTC
oVirt Team: UX
Embargoed:
rule-engine: ovirt-4.1?
pnovotny: planning_ack?
sdickers: devel_ack+
pnovotny: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 75305 0 master MERGED webadmin: Adjust DashboardDataServlet query cache handling 2017-04-06 14:10:18 UTC
oVirt gerrit 75308 0 ovirt-engine-4.1 MERGED webadmin: Adjust DashboardDataServlet query cache handling 2017-04-07 07:45:08 UTC

Description Scott Dickerson 2017-04-04 21:52:35 UTC
DashboardDataServlet maintains a DB query cache of the data it servers to the frontend UI plugin.  In a sufficiently large environment, it is possible that the auto-eviction time will be shorter then the run time of the query.

Currently the query cache is updated at a fixed rate, regardless of how long the query takes to run.  Also, if the cache is empty at the time of a data request, the servlet will start a new query to return non-cached results.

If the dashboard DWH queries take longer then 5 minutes to run, there could be problems.  Every 5 minutes the background updater will start a new query, even though the previous hasn't finished yet.  Between the 5 minute mark and the time when the initial query completes, a user may request the data and cause a 3rd query to start.  Now 3 queries will be running.  

The user who caused the 3rd query to be spawned will have to wait for that query to complete before getting their data.  The 1st query will be pushed into the cache, but only for 5 minutes.  The 2nd query will complete and it will be pushed into the cache for 5 minutes.  All the while another query is kicked off every 5 minutes by the background cache update.

The easiest way to fix this problem is 2 fold.  First, change the background cache update to wait a certain amount of time AFTER the task is completed instead of starting a task at a regular interval.  Second, change the cache itself so it doesn't auto-evict the data.  There will always be data in the cache and the background updater will never have multiple update queries running at the same time.

Comment 1 Yaniv Kaul 2017-04-05 06:43:27 UTC
If the query take more than 5 minutes, we have a very very serious bug!
Nevertheless, I agree it should fire the next event 5 minutes after the last cache was refreshed, not every 5 minutes.

Comment 2 Greg Sheremeta 2017-05-05 15:58:05 UTC
[QE: not really anything to test here. It's a code change / refactor. As long as dashboard loads and successfully does its usual partial refresh after 5 minutes, it's good.]

Comment 3 Pavel Novotny 2017-05-10 14:59:25 UTC
Based on Greg's comment, adding CodeChange keyword.

Dashboard sanity passed, no regressions found, moving to verified:
  rhevm-4.1.2.1-0.1.el7.noarch
  ovirt-engine-dashboard-1.1.2-1.el7ev.noarch


Note You need to log in before you can comment on or make changes to this bug.