Bug 1390484 - Postgres DB overloads the CPU when specific bookmarks queries are triggered.
Summary: Postgres DB overloads the CPU when specific bookmarks queries are triggered.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.0.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.1.0-alpha
: ---
Assignee: Martin Perina
QA Contact: Lucie Leistnerova
URL:
Whiteboard:
Depends On:
Blocks: 1388462 1398550
TreeView+ depends on / blocked
 
Reported: 2016-11-01 08:22 UTC by Roman Hodain
Modified: 2020-03-11 15:20 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1398550 (view as bug list)
Environment:
Last Closed: 2017-04-25 00:56:01 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
processes_queries (7.33 KB, text/plain)
2017-01-30 16:48 UTC, Lucie Leistnerova
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:0997 0 normal SHIPPED_LIVE Red Hat Virtualization Manager (ovirt-engine) 4.1 GA 2017-04-18 20:11:26 UTC
oVirt gerrit 66165 0 'None' MERGED core: improving vms_with_tags view performance 2020-09-03 12:08:17 UTC

Description Roman Hodain 2016-11-01 08:22:41 UTC
Description of problem:
     When the following bookmark is chosen 

          Events: Users=xxx  or vm=yyyy

     the postgres DB gets overloaded in case of high amount of VMs.

Version-Release number of selected component (if applicable):

     rhv-m 4.0.4.4-0.1.el7ev

How reproducible:

     100%

Steps to Reproduce:
     1. Create 100VMs
     2. Run the query above

Actual results:

     Postgres consumes 100% of CPU and the query is returned in minutes.

Expected results:

     The records are returned in reasonable time

Comment 3 Martin Perina 2016-11-03 08:31:47 UTC
Eli, could you please take a look?

Comment 8 Lucie Leistnerova 2017-01-30 16:47:30 UTC
testing on engine 4.1 with 8GB memory, 2 CPU
events count 200 000
users 300
vms 1000

time 40-50s for query in Search input and query from already saved bookmark
That seems reasonable against minutes.

But there is another problem and engine runs more then one search at one time and to show result then takes minutes. Sometimes it shows result of other search first.

Steps to Reproduce:
1. restart engine
2. have already saved bookmark with query - Events: Users=test-144  or vm=test1-144
3. run this bookmark, search takes 40-50s and runs one postgres process
4. change string in Search input to - Events: Users=test-155  or vm=test1-155 and run it, search takes 2min and more search postgres processes are running

tested in ovirt-engine-4.1.0.2-0.1.el7.noarch

see running processes and queries in postgres in attachment

Comment 9 Lucie Leistnerova 2017-01-30 16:48:56 UTC
Created attachment 1245927 [details]
processes_queries

Comment 10 Yaniv Kaul 2017-01-30 18:14:10 UTC
(In reply to Lucie Leistnerova from comment #8)
> testing on engine 4.1 with 8GB memory, 2 CPU
> events count 200 000
> users 300
> vms 1000

For 1000 VMs, 8GB and 2 CPU (physical engine?) is REALLY low.
What is the storage that is being used?

Did you reproduce the issues previously with the same DB and query and setup and could not see a difference in performance?

> 
> time 40-50s for query in Search input and query from already saved bookmark
> That seems reasonable against minutes.
> 
> But there is another problem and engine runs more then one search at one
> time and to show result then takes minutes. Sometimes it shows result of
> other search first.
> 
> Steps to Reproduce:
> 1. restart engine
> 2. have already saved bookmark with query - Events: Users=test-144  or
> vm=test1-144
> 3. run this bookmark, search takes 40-50s and runs one postgres process
> 4. change string in Search input to - Events: Users=test-155  or
> vm=test1-155 and run it, search takes 2min and more search postgres
> processes are running
> 
> tested in ovirt-engine-4.1.0.2-0.1.el7.noarch
> 
> see running processes and queries in postgres in attachment

Comment 11 Lucie Leistnerova 2017-01-31 09:03:42 UTC
Running bookmark is about 5s faster on engine with 16GB memory, 2 CPU/2 cores and NFS storage.
I tested the progress in ovirt with not the same database but similar number of events and vms. And it went from about 1.5 min to 30 s, so that is by my opinion alright.

Still there are running searches that should not run (when I change the search from bookmark in search input) and it is then slower. Should I create another BZ for it?

Comment 12 Lucie Leistnerova 2017-01-31 09:38:04 UTC
After talking with Yaniv, I will create new BZ for the multiple search problem. 
I tested the search in engine on virtual machine. I think the time is appropriate to it and it is real progress.

verified in ovirt-engine-4.1.0.2-0.1.el7.noarch

Feel free to reopen the bug if the time is still not reasonable for you.


Note You need to log in before you can comment on or make changes to this bug.