Bug 971237 - RHEVM slow due to stored procedure getdisksvmguid() consuming most CPU
Summary: RHEVM slow due to stored procedure getdisksvmguid() consuming most CPU
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.1.3
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 3.3.0
Assignee: Eli Mesika
QA Contact: Barak Dagan
URL:
Whiteboard: infra
Depends On:
Blocks: 998523
TreeView+ depends on / blocked
 
Reported: 2013-06-06 05:52 UTC by Mark Huth
Modified: 2018-12-03 19:04 UTC (History)
21 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the getdisksvmguid() procedure would consume a large quantity of available CPU resources due to the inclusion of a large volume of irrelevant data in database queries, causing those queries to be slow. With this update, queries now only retrieve relevant data, increasing the speed of queries.
Clone Of:
: 986700 998523 (view as bug list)
Environment:
Last Closed: 2014-01-21 17:26:09 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2014:0038 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Virtualization Manager 3.3.0 update 2014-01-21 22:03:06 UTC
oVirt gerrit 16657 0 None None None Never
oVirt gerrit 18268 0 None None None Never

Comment 2 Kevein Liu 2013-06-13 02:01:07 UTC
Hi team,

Any update here? 
Please contact me if you need any assistance.
Thank you,

Kevein

Comment 10 Liran Zelkha 2013-06-25 10:16:52 UTC
Any chance you can supply a stack trace of the engine server when this happens?

Comment 11 Kevein Liu 2013-06-25 11:30:41 UTC
(In reply to Liran Zelkha from comment #10)
> Any chance you can supply a stack trace of the engine server when this
> happens?

Sure, I can do it. But could you please tell me how to do it?
Can I just use this command "echo t > /proc/sysrq-trigger" ?

Regards,
Kevein

Comment 12 Liran Zelkha 2013-06-25 12:52:42 UTC
Hi Kevein,

You should use the jstack utility. It should be under JDK_HOME/bin/jstack. 
jstack -l -F engine_pid >jstack.txt

It's best if you can do it 2-3 times, so I can compare the results and see if we have some stuck threads.

Comment 14 Kevein Liu 2013-06-26 01:12:31 UTC
(In reply to Liran Zelkha from comment #10)
> Any chance you can supply a stack trace of the engine server when this
> happens?

Hi,
Could you please check comment #13? The high CPU usage was not caused by java process, but `postmaster`. And we can see the main tasks are "getdisksvmguid()".

Cheers,
Kevein

Comment 17 Barak 2013-07-11 10:21:12 UTC
The problem occurs on UP basic view,
each 5 sec (when in focus) the browser retrieves all the information about the VM in focus.
I don't see any reason to refresh the information displayed on the right side of the screen(specifically the disks, but it can be applies also on ("Operating system, defined memory and number of cores). as it rarely changes.

I would say that this information should be retrieved only when changing focus of VM (and of course at login time ... which that has a default focused VM), and all the rest of the refreshes should bring the minimal information required to update the VM status.

Comment 18 Barak 2013-07-11 10:23:10 UTC
(In reply to Barak from comment #17)
> The problem occurs on UP basic view,
> each 5 sec (when in focus) the browser retrieves all the information about
> the VM in focus.
> I don't see any reason to refresh the information displayed on the right
> side of the screen(specifically the disks, but it can be applies also on
> ("Operating system, defined memory and number of cores). as it rarely
> changes.
> 
> I would say that this information should be retrieved only when changing
> focus of VM (and of course at login time ... which that has a default
> focused VM), and all the rest of the refreshes should bring the minimal
> information required to update the VM status.

Continuing that thought,
If someone edits the VM and changes this information (rarely changed), than it looks reasonable to simply change focus or even re-login.

Comment 19 Andrew Cathrow 2013-07-11 11:22:11 UTC
> 
> Continuing that thought,
> If someone edits the VM and changes this information (rarely changed), than
> it looks reasonable to simply change focus or even re-login.

or hit refresh.

ACK agree.

Comment 20 Itamar Heim 2013-07-11 11:43:29 UTC
(In reply to Andrew Cathrow from comment #19)
> > 
> > Continuing that thought,
> > If someone edits the VM and changes this information (rarely changed), than
> > it looks reasonable to simply change focus or even re-login.
> 
> or hit refresh.

that's a distinction between manual and auto refresh (which i'm fine with, just not implied by barak's comment).

barak - everything else on the right would be refreshed by refreshing the vm status probably anyway - it all comes from the same table. i would just avoid asking to refresh the disks on auto-refresh.

question: what happens if a new VM is added to the user? since it is via auto-refresh which isn't populating the disks, it will be without the disks until a manual refresh will be hit or focus change (i'm fine with that - just need to remember while implementing)

Comment 21 Barak 2013-07-11 16:56:03 UTC
(In reply to Itamar Heim from comment #20)
> (In reply to Andrew Cathrow from comment #19)
> > > 
> > > Continuing that thought,
> > > If someone edits the VM and changes this information (rarely changed), than
> > > it looks reasonable to simply change focus or even re-login.
> > 
> > or hit refresh.
> 
> that's a distinction between manual and auto refresh (which i'm fine with,
> just not implied by barak's comment).
> 
> barak - everything else on the right would be refreshed by refreshing the vm
> status probably anyway - it all comes from the same table. i would just
> avoid asking to refresh the disks on auto-refresh.
> 
> question: what happens if a new VM is added to the user? since it is via
> auto-refresh which isn't populating the disks, it will be without the disks
> until a manual refresh will be hit or focus change (i'm fine with that -
> just need to remember while implementing)

Adding a new VM will continue behave the same as today (when a VM is added while user logged in to UP) meaning when this VM will get focus it will retrieve info with Disks all the rest will be without the disk info

Comment 22 Itamar Heim 2013-07-11 17:20:50 UTC
just make sure getting the focus isn't happening for the VM after every refresh, only on the first time it appears (if focus happened to be set to it automatically)

Comment 26 Eli Mesika 2013-08-19 10:49:07 UTC
fixed in commit: f00e810

Comment 28 Barak Dagan 2013-09-22 14:59:20 UTC
following https://bugzilla.redhat.com/show_bug.cgi?id=986700#c3, verified using is15

1. create 10 VMs with 3 disks each.
2. add different users with "view VM" permissions to the VMs
3. opened 7 UserPortals browsers with different users
4. check that all disks appear for all vms
5. from webadmin, remove 2 out of 3 disks (for 3 VMs, one after another)
6, return to the UserPortal and check nothing updated.
7, switch between VM's tabs or wait for 10 seconds - check that disks list is updated.

Comment 29 Charlie 2013-11-28 00:14:49 UTC
This bug is currently attached to errata RHEA-2013:15231. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.

Comment 30 errata-xmlrpc 2014-01-21 17:26:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0038.html


Note You need to log in before you can comment on or make changes to this bug.