Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 971237

Summary: RHEVM slow due to stored procedure getdisksvmguid() consuming most CPU
Product: Red Hat Enterprise Virtualization Manager Reporter: Mark Huth <mhuth>
Component: ovirt-engineAssignee: Eli Mesika <emesika>
Status: CLOSED ERRATA QA Contact: Barak Dagan <bdagan>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.1.3CC: acathrow, adahms, amureini, bazulay, bdagan, cpelland, eedri, emesika, hchiramm, iheim, jkt, jwang, lpeer, lyarwood, mhuth, pstehlik, Rhev-m-bugs, yaliu, yeylon, yqu, yzaslavs
Target Milestone: ---Keywords: ZStream
Target Release: 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, the getdisksvmguid() procedure would consume a large quantity of available CPU resources due to the inclusion of a large volume of irrelevant data in database queries, causing those queries to be slow. With this update, queries now only retrieve relevant data, increasing the speed of queries.
Story Points: ---
Clone Of:
: 986700 998523 (view as bug list) Environment:
Last Closed: 2014-01-21 17:26:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 998523    

Comment 2 Kevein Liu 2013-06-13 02:01:07 UTC
Hi team,

Any update here? 
Please contact me if you need any assistance.
Thank you,

Kevein

Comment 10 Liran Zelkha 2013-06-25 10:16:52 UTC
Any chance you can supply a stack trace of the engine server when this happens?

Comment 11 Kevein Liu 2013-06-25 11:30:41 UTC
(In reply to Liran Zelkha from comment #10)
> Any chance you can supply a stack trace of the engine server when this
> happens?

Sure, I can do it. But could you please tell me how to do it?
Can I just use this command "echo t > /proc/sysrq-trigger" ?

Regards,
Kevein

Comment 12 Liran Zelkha 2013-06-25 12:52:42 UTC
Hi Kevein,

You should use the jstack utility. It should be under JDK_HOME/bin/jstack. 
jstack -l -F engine_pid >jstack.txt

It's best if you can do it 2-3 times, so I can compare the results and see if we have some stuck threads.

Comment 14 Kevein Liu 2013-06-26 01:12:31 UTC
(In reply to Liran Zelkha from comment #10)
> Any chance you can supply a stack trace of the engine server when this
> happens?

Hi,
Could you please check comment #13? The high CPU usage was not caused by java process, but `postmaster`. And we can see the main tasks are "getdisksvmguid()".

Cheers,
Kevein

Comment 17 Barak 2013-07-11 10:21:12 UTC
The problem occurs on UP basic view,
each 5 sec (when in focus) the browser retrieves all the information about the VM in focus.
I don't see any reason to refresh the information displayed on the right side of the screen(specifically the disks, but it can be applies also on ("Operating system, defined memory and number of cores). as it rarely changes.

I would say that this information should be retrieved only when changing focus of VM (and of course at login time ... which that has a default focused VM), and all the rest of the refreshes should bring the minimal information required to update the VM status.

Comment 18 Barak 2013-07-11 10:23:10 UTC
(In reply to Barak from comment #17)
> The problem occurs on UP basic view,
> each 5 sec (when in focus) the browser retrieves all the information about
> the VM in focus.
> I don't see any reason to refresh the information displayed on the right
> side of the screen(specifically the disks, but it can be applies also on
> ("Operating system, defined memory and number of cores). as it rarely
> changes.
> 
> I would say that this information should be retrieved only when changing
> focus of VM (and of course at login time ... which that has a default
> focused VM), and all the rest of the refreshes should bring the minimal
> information required to update the VM status.

Continuing that thought,
If someone edits the VM and changes this information (rarely changed), than it looks reasonable to simply change focus or even re-login.

Comment 19 Andrew Cathrow 2013-07-11 11:22:11 UTC
> 
> Continuing that thought,
> If someone edits the VM and changes this information (rarely changed), than
> it looks reasonable to simply change focus or even re-login.

or hit refresh.

ACK agree.

Comment 20 Itamar Heim 2013-07-11 11:43:29 UTC
(In reply to Andrew Cathrow from comment #19)
> > 
> > Continuing that thought,
> > If someone edits the VM and changes this information (rarely changed), than
> > it looks reasonable to simply change focus or even re-login.
> 
> or hit refresh.

that's a distinction between manual and auto refresh (which i'm fine with, just not implied by barak's comment).

barak - everything else on the right would be refreshed by refreshing the vm status probably anyway - it all comes from the same table. i would just avoid asking to refresh the disks on auto-refresh.

question: what happens if a new VM is added to the user? since it is via auto-refresh which isn't populating the disks, it will be without the disks until a manual refresh will be hit or focus change (i'm fine with that - just need to remember while implementing)

Comment 21 Barak 2013-07-11 16:56:03 UTC
(In reply to Itamar Heim from comment #20)
> (In reply to Andrew Cathrow from comment #19)
> > > 
> > > Continuing that thought,
> > > If someone edits the VM and changes this information (rarely changed), than
> > > it looks reasonable to simply change focus or even re-login.
> > 
> > or hit refresh.
> 
> that's a distinction between manual and auto refresh (which i'm fine with,
> just not implied by barak's comment).
> 
> barak - everything else on the right would be refreshed by refreshing the vm
> status probably anyway - it all comes from the same table. i would just
> avoid asking to refresh the disks on auto-refresh.
> 
> question: what happens if a new VM is added to the user? since it is via
> auto-refresh which isn't populating the disks, it will be without the disks
> until a manual refresh will be hit or focus change (i'm fine with that -
> just need to remember while implementing)

Adding a new VM will continue behave the same as today (when a VM is added while user logged in to UP) meaning when this VM will get focus it will retrieve info with Disks all the rest will be without the disk info

Comment 22 Itamar Heim 2013-07-11 17:20:50 UTC
just make sure getting the focus isn't happening for the VM after every refresh, only on the first time it appears (if focus happened to be set to it automatically)

Comment 26 Eli Mesika 2013-08-19 10:49:07 UTC
fixed in commit: f00e810

Comment 28 Barak Dagan 2013-09-22 14:59:20 UTC
following https://bugzilla.redhat.com/show_bug.cgi?id=986700#c3, verified using is15

1. create 10 VMs with 3 disks each.
2. add different users with "view VM" permissions to the VMs
3. opened 7 UserPortals browsers with different users
4. check that all disks appear for all vms
5. from webadmin, remove 2 out of 3 disks (for 3 VMs, one after another)
6, return to the UserPortal and check nothing updated.
7, switch between VM's tabs or wait for 10 seconds - check that disks list is updated.

Comment 29 Charlie 2013-11-28 00:14:49 UTC
This bug is currently attached to errata RHEA-2013:15231. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.

Comment 30 errata-xmlrpc 2014-01-21 17:26:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0038.html