Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1839660

Summary: [UI] Cluster edit , takes too long even without changes while submitting OK
Product: [oVirt] ovirt-engine Reporter: Tzahi Ashkenazi <tashkena>
Component: GeneralAssignee: Arik <ahadas>
Status: CLOSED CURRENTRELEASE QA Contact: Tzahi Ashkenazi <tashkena>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.4.1CC: ahadas, bugs, dagur, dholler, michal.skrivanek, mlehrer, mperina
Target Milestone: ovirt-4.4.5Keywords: Performance
Target Release: ---Flags: pm-rhel: ovirt-4.5?
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-21 16:46:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine and vdsm logs
none
Trace Report showing query break down upon Edit Cluster via UI none

Description Tzahi Ashkenazi 2020-05-25 08:17:56 UTC
Created attachment 1691738 [details]
engine and vdsm logs

Description of problem:

while pressing the cluster edit option  on cluster > L0_Group_0 with  1032 VMs
Cluster edit takes too long even without commit any changes while pressing the OK button 

Version-Release number of selected component (if applicable):

rhv-release-4.4.0-32-001.noarch
Red Hat Enterprise Linux release 8.2 (Ootpa)
vdsm-4.40.14-1.el8ev.x86_64

How reproducible:

On rhev-red-03.rdu2.scalelab.redhat.com
with 2 clusters, one with 2 VMs ( DWH & hosted-engine) using 2 hosts
and one cluster  with 4 hosts for VMs with 1032 VMs


Steps to Reproduce:

1. on the cluster view L0_Group_0 press edit option 
2. dont do any changes and press OK for confirmation 
3. operation takes ~one minute  or more 

Actual results:
operation  takes more than one minute with or without commit any changes on the cluster edit view 

Expected results:
a few seconds 

Additional info:

Comment 1 mlehrer 2020-05-25 11:49:34 UTC
Created attachment 1691920 [details]
Trace Report showing query break down upon Edit Cluster via UI

Comment 2 mlehrer 2020-05-25 11:56:58 UTC

No resource saturation occurs on Engine during this 'slow edit cluster' activity.
Open Edit Clusters, then press OK  (nothing is changed) total time: 117s
The cause of the long duration is /ovirt-engine/webadmin/GenericApiGWTService 

Of the total 117s, 57s are spent on 41,618 queries.


Breakdown:	total (ms)	count
http request   117,775.4	  1
jdbc query      57,447.5	41,618
jdbc commit      3,156.2	2,025
wait on future   1,719.7	187
jdbc get conn      655.9	40,033
logging            271.1	2,655

The following queries are executed more than 1000 times each

select * from getquotabyquotaguid(?)
select * from getdisksvmguid(?, ?, ?, ?)
select * from getvmbyvmguid(?, ?, ?)
select * from getvmdevicebyvmidtypeanddevice(?, ?, ?, ?, ?)
select * from getclusterbyclusterid(?, ?, ?)
select * from getvmnetworkinterfaceviewbyvmid(?, ?, ?)
SELECT NULL AS PROCEDURE_CAT, n.nspname AS PROCEDURE_SCHEM, p.proname A ... balloonenabled' ORDER BY PROCEDURE_SCHEM, PROCEDURE_NAME, p.oid::text
select * from getvdsbyvdsid(?, ?, ?)
select * from getvmdevicebyvmidandtype(?, ?)
select * from getvmdevicebyvmid(?, ?, ?)
SELECT n.nspname,p.proname,p.prorettype,p.proargtypes, t.typtype,t.typr ... LIKE 'ismemballoonenabled' ORDER BY n.nspname, p.proname, p.oid::text
select * from getvmtemplatebyvmtguid(?, ?, ?)
select * from getsnapshotbyvmidandtype(?, ?, ?, ?)
{call updatevmstatic(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ...  ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}
select * from getdiskvmelementsforvm(?, ?, ?)
select * from getstorage_poolbyid(?, ?, ?)
{call insert_entity_snapshot(?, ?, ?, ?, ?, ?, ?, ?, ?)}
{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}
select * from isvmiconexist(?)
{call incrementdbgeneration(?)}
{? = call ismemballoonenabled(?)}
select * from getquotaclusterbyquotaguid(?)
select * from getsnapshotsbyvmsnapshotid(?)
select * from getquotastoragebyquotaguid(?)
select * from getvminitbyvmid(?)
select * from getnumanodebyvmid(?)
select * from getvminterfacesbyvmid(?)
select * from getallassignednumanodeinfomation()


Please see attached: Cluster_Edit_GWT_trace.zip and open the resulting html file to see the full list of queries, and number of executions, and rows returned by query.

This issue maybe related to: https://bugzilla.redhat.com/show_bug.cgi?id=1811866  which has to do with many networks and Cluster UI view.

Comment 3 Michal Skrivanek 2020-05-25 14:40:01 UTC
clicking OK does update all the VMs so naturally there will be 1000+ of each query
doesn't sound too interesting since it still works fine. 2 mins is not too bad

Comment 5 Dominik Holler 2020-05-27 09:38:09 UTC
(In reply to Michal Skrivanek from comment #3)
> clicking OK does update all the VMs so naturally there will be 1000+ of each
> query

Is it required to update all VMs, even nothing is changed?

Comment 6 Tzahi Ashkenazi 2020-08-25 12:52:26 UTC
Tested again on Red-01  in order to reproduce.

version 4.4.2 :
       rhv-release-4.4.2-3-001.noarch
       vdsm-4.40.25-1.el8ev.x86_64
       L0_Group_0 with 1350 VMs 

didnt reproduce..

Comment 7 Arik 2021-03-21 16:46:46 UTC
Yeah, triggering update-cluster with no change doesn't trigger update VMs/templates anymore

Comment 8 Sandro Bonazzola 2021-03-22 12:55:44 UTC
This bugzilla is included in oVirt 4.4.5 release, published on March 18th 2021.

Since the problem described in this bug report should be resolved in oVirt 4.4.5 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.