Bug 1803856

Summary:	[Scale] ovirt-vmconsole takes too long or times out in a 500+ VM environment.
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Federico Sun <fsun>
Component:	ovirt-engine	Assignee:	Arik <ahadas>
Status:	CLOSED ERRATA	QA Contact:	Tzahi Ashkenazi <tashkena>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.3.7	CC:	ahadas, bugs, dagur, mlehrer, mtessun, pelauter
Target Milestone:	ovirt-4.4.2	Keywords:	Performance, ZStream
Target Release:	4.4.2
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	ovirt-engine-4.4.2.1	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-09-23 16:11:04 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Virt	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Federico Sun 2020-02-17 15:42:42 UTC

Description of problem:

In a big 500+ VMs environment when attempt to use virt-vmconsle to connect to a VM's virtual serial console (using --vm-name) it either succeeds after a long time or it times out.

The bottleneck seems to be from this stored procedure: getallfromvmsforuserandactiongroup()

Version-Release number of selected component (if applicable):

rhvm-4.3.7.2-0.1.el7.noarch
ovirt-vmconsole-1.0.7-3.el7ev.noarch


Steps to Reproduce:
1. Create a big RHV environment with 500+ VMs
2. Connect to a VM's virtual serial console.


Actual results:
Either times out or connects after 30mins.

Expected results:

Should connect fast.

Comment 2 Francesco Romani 2020-02-17 16:08:16 UTC

I cannot work on this bug in the near/mid term

Comment 6 mlehrer 2020-06-16 09:25:04 UTC

Spoke with Arik we'll provide him some time on this size environment for 4.4.2 when possible.

Comment 7 Arik 2020-06-30 16:57:08 UTC

version: 1.0.7

Comment 8 mlehrer 2020-07-30 11:59:00 UTC

Please supply validation steps and required env details if / where relevant.

Comment 9 Arik 2020-08-02 16:18:38 UTC

The validation steps are as Federico wrote above:
1. Run 500+ VMs
2. Connect to a VM's virtual serial console.

The load on the database should reduce (compared to the 4.4.1/4.3)

Comment 13 Tzahi Ashkenazi 2020-08-20 11:10:54 UTC

Tested and verify on Red-01 
on Cluster >  L0_Group_1 with 750 VMs 

verify the BZ is reproducible on version :
   rhv-release-4.4.1-12-001.noarch
   vdsm-4.40.22-1.el8ev.x86_64

The API call that was used  during the vmconsole command ( slow traces )  :
/ovirt-engine/services/vmconsole-proxy

the API call took :
        37,957.0 ms 	


tested again on the same environment with version :
   rhv-release-4.4.2-3-001.noarch
   vdsm-4.40.25-1.el8ev.x86_64

The API call that was used  during the vmconsole command :
/ovirt-engine/services/vmconsole-proxy

the API call took :
        1,605.7 ms 

from the end-user the connect using vm-console takes around 3-5 sec on version > 4.4.2-3

Comment 17 errata-xmlrpc 2020-09-23 16:11:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Virtualization security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3807