Bug 1803856 - [Scale] ovirt-vmconsole takes too long or times out in a 500+ VM environment.
Summary: [Scale] ovirt-vmconsole takes too long or times out in a 500+ VM environment.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.3.7
Hardware: All
OS: Linux
high
high
Target Milestone: ovirt-4.4.2
: 4.4.2
Assignee: Arik
QA Contact: Tzahi Ashkenazi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-17 15:42 UTC by Federico Sun
Modified: 2023-12-15 17:21 UTC (History)
6 users (show)

Fixed In Version: ovirt-engine-4.4.2.1
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-23 16:11:04 UTC
oVirt Team: Virt
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-37450 0 None None None 2022-07-09 12:54:45 UTC
Red Hat Product Errata RHSA-2020:3807 0 None None None 2020-09-23 16:11:24 UTC
oVirt gerrit 110157 0 master MERGED vmconsole: query vm_dynamic rather than vms 2020-09-14 21:22:48 UTC
oVirt gerrit 110158 0 master MERGED vmconsole: filter running vms in database query 2020-09-14 21:22:48 UTC
oVirt gerrit 110159 0 master MERGED vmconsole: query vds_static rather than vds 2020-09-14 21:22:48 UTC
oVirt gerrit 110160 0 master MERGED vmconsole: get hosts with bulk query 2020-09-14 21:22:48 UTC
oVirt gerrit 110161 0 master MERGED vmconsole: cleanup 2020-09-14 21:22:48 UTC

Description Federico Sun 2020-02-17 15:42:42 UTC 
Description of problem:

In a big 500+ VMs environment when attempt to use virt-vmconsle to connect to a VM's virtual serial console (using --vm-name) it either succeeds after a long time or it times out.

The bottleneck seems to be from this stored procedure: getallfromvmsforuserandactiongroup()

Version-Release number of selected component (if applicable):

rhvm-4.3.7.2-0.1.el7.noarch
ovirt-vmconsole-1.0.7-3.el7ev.noarch


Steps to Reproduce:
1. Create a big RHV environment with 500+ VMs
2. Connect to a VM's virtual serial console.


Actual results:
Either times out or connects after 30mins.

Expected results:

Should connect fast.
Comment 2 Francesco Romani 2020-02-17 16:08:16 UTC 
I cannot work on this bug in the near/mid term
Comment 6 mlehrer 2020-06-16 09:25:04 UTC 
Spoke with Arik we'll provide him some time on this size environment for 4.4.2 when possible.
Comment 7 Arik 2020-06-30 16:57:08 UTC 
version: 1.0.7
Comment 8 mlehrer 2020-07-30 11:59:00 UTC 
Please supply validation steps and required env details if / where relevant.
Comment 9 Arik 2020-08-02 16:18:38 UTC 
The validation steps are as Federico wrote above:
1. Run 500+ VMs
2. Connect to a VM's virtual serial console.

The load on the database should reduce (compared to the 4.4.1/4.3)
Comment 13 Tzahi Ashkenazi 2020-08-20 11:10:54 UTC 
Tested and verify on Red-01 
on Cluster >  L0_Group_1 with 750 VMs 

verify the BZ is reproducible on version :
   rhv-release-4.4.1-12-001.noarch
   vdsm-4.40.22-1.el8ev.x86_64

The API call that was used  during the vmconsole command ( slow traces )  :
/ovirt-engine/services/vmconsole-proxy

the API call took :
        37,957.0 ms 	


tested again on the same environment with version :
   rhv-release-4.4.2-3-001.noarch
   vdsm-4.40.25-1.el8ev.x86_64

The API call that was used  during the vmconsole command :
/ovirt-engine/services/vmconsole-proxy

the API call took :
        1,605.7 ms 

from the end-user the connect using vm-console takes around 3-5 sec on version > 4.4.2-3
Comment 17 errata-xmlrpc 2020-09-23 16:11:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Virtualization security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3807
Note You need to log in before you can comment on or make changes to this bug.