Bug 1175504

Summary: Navigating to virtual machines explorer is slow on Large Environment (15k Total VMs, 11k Archived VMs)
Product: Red Hat CloudForms Management Engine Reporter: Alex Krzos <akrzos>
Component: PerformanceAssignee: Keenan Brock <kbrock>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Dave Johnson <dajohnso>
Severity: high Docs Contact:
Priority: medium    
Version: 5.3.0CC: apatters, cpelland, jhardy, jprause, obarenbo, perfbz, xlecauch
Target Milestone: GA   
Target Release: 5.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: ui:perf
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1290209 (view as bug list) Environment:
Last Closed: 2016-02-18 16:01:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1290209    
Attachments:
Description Flags
Production logs for navigating to vm_infra explorer and expanding archived and orphaned vms none

Description Alex Krzos 2014-12-17 21:33:20 UTC 
Created attachment 970320 [details]
Production logs for navigating to vm_infra explorer and expanding archived and orphaned vms

Description of problem:
On a Global VMDB appliance which is receiving replicated records from 4 other environments now has over 15,000 VM Records mostly due to archived VMs the rendering time for vm_infra/explorer is too slow.

A large amount of time is spent rendering the page 

Version-Release number of selected component (if applicable):
5.3.1.10

How reproducible:
Always with this environment/database

Steps to Reproduce:
1. Observe page rendering time while navigating to Infrastructure->Virtual Machines
2. Can duplicate similar issue by expanding archived instances on the VM & Templates Accordion. 
3.

Actual results:
INFO -- : Completed 200 OK in 87072.7ms (Views: 259.9ms | ActiveRecord: 903.4ms)

>80s page rendering time with less than 1s in active record

Expected results:
<5s page rendering time


Additional info:
$ grep "select" -i production.* -c
production.expand_archived.log:30
production.expand_orphaned.log:1618
production.vm_explorer.log:1881

$ grep "cache" -i production.* -c
production.expand_archived.log:18
production.expand_orphaned.log:1594
production.vm_explorer.log:1712

When expanding the archived VMs, a large portion of time is spent immediately after this query:
CACHE (0.0ms)  SELECT COUNT(*) FROM "vms" WHERE ((vms.id IN (1000000002911,1000000000580, .....

N+1 queries when expanding orphaned vms portion of tree:
$ grep 'SELECT "storages".* FROM "storages" WHERE "storages"."id" =' production.expand_orphaned.log  -c
1588
Comment 2 Alex Krzos 2015-05-21 12:54:15 UTC 
In my CFME scale environment with two 10k VM VMware providers attached, 20k vms total in the regional vmdb + some archived VMs (~200), I am seeing apache timeouts due to the rendering time of vm_infra page exceeding 2 minutes.

[----] I, [2015-05-21T12:30:19.560097 #10611:552080]  INFO -- : Completed 200 OK in 148212.3ms (Views: 625.1ms | ActiveRecord: 2951.2ms)
Comment 3 Dave Johnson 2016-02-18 16:01:17 UTC 
This BZ is now over a year old, because of this, we are closing it.  If you feel like this is a mistake and it is still a issue, please reproduce it in the latest release and create a new BZ.