Bug 580113

Summary:	Performance regression in UI - all pages taking 3-5x longer to load, 2x CPU use
Product:	[Other] RHQ Project	Reporter:	Jeff Weiss <jweiss>
Component:	Core UI	Assignee:	Heiko W. Rupp <hrupp>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jeff Weiss <jweiss>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	1.4	CC:	dajohnso
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	2.4	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2010-08-12 16:59:20 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	543122, 565628

Description Jeff Weiss 2010-04-07 14:08:13 UTC

Description of problem:
In recent builds (within the past 2 weeks or so - the build I'm currently using is d82d1581f18dcd6b9fb2a4e189df52ef82e43ece ) pages all load very slowly, coupled with very high CPU utilization.  Normally with a 2 processor system, CPU is normally around 60-80% for the RHQ server process when automation is running (which clicks links in quick succession, keeping pages loading continuously).  In recent builds, the cpu use is up to 150%. 

Version-Release number of selected component (if applicable):
d82d1581f18dcd6b9fb2a4e189df52ef82e43ece

How reproducible:
Always

Steps to Reproduce:
1. Install and start server
2. Login and view Dashboard page

Actual results:
Page loads extremely slow, can see each portlet appearing on the page one by one.  All pages are equally affected.
  

Expected results:
Page loads should be faster, how fast of course is subjective, but we should all have a good feel for how the app behaved in the past on our test/dev boxes.

Additional info:
jshaughn and jsefler have seen the same problem.

Comment 1 Jeff Weiss 2010-04-08 14:57:33 UTC

I just tested on rhq3.0.0.B04, and the same problem is present in that build also.  So it appears to have been introduced before then.

Comment 2 Jeff Weiss 2010-04-08 17:08:57 UTC

More testing reveals the two bookends for a git bisect:

Latest known good build that doesn't contain this bug - rhq3.0.0.B02
Earliest known bad build that contains this bug - rhq3.0.0.B03

Comment 3 Jeff Weiss 2010-04-08 18:40:04 UTC

Here's an interesting wrinkle.  Corey's recent JON install does not exhibit the problem.  So I installed JON build 10593 (built Apr 7, 6pm EDT), on my test machine at it also did NOT exhibit the problem.  Apparently RHQ is affected (including recent builds from hudson, dating all the way back to beta builds as far back as Feb 22).

Comment 4 Jeff Weiss 2010-04-14 15:39:47 UTC

java environment:
[rhq@auto-rhq01 bin]$ java -version
java version "1.6.0"
OpenJDK  Runtime Environment (build 1.6.0-b09)
OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode)

Comment 5 Heiko W. Rupp 2010-04-16 12:10:25 UTC

GetCoreGui is taking a long time here - on average 1.5 s , as it gets called 310 times for each click (for me). This number does not change with the number of resources in inventory.

Comment 6 Heiko W. Rupp 2010-04-16 13:11:42 UTC

Disabling menu generation in /rhq/common/menu/menu.xhtml speeds up dashboard display from 3.5s+ to 0.5s

Comment 7 Jeff Weiss 2010-04-20 20:04:29 UTC

I believe joseph fixed this with commit df54d527d9d995ae21a2b907d019500dc059a39e .

Performance seems to be back to normal, I'll reopen this if the problem comes back.

Comment 8 Corey Welton 2010-08-12 16:59:20 UTC

Mass-closure of verified bugs against JON.