Bug 727270

Summary: RHQ Agent leaks java classes on scheduled operations
Product: [Other] RHQ Project Reporter: Mike Foley <mfoley>
Component: AgentAssignee: John Mazzitelli <mazz>
Status: CLOSED NOTABUG QA Contact: Mike Foley <mfoley>
Severity: medium Docs Contact:
Priority: high    
Version: 4.1CC: hrupp
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-10-31 20:48:03 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 625146    
Attachments:
Description Flags
image showing the java classes that leak as operations execute
none
JProfiler usage after 351 operations none

Description Mike Foley 2011-08-01 17:40:33 UTC
Created attachment 516185 [details]
image showing the java classes that leak as operations execute

Description of problem:  RHQ Agent leaks java classes on scheduled operations.  Unbounded increase in class instance counts as scheduled operations execute on the RHQ Agent resource.  Can lead to robustness, duration, longevity issues.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.   jprofiler attached to agent
2.   create a scheduled operation on the RHQ Agent, to execute an availability scan every 60 seconds for an unlimited amount of time
3.   observe the scheduled operation execute a few times
4.   in jprofiler ... perform GC ...mark baseline
5.   observe instance counts increase as operation executes ... even after performing GC ... in an unbounded manner.
  
Actual results:
unbounded increase in class instance counts

Expected results:
no unbounded increases 

Additional info:

Comment 1 Charles Crouch 2011-08-02 02:23:41 UTC
Assigning to mazz, to investigate further. This seems worse than 727245 since scheduled operations should be a fairly common thing.
Severity is going to depend on rate of leak, and whether its related to the operation being called for example, e.g. are some operations really bad.

Comment 2 John Mazzitelli 2011-08-03 17:59:36 UTC
I'm not convinced this is anything. Would need someone to perform true, long term testing to see if memory actually does bloat over time. (BTW: I assume this BZ is reporting bloating PERMGEN memory).

Read bug 615377 - I spent a good week on tracking down and fixing as many memory leaks as I could - some could not be fixed that I could see. Note that JDK7 has introduced a ClassLoader.close() API that could help us. As it stands, there is a design flaw in the JRE6's ClassLoader impl that means we may never correct all memory leaks until we go to JDK7

Also, see if these JDK settings help:

https://bugzilla.redhat.com/show_bug.cgi?id=615377#c9

-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled

Comment 5 John Mazzitelli 2011-10-31 20:43:16 UTC
Created attachment 531037 [details]
JProfiler usage after 351 operations

I'm not seeing any leak. I ran a simple operation ("Get Current Date/Time") on the RHQ Agent resource. I scheduled it to run every 4 seconds. I ended up running it 351 times is rapid succession. After waiting for GC to run, and taking snapshots before and after the test, I do not see any memory leakage. In fact, I see the memory with the typical "sawtooth" usage behavior - heap usage goes up, then GC kicks in and frees the used memory, back to roughly the starting point of the 20MB mark. At the end, I was still using 20MB as per the attached image from jprofiler.

Comment 6 John Mazzitelli 2011-10-31 20:48:03 UTC
There is no leak detected. After examining the heap memory usage after rapidly executing an operation every 4 seconds for 25 minutes straight, memory consumption looks normal. Nothing appears to leak unbounded. Garbage collection reclaims heap periodically back down to its original level.