Bug 727270 - RHQ Agent leaks java classes on scheduled operations
Summary: RHQ Agent leaks java classes on scheduled operations
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: RHQ Project
Classification: Other
Component: Agent
Version: 4.1
Hardware: All
OS: All
high
medium
Target Milestone: ---
: ---
Assignee: John Mazzitelli
QA Contact: Mike Foley
URL:
Whiteboard:
Depends On:
Blocks: jon30-bugs
TreeView+ depends on / blocked
 
Reported: 2011-08-01 17:40 UTC by Mike Foley
Modified: 2011-10-31 20:48 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-10-31 20:48:03 UTC
Embargoed:


Attachments (Terms of Use)
image showing the java classes that leak as operations execute (112.73 KB, image/png)
2011-08-01 17:40 UTC, Mike Foley
no flags Details
JProfiler usage after 351 operations (14.19 KB, image/png)
2011-10-31 20:43 UTC, John Mazzitelli
no flags Details

Description Mike Foley 2011-08-01 17:40:33 UTC
Created attachment 516185 [details]
image showing the java classes that leak as operations execute

Description of problem:  RHQ Agent leaks java classes on scheduled operations.  Unbounded increase in class instance counts as scheduled operations execute on the RHQ Agent resource.  Can lead to robustness, duration, longevity issues.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.   jprofiler attached to agent
2.   create a scheduled operation on the RHQ Agent, to execute an availability scan every 60 seconds for an unlimited amount of time
3.   observe the scheduled operation execute a few times
4.   in jprofiler ... perform GC ...mark baseline
5.   observe instance counts increase as operation executes ... even after performing GC ... in an unbounded manner.
  
Actual results:
unbounded increase in class instance counts

Expected results:
no unbounded increases 

Additional info:

Comment 1 Charles Crouch 2011-08-02 02:23:41 UTC
Assigning to mazz, to investigate further. This seems worse than 727245 since scheduled operations should be a fairly common thing.
Severity is going to depend on rate of leak, and whether its related to the operation being called for example, e.g. are some operations really bad.

Comment 2 John Mazzitelli 2011-08-03 17:59:36 UTC
I'm not convinced this is anything. Would need someone to perform true, long term testing to see if memory actually does bloat over time. (BTW: I assume this BZ is reporting bloating PERMGEN memory).

Read bug 615377 - I spent a good week on tracking down and fixing as many memory leaks as I could - some could not be fixed that I could see. Note that JDK7 has introduced a ClassLoader.close() API that could help us. As it stands, there is a design flaw in the JRE6's ClassLoader impl that means we may never correct all memory leaks until we go to JDK7

Also, see if these JDK settings help:

https://bugzilla.redhat.com/show_bug.cgi?id=615377#c9

-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled

Comment 5 John Mazzitelli 2011-10-31 20:43:16 UTC
Created attachment 531037 [details]
JProfiler usage after 351 operations

I'm not seeing any leak. I ran a simple operation ("Get Current Date/Time") on the RHQ Agent resource. I scheduled it to run every 4 seconds. I ended up running it 351 times is rapid succession. After waiting for GC to run, and taking snapshots before and after the test, I do not see any memory leakage. In fact, I see the memory with the typical "sawtooth" usage behavior - heap usage goes up, then GC kicks in and frees the used memory, back to roughly the starting point of the 20MB mark. At the end, I was still using 20MB as per the attached image from jprofiler.

Comment 6 John Mazzitelli 2011-10-31 20:48:03 UTC
There is no leak detected. After examining the heap memory usage after rapidly executing an operation every 4 seconds for 25 minutes straight, memory consumption looks normal. Nothing appears to leak unbounded. Garbage collection reclaims heap periodically back down to its original level.


Note You need to log in before you can comment on or make changes to this bug.