Bug 871610

Summary:	JMX Plugin - Leaking connector threads - com.sun.jmx.remote.internal.ClientCommunicatorAdminClientCommunicatorAdmin$Checker.run
Product:	[JBoss] JBoss Operations Network	Reporter:	Charles Crouch <ccrouch>
Component:	Plugin -- Other	Assignee:	John Mazzitelli <mazz>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Mike Foley <mfoley>
Severity:	high	Docs Contact:
Priority:	high
Version:	JON 3.1.1	CC:	genman, hbrock, hrupp, mazz, spinder
Target Milestone:	---
Target Release:	JON 3.1.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	848938	Environment:
Last Closed:	2013-09-11 10:59:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	848938
Bug Blocks:

Description Charles Crouch 2012-10-30 20:54:20 UTC

+++ This bug was initially created as a clone of Bug #848938 +++

Description of problem:

Number of process threads increases over time.

One such example:

"Thread-2461" daemon prio=10 tid=0x000000006103c800 nid=0x5fc0 waiting on condition [0x00002aab178bb000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154)
        at java.lang.Thread.run(Thread.java:619)



Version-Release number of selected component (if applicable):

RHQ 4.4 version of the plugin running.


How reproducible:

Unclear what specific conditions cause this in my case, but I have seen it almost universally. It could be to discovery failing.

There are obvious code paths which can likely lead to this condition worth exploring.


Additional info:

See: http://stackoverflow.com/questions/564589/dynamic-proxies-with-jmx-can-cause-thread-leaks
> When you do:
> JMXConnectorFactory.connect(url)
> Make sure to close it when you're done using it rather than leaving it up to garbage collection or it seems they can keep piling up...


There are two instances where close is never called:
1) Getting Java version
2) Getting Java PID

** patch forthcoming

--- Additional comment from genman on 2012-09-13 18:01:15 EDT ---

Created attachment 612647 [details]
fixes connection leak patch

--- Additional comment from mazz on 2012-09-19 06:55:33 EDT ---

Tried applying the patch - it failed on current master (which is 6887631b) - please correct and re-attach to this BZ. Here's the error I get:

$ git am ~/Downloads/0001-Bug-848938-JMX-Plugin-Leaking-connector-thread.patch
Applying: Bug 848938 - JMX Plugin - Leaking connector thread
error: patch failed: modules/plugins/jmx/src/main/java/org/rhq/plugins/jmx/JMXDiscoveryComponent.java:335
error: modules/plugins/jmx/src/main/java/org/rhq/plugins/jmx/JMXDiscoveryComponent.java: patch does not apply
Patch failed at 0001 Bug 848938 - JMX Plugin - Leaking connector thread

--- Additional comment from genman on 2012-09-19 12:01:58 EDT ---

Created attachment 614447 [details]
updated patch against master - 9f175839392

--- Additional comment from mazz on 2012-09-19 12:30:42 EDT ---

git commit to master: 26b37482f90fb01a6771c5c51471e2754dc3cdb4

--- Additional comment from ccrouch on 2012-10-30 16:51:33 EDT ---

Set Target Release correctly.

Comment 1 Charles Crouch 2012-10-30 21:27:00 UTC

Waiting to be cherry-picked to the release/jon31x branch

Comment 2 John Mazzitelli 2012-10-31 18:10:29 UTC

cherry picked to release/jon3.1.x branch: 7aa947262aba8dd3bbebeea9243165b659e11e5e

had to correct minor conflicts - no code conflicts, just comments

Comment 3 Simeon Pinder 2012-11-16 18:33:07 UTC

Moving this to ON_QA as available for testing in https://brewweb.devel.redhat.com//buildinfo?buildID=243389.

Comment 4 Mike Foley 2012-11-16 18:58:38 UTC

no repro steps 
unclear what causes this 

marking verified based on developer code review ... low risk.  adding close() in several spots.

plugins will be regression tested as part of JON 3.1.2.