Bug 965093 - Thread leak test failure with IBM JRE
Thread leak test failure with IBM JRE
Status: VERIFIED
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: HornetQ (Show other bugs)
6.1.0
All Linux
medium Severity medium
: DR11
: EAP 6.4.0
Assigned To: Martin Svehla
Martin Svehla
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-20 07:18 EDT by Yong Hao Gao
Modified: 2014-12-01 02:53 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker HORNETQ-1206 Major Closed Thread leak test failure with IBM JRE 2017-02-13 06:43 EST

  None (edit)
Description Yong Hao Gao 2013-05-20 07:18:19 EDT
Description of problem:

In HornetQ test suite it checks thread leak after each test. It doesn't this by checking all the running threads at the end of the test and if any of the thread is not in the expected set, the test will fail. 

The expected set if a set of thread names that can be ignored when checking thread leak. With IBM jre, the hornetq server will cause a daemon thread called "MemoryPoolMXBean notification dispatcher" to be started by the VM because the hornetq server invoked the management service. This thread is not controlled by hornetq and should be put to the expected set. Otherwise tests will report false thread leak error.

Version-Release number of selected component (if applicable):

HornetQ 2.3.2

How reproducible:

Setup the test env using IBM jdk and run the tests, it'll get thread leaking error.

Steps to Reproduce:
1. Setup evn using IBM jdk (version 7 preferred)
2. Run any hornetq test that checks thread leak at teardown.
3. Observe the test failure.

Actual results:

org.hornetq.tests.integration.cluster.distribution.ClusterHeadersRemovedTest::#test testHeadersRemoved
org.hornetq.tests.integration.cluster.distribution.ClusterHeadersRemovedTest::Thread leaked on test org.hornetq.tests.integration.cluster.distribution.ClusterHeadersRemovedTest::testHeadersRemoved
*********************************************************************************
LEAKING THREADS
=============================================================================
Thread Thread[MemoryPoolMXBean notification dispatcher,6,main] is still alive with the following stackTrace:
com.ibm.lang.management.MemoryNotificationThread.processNotificationLoop(Native Method)
com.ibm.lang.management.MemoryNotificationThread.run(MemoryNotificationThread.java:54)
*********************************************************************************

org.hornetq.tests.integration.cluster.distribution.ClusterHeadersRemovedTest::Thread leakage

Expected results:

The test should pass.

Additional info:
Comment 1 JBoss JIRA Server 2013-05-20 13:14:24 EDT
Clebert Suconic <clebert.suconic@jboss.com> updated the status of jira HORNETQ-1206 to Closed
Comment 3 Martin Svehla 2013-12-06 08:27:06 EST
I'm still seeing following leaking thread errors with IBM java:

*********************************************************************************
LEAKING THREADS
=============================================================================
Thread Thread[process reaper,10,main] is still alive with the following stackTrace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:237)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:471)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:370)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:953)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1079)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1141)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:626)
java.lang.Thread.run(Thread.java:780)
*********************************************************************************

Tested it with HornetQ_2_3_12_Final tag and 64bit IBM java 7. I'm getting it for example with tests in org.hornetq.tests.integration.cluster.RealNodeManagerTest

Seems the issue is that the UnitTestCase ignores "process reaper" thread only if it's in "system" thread group (UnitTestCase.java::1222). However it's in "main" group on IBM java (as well as on OpenJDK7 btw, although tests don't leak threads on OpenJDK). I'm not sure why there's the test on it being system thread.
Comment 4 Clebert Suconic 2013-12-06 11:08:29 EST
Could you patch it and test this? [1]


If that fixes your issue I could even retag 2.3.12 since it only affect testcases (no production code changed).


[1]:
diff --git a/hornetq-server/src/test/java/org/hornetq/tests/util/UnitTestCase.java b/hornetq-server/src/test/java/org/hornetq/tests/util/UnitTestCase.java
index 33a0af8..4abb51a 100644
--- a/hornetq-server/src/test/java/org/hornetq/tests/util/UnitTestCase.java
+++ b/hornetq-server/src/test/java/org/hornetq/tests/util/UnitTestCase.java
@@ -1224,6 +1224,7 @@ public abstract class UnitTestCase extends CoreUnitTestCase
       final String name = thread.getName();
       final ThreadGroup group = thread.getThreadGroup();
       final boolean isSystemThread = group != null && "system".equals(group.getName());
+      final String javaVendor = System.getProperty("java.vendor");
 
       if (name.contains("SunPKCS11"))
       {
@@ -1237,6 +1238,10 @@ public abstract class UnitTestCase extends CoreUnitTestCase
       {
          return true;
       }
+      else if (javaVendor.contains("IBM") && name.equals("MemoryPoolMXBean notification dispatcher"))
+      {
+         return true;
+      }
       else
       {
          for (StackTraceElement element : thread.getStackTrace())
Clebert-mac:hornetq clebertsuconic$
Comment 5 Clebert Suconic 2013-12-06 11:09:56 EST
I'm assigning this to you.. let me know if you can't make the test and I will assume it back.
Comment 6 Martin Svehla 2013-12-09 03:01:34 EST
Clebert,

Are you sure you pasted correct patch? This change is already present in 2.3.12.

I can fix the issue by applying this patch, but I'm not sure _why_ was there the test for process reaper thread being in the system thread group in the first place.


diff --git a/hornetq-server/src/test/java/org/hornetq/tests/util/UnitTestCase.java b/hornetq-server/src/test/java/org/
index d63fe6c..075eb7e 100644
--- a/hornetq-server/src/test/java/org/hornetq/tests/util/UnitTestCase.java
+++ b/hornetq-server/src/test/java/org/hornetq/tests/util/UnitTestCase.java
@@ -1208,7 +1208,6 @@ public abstract class UnitTestCase extends CoreUnitTestCase
    {
       final String threadName = thread.getName();
       final ThreadGroup group = thread.getThreadGroup();
-      final boolean isSystemThread = group != null && "system".equals(group.getName());
       final String javaVendor = System.getProperty("java.vendor");
 
       if (threadName.contains("SunPKCS11"))
@@ -1219,7 +1218,7 @@ public abstract class UnitTestCase extends CoreUnitTestCase
       {
          return true;
       }
-      else if (isSystemThread && threadName.equals("process reaper"))
+      else if (threadName.equals("process reaper"))
       {
          return true;
       }
Comment 7 Martin Svehla 2013-12-09 07:12:50 EST
Noticed you're not on CC list Clebert. Check previous comment please
Comment 8 Justin Bertram 2014-07-21 15:22:19 EDT
Clebert, I looked at the original commit for the code that Martin mentioned.  I wasn't able to determine why the "process reaper" thread is expected in the "system" thread group.  Can you shed any light on this?  If it's not necessary we can remove that particular check and close this issue.
Comment 9 Clebert Suconic 2014-07-21 15:55:20 EDT
We can probably ignore that check on IBM JDK.

We just need to run with the IBM JDK and see where it hangs. There could be other thread checks being done that we need to ignore on the thread leakage detection on teardown
Comment 10 Justin Bertram 2014-08-28 11:27:10 EDT
Fix was committed to the 2.3.x branch via fb5e77a6ccd3046e9bc6b49f408c99ae48d584e0.
Comment 11 Martin Svehla 2014-09-26 09:38:16 EDT
While the fix is in the 2.3.x branch (and we can ofc use that with latest EAP build), please move this to ON_QA only once the fix is in the HornetQ tag that's been used to build EAP. Thanks

Moving it back to modified until there's new HornetQ tag (2.3.22+) in EAP.
Comment 12 Martin Svehla 2014-12-01 02:53:57 EST
Verified with EAP 6.4.0.DR11 / HornetQ 2.3.24.Final. Thanks!

Note You need to log in before you can comment on or make changes to this bug.