Bug 1067260
| Summary: | [GSS] (6.2.x) Thread leak and OutOfMemoryError on Tomcat using jboss-client.jar, calling an EJB on EAP | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Osamu Nagano <onagano> | ||||||
| Component: | Remoting | Assignee: | David M. Lloyd <david.lloyd> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Jitka Kozana <jkudrnac> | ||||||
| Severity: | unspecified | Docs Contact: | Russell Dickenson <rdickens> | ||||||
| Priority: | unspecified | ||||||||
| Version: | 6.2.0 | CC: | bmaxwell, ksuzumur, rjanik, smumford, vtunka | ||||||
| Target Milestone: | CR2 | ||||||||
| Target Release: | EAP 6.2.2 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: |
Fixes a memory leak that can result from pending connections in org.jboss.remoting3.remote.RemoteConnectionProvider
|
Story Points: | --- | ||||||
| Clone Of: | |||||||||
| : | 1069075 (view as bug list) | Environment: | |||||||
| Last Closed: | 2014-06-02 12:49:40 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1067264, 1068785, 1103785 | ||||||||
| Attachments: |
|
||||||||
We got a heap dump and found that org.jboss.remoting3.remote.RemoteConnectionProvider.pendingInboundConnections is holding over 23 thausands IoFutures. I found the following commit in the upstream, without JIRA, on the same class. [Stop memory leak from pending connections] https://github.com/jboss-remoting/jboss-remoting/commit/7d20e8ebc141cabafd5f13ed92b5703bc5579c39 I send a debug patch of JBoss Remoting 3.2.19.GA, which include the above commit, and corresponding jboss-client.jar. The issue resolved, though not sure about the detailed logic. % git log 3.2.18.GA..3.2.19.GA
commit 7bf6c480d7c640075c4692843767e5c355e0b2c4
Author: David M. Lloyd <david.lloyd>
Date: Fri Dec 20 16:09:58 2013 -0600
Prep 3.2.19.GA
commit 2665edffb42ca3add9222ed90721d4024a6a3584
Author: David M. Lloyd <david.lloyd>
Date: Thu Nov 14 12:14:34 2013 -0600
[REM3-176] Fix CME if some connections are being opened
commit 7d20e8ebc141cabafd5f13ed92b5703bc5579c39
Author: David M. Lloyd <david.lloyd>
Date: Thu Nov 7 21:25:52 2013 -0600
Stop memory leak from pending connections
commit 1fc3e59a31b84d69a3792a99bfd77f72c6f1accf
Author: David M. Lloyd <david.lloyd>
Date: Tue Oct 29 17:11:57 2013 -0500
Fix wrong name for AUTHORIZE_ID simple option
commit cec41089037f187c594cfc1e4501ba795518d4e7
Author: David M. Lloyd <david.lloyd>
Date: Fri Oct 25 12:41:37 2013 -0500
Next is 3.2.19.GA
Created attachment 865755 [details]
remoting 3.2.19 and jboss-client.jar for eap620
This is just a debug patch for EAP 6.2.0 to see the commit actually fixes or not.
I couldn't properly reproduce it in time, but since the issue was resolved I'll set this to verified. If this is an issue that requires a release note (as the requires_doc_text suggests) is it possible to get a few details added to the Doc Text field above to enable ECS to draft a release note? |
Created attachment 865334 [details] client side thread dump from the customer Description of problem: There is a web app on Tomcat calling an EJB on EAP 6.2.0. After a stress test there are hundreds of "ejb-client-context-tasks-NNN-thread-2" threads on Tomcat and results in OOME. ~~~ "ejb-client-context-tasks-463-thread-2" daemon prio=10 tid=0x00007fb800233000 nid=0x646c waiting on condition [0x00007fb7b888a000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000e2a25810> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Locked ownable synchronizers: - None ~~~ Version-Release number of selected component (if applicable): EAP 6.2.0 (JBoss Remoting 3.2.18.GA) EAP 6.1.1 (JBoss Remoting 3.2.16.GA) How reproducible: We tried to reproduce but we couldn't. Though it is always reproducible in the customer's env. The attachement is a thread dump from the customer. There are 37 samples and started to increase from 21st. I made this internal because it includes some specific class names. Additinal Info: We dcided to debug in the customer's env. Will follow up soon about it.