Bug 913507
| Summary: | OutOfMemory in SpecJMS2007 satellite driver | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Miroslav Novak <mnovak> | ||||
| Component: | HornetQ | Assignee: | Clebert Suconic <clebert.suconic> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 6.1.0 | CC: | atangrin, csuconic, nmaurer, pslavice, rdickens, rsvoboda | ||||
| Target Milestone: | ER6 | Keywords: | TestBlocker | ||||
| Target Release: | EAP 6.1.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
When NIO was enabled on a connector and a very high number of threads was active, an *OutOfMemory* error could occur. This occurred because NioWorker was not shutdown correctly and so the threads and the memory they were using was not released. The underlying problem with thread handling has been corrected, reducing the risk of an *OutOfMemory* error.
It is recommended that the parameter "use-nio-global-worker-pool" be enabled to minimize the risk of an OutOfMemory error. An example connector configuration is as follows:
[source,XML]
----
<netty-connector name="netty" socket-binding="messaging">
<param key="use-nio" value="true"/>
<param key="use-nio-global-worker-pool" value="true"/>
</netty-connector>
----
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | Type: | Bug | |||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Miroslav Novak
2013-02-21 11:50:18 UTC
Created attachment 700810 [details]
got threadump, heapdump and histogram.
Threadump, heapdump and histogram attached. The same result with JVM_OPTIONS="-Xms15G -Xmx15G -Xss1048K" This seems to be a thread leak. Original OutOfMemoryError exception was caused by limited value of max user processes (ulimit -u): max user processes (-u) 8192 This value was sufficient for EAP 6.0.1 with 3x time higher load. There are thousands of threads like: Stack trace: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.Util$2@5644c8ca - locked java.util.Collections$UnmodifiableSet@3dda7205 - locked sun.nio.ch.EPollSelectorImpl@5073c5fc sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Sounds for me like the NioWorker is not shutdown correctly and so the threads are not released. Will check with clebert currently try to reproduce with Miroslav Novak Just updating keywords. HornetQ was not updated in EAP 6.1.0.ER5 - will be verified with ER6. Issue is fixed in EAP 6.1.0.ER6 (HQ 2.3.0.Final). Thanks to all involved! Note for documentation: This issue was related only to NIO connectors. When NIO is enabled on the connector then it's also recommended enable parameter "use-nio-global-worker-pool" to avoid situation where too many threads is created on a machine which could possible lead to OOM. Example of configuration of such connector: <netty-connector name="netty" socket-binding="messaging"> <param key="use-nio" value="true"/> <param key="use-nio-global-worker-pool" value="true"/> </netty-connector> I have edited the proposed Release Notes text for this ticket. Please review it for technical accuracy. I find it strange that although this ticket is marked 'Resolved', we are recommending a parameter be set to minimize the risk of an OOM error. Is the setting of this parameter merely a safety measure or is it *required*? Attention: Miroslav Do I understand correctly that for this issue you're recommending this text also appear in the appropriate EAP book, or only in the Release Notes? Note from ataylor: "ataylor: the new parameter will use a shared pool of workers for netty per connection [10:49pm] ataylor: rather that a pool per connection" @ Russel There are some bad characters in the note for this bz in: http://documentation-devel.engineering.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/6.1/html-single/6.1.0_Release_Notes/index.html#idp1786488 Text: <netty-connector name="netty" socket-binding="messaging"> <param key="use-nio" value="true"/> <param key="use-nio-global-worker-pool" value="true"/> </netty-connector> Can you take a look at it, please? Thanks, Mirek |