Bug 1032552
Summary: | [QE] (6.3.0) OOM due lots of org.apache.tomcat.util.net.JIoEndpoint$Poller objects | |||
---|---|---|---|---|
Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Radim Hatlapatka <rhatlapa> | |
Component: | Web, Remoting | Assignee: | Rémy Maucherat <rmaucher> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Radim Hatlapatka <rhatlapa> | |
Severity: | high | Docs Contact: | Russell Dickenson <rdickens> | |
Priority: | unspecified | |||
Version: | 6.2.0 | CC: | chaowan, lthon, myarboro, nobody, smumford | |
Target Milestone: | --- | |||
Target Release: | EAP 6.3.0 | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | Bug Fix | ||
Doc Text: |
A thread leak that could result in an OutOfMemoryError has been corrected in this release of JBoss EAP 6. The leak was traced to the JIoEndpoint class. The related code has been patched and the OOM error no longer presents.
|
Story Points: | --- | |
Clone Of: | ||||
: | 1065333 (view as bug list) | Environment: | ||
Last Closed: | 2014-06-28 15:38:08 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1062298, 1065333 |
Description
Radim Hatlapatka
2013-11-20 11:46:15 UTC
This is error which is shown when the OOM error occurs: 19:04:52,257 ERROR [org.xnio.listener] (Remoting "insignia:MANAGEMENT" read-1) A channel event listener threw an exception: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_45] at java.lang.Thread.start(Thread.java:713) [rt.jar:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) [rt.jar:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) [rt.jar:1.7.0_45] at org.xnio.XnioWorker.execute(XnioWorker.java:577) [xnio-api-3.0.7.GA-redhat-1.jar:3.0.7.GA-redhat-1] at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:218) [jboss-remoting-3.2.18.GA-redhat-1.jar:3.2.18.GA-redhat-1] at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45) [jboss-remoting-3.2.18.GA-redhat-1.jar:3.2.18.GA-redhat-1] at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72) [xnio-api-3.0.7.GA-redhat-1.jar:3.0.7.GA-redhat-1] at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189) [xnio-api-3.0.7.GA-redhat-1.jar:3.0.7.GA-redhat-1] at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103) [xnio-api-3.0.7.GA-redhat-1.jar:3.0.7.GA-redhat-1] at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72) [xnio-api-3.0.7.GA-redhat-1.jar:3.0.7.GA-redhat-1] at org.xnio.nio.NioHandle.run(NioHandle.java:90) at org.xnio.nio.WorkerThread.run(WorkerThread.java:187) Some connector components use "a lot" of memory, but will last for the lifetime of the connector and are used to keep track of the connections state. The test could be doing a lot of stop/start of the connector, which is allowed but obviously not very common use. The OOM actually occurs in another component (remoting, which uses xnio for its IO). Yes, that probably will be the case and yes this is definitely not a very common use case. The scenario is that I run lots of management operations via CLI and verify whether the changed configuration was correctly handled by the server (mostly via deployed simple app. Many changes require reload, which is probably the place where the connectors stops/starts are done leading to the OOM. The question is, is it correct behavior of connectors and memory when server reload is done? Or may I somehow influence its behavior to prevent OOM when running many CLI operations => many operations through native interface? No I've got this OOM error [1], this time it occurred in Web subsystem component. Isn't there option to limit amount of threads used for connectors? [1] ERROR [org.apache.tomcat.util.net] (http-/127.0.0.1:8080-Acceptor-0) JBWEB003011: Error allocating socket processor: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) [rt.jar:1.7.0_45] at java.lang.Thread.start(Thread.java:713) [rt.jar:1.7.0_45] at org.apache.tomcat.util.net.JIoEndpoint$Worker.start(JIoEndpoint.java:940) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint.newWorkerThread(JIoEndpoint.java:1162) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint.createWorkerThread(JIoEndpoint.java:1141) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint.getWorkerThread(JIoEndpoint.java:1173) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:1211) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:312) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45] I don't think the connector can be immediately GCed right after stopping, as the threads may not all be interrupted etc. So there could be a limit on what you can do here. How long does it take to GC in practice ? In my case I am getting the OOM after 13 minutes of testing with multiple CLI operations. This is most likely related to BZ#1035787, I have applied patch proposed by lthon and the OOM no longer occurs. I committed a fix to 7.3 and 7.4. Good catch (it leaks two to three threads per restart of the connector). *** Bug 1035787 has been marked as a duplicate of this bug. *** It was r2313 in 7.4 and r2315 in 7.3. I have verified that the issue is fixed in EAP 6.3.0.DR4 Adding Release Notes text and marking for inclusion in the 6.3.0 Release Notes document. |