Bug 985191

Summary: RejectedExecutionException printed at ERROR level when non-blocking web thread pool is used
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: James Livingston <jlivings>
Component: WebAssignee: Rémy Maucherat <rmaucher>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.1.0CC: jlivings, joallen, rmaucher
Target Milestone: Pending   
Target Release: EAP 6.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-17 23:22:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 995432    
Bug Blocks: 1001872    

Description James Livingston 2013-07-17 04:50:49 UTC
You can configure a non-blocking thread pool for the web container, such as the following, so that extra connections are dropped.
    <queueless-thread-pool name="web">
        <max-threads count="10"/>
        <keepalive-time time="60" unit="seconds"/>
    </queueless-thread-pool>


When it does drop extra requests, the RejectedExecutionException is logged at ERROR level:

JBWEB003011: Error allocating socket processor: java.util.concurrent.RejectedExecutionException
	at org.jboss.threads.QueuelessExecutor.execute(QueuelessExecutor.java:375) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
	at org.jboss.threads.DelegatingBlockingExecutorService.execute(DelegatingBlockingExecutorService.java:42) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
	at org.jboss.as.threads.ManagedExecutorService.execute(ManagedExecutorService.java:64) [jboss-as-threads-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
	at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:1218) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:312) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]


I think that ERROR level logging is too high, because this will only happen when someone configured a non-blocking finite thread pool for the web container, and then dropping requests is what they would expect.

DEBUG level may be appropriate, since then it will not show up unless the log level is changed.


This could be done by adding a catch for the specific exception to JIoEndpoint.processSocket()

Comment 1 JBoss JIRA Server 2013-07-18 02:10:02 UTC
James Livingston <jlivings> made a comment on jira JBWEB-276

This is a potential patch which adds a new error message (is using one past the last number correct?) and uses it when a RejectedExceutionException is caught instead of the normal message.

It alters the NIO, APR and JIO endpoints

Comment 2 Rémy Maucherat 2013-07-18 16:33:54 UTC
It's a serious problem when it occurs, so I don't think I can agree with this change.

Comment 3 James Livingston 2013-07-19 04:44:02 UTC
For the default blocking thread pools, a RejectedExecutionException means a serious problem occurred, yes.

If you switch the web container over to using a non-blocking thread pool however, so that it drops requests rather than blocking, it is expected that you would receive these.


Currently if you want to make JBoss drop web requests when there are too many, it writes a large number of these errors out into the log, which would have performance implications reducing the usefulness of dropping request under excessive load.

If you don't want to hide the message since it is serious for blocking thread pools, is there an alternative which would let people using non-blocking thread pools hide it?  It would be possible to move it to a different logging category, which could be turned off, but adding a new logging category for one message seems excessive.

Comment 4 Rémy Maucherat 2013-07-22 07:06:47 UTC
I agree it is obviously the intended design in that case, but I don't know how to differentiate between the two situations since the executor is a black box. So using a dedicated subcategory for these few messages would allow filtering them out, but they should remain at error level.

Comment 5 JBoss JIRA Server 2013-07-31 05:24:03 UTC
James Livingston <jlivings> made a comment on jira JBWEB-276

This patch is similar, except it creates a new logging category (org.apache.tomcat.util.executor) and leaves the level at ERROR.

Comment 7 JBoss JIRA Server 2013-08-26 07:16:28 UTC
Remy Maucherat <rmaucher> updated the status of jira JBWEB-276 to Resolved

Comment 8 JBoss JIRA Server 2013-08-26 07:16:28 UTC
Remy Maucherat <rmaucher> made a comment on jira JBWEB-276

The logging now occurs on a different category.

Comment 9 James Livingston 2013-08-28 02:41:01 UTC
The logging change was committed upstream.

Comment 10 Paul Gier 2013-09-17 22:11:56 UTC
Can this issue be closed?

Comment 11 James Livingston 2013-09-17 23:21:37 UTC
The fix should be in 7.2.2, which according to bug 995432 is in 6.1.1, so I would say yes.