Bug 985191 - RejectedExecutionException printed at ERROR level when non-blocking web thread pool is used
Summary: RejectedExecutionException printed at ERROR level when non-blocking web threa...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Web
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: Pending
: EAP 6.2.0
Assignee: Rémy Maucherat
QA Contact:
URL:
Whiteboard:
Depends On: 995432
Blocks: 1001872
TreeView+ depends on / blocked
 
Reported: 2013-07-17 04:50 UTC by James Livingston
Modified: 2018-12-02 16:12 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-09-17 23:22:37 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker JBWEB-276 0 Major Resolved Using non-blocking thread pool causes excessive RejectedExecutionExceptions to be logged 2015-06-12 15:36:50 UTC

Description James Livingston 2013-07-17 04:50:49 UTC
You can configure a non-blocking thread pool for the web container, such as the following, so that extra connections are dropped.
    <queueless-thread-pool name="web">
        <max-threads count="10"/>
        <keepalive-time time="60" unit="seconds"/>
    </queueless-thread-pool>


When it does drop extra requests, the RejectedExecutionException is logged at ERROR level:

JBWEB003011: Error allocating socket processor: java.util.concurrent.RejectedExecutionException
	at org.jboss.threads.QueuelessExecutor.execute(QueuelessExecutor.java:375) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
	at org.jboss.threads.DelegatingBlockingExecutorService.execute(DelegatingBlockingExecutorService.java:42) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
	at org.jboss.as.threads.ManagedExecutorService.execute(ManagedExecutorService.java:64) [jboss-as-threads-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
	at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:1218) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:312) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]


I think that ERROR level logging is too high, because this will only happen when someone configured a non-blocking finite thread pool for the web container, and then dropping requests is what they would expect.

DEBUG level may be appropriate, since then it will not show up unless the log level is changed.


This could be done by adding a catch for the specific exception to JIoEndpoint.processSocket()

Comment 1 JBoss JIRA Server 2013-07-18 02:10:02 UTC
James Livingston <jlivings> made a comment on jira JBWEB-276

This is a potential patch which adds a new error message (is using one past the last number correct?) and uses it when a RejectedExceutionException is caught instead of the normal message.

It alters the NIO, APR and JIO endpoints

Comment 2 Rémy Maucherat 2013-07-18 16:33:54 UTC
It's a serious problem when it occurs, so I don't think I can agree with this change.

Comment 3 James Livingston 2013-07-19 04:44:02 UTC
For the default blocking thread pools, a RejectedExecutionException means a serious problem occurred, yes.

If you switch the web container over to using a non-blocking thread pool however, so that it drops requests rather than blocking, it is expected that you would receive these.


Currently if you want to make JBoss drop web requests when there are too many, it writes a large number of these errors out into the log, which would have performance implications reducing the usefulness of dropping request under excessive load.

If you don't want to hide the message since it is serious for blocking thread pools, is there an alternative which would let people using non-blocking thread pools hide it?  It would be possible to move it to a different logging category, which could be turned off, but adding a new logging category for one message seems excessive.

Comment 4 Rémy Maucherat 2013-07-22 07:06:47 UTC
I agree it is obviously the intended design in that case, but I don't know how to differentiate between the two situations since the executor is a black box. So using a dedicated subcategory for these few messages would allow filtering them out, but they should remain at error level.

Comment 5 JBoss JIRA Server 2013-07-31 05:24:03 UTC
James Livingston <jlivings> made a comment on jira JBWEB-276

This patch is similar, except it creates a new logging category (org.apache.tomcat.util.executor) and leaves the level at ERROR.

Comment 7 JBoss JIRA Server 2013-08-26 07:16:28 UTC
Remy Maucherat <rmaucher> updated the status of jira JBWEB-276 to Resolved

Comment 8 JBoss JIRA Server 2013-08-26 07:16:28 UTC
Remy Maucherat <rmaucher> made a comment on jira JBWEB-276

The logging now occurs on a different category.

Comment 9 James Livingston 2013-08-28 02:41:01 UTC
The logging change was committed upstream.

Comment 10 Paul Gier 2013-09-17 22:11:56 UTC
Can this issue be closed?

Comment 11 James Livingston 2013-09-17 23:21:37 UTC
The fix should be in 7.2.2, which according to bug 995432 is in 6.1.1, so I would say yes.


Note You need to log in before you can comment on or make changes to this bug.