Bug 985191 - RejectedExecutionException printed at ERROR level when non-blocking web thread pool is used
RejectedExecutionException printed at ERROR level when non-blocking web threa...
Status: CLOSED CURRENTRELEASE
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Web (Show other bugs)
6.1.0
Unspecified Unspecified
unspecified Severity unspecified
: Pending
: EAP 6.2.0
Assigned To: Rémy Maucherat
:
Depends On: 995432
Blocks: 1001872
  Show dependency treegraph
 
Reported: 2013-07-17 00:50 EDT by James Livingston
Modified: 2016-02-21 19:56 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-17 19:22:37 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker JBWEB-276 Major Resolved Using non-blocking thread pool causes excessive RejectedExecutionExceptions to be logged 2015-06-12 11:36:50 EDT

  None (edit)
Description James Livingston 2013-07-17 00:50:49 EDT
You can configure a non-blocking thread pool for the web container, such as the following, so that extra connections are dropped.
    <queueless-thread-pool name="web">
        <max-threads count="10"/>
        <keepalive-time time="60" unit="seconds"/>
    </queueless-thread-pool>


When it does drop extra requests, the RejectedExecutionException is logged at ERROR level:

JBWEB003011: Error allocating socket processor: java.util.concurrent.RejectedExecutionException
	at org.jboss.threads.QueuelessExecutor.execute(QueuelessExecutor.java:375) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
	at org.jboss.threads.DelegatingBlockingExecutorService.execute(DelegatingBlockingExecutorService.java:42) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
	at org.jboss.as.threads.ManagedExecutorService.execute(ManagedExecutorService.java:64) [jboss-as-threads-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
	at org.apache.tomcat.util.net.JIoEndpoint.processSocket(JIoEndpoint.java:1218) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:312) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]


I think that ERROR level logging is too high, because this will only happen when someone configured a non-blocking finite thread pool for the web container, and then dropping requests is what they would expect.

DEBUG level may be appropriate, since then it will not show up unless the log level is changed.


This could be done by adding a catch for the specific exception to JIoEndpoint.processSocket()
Comment 1 JBoss JIRA Server 2013-07-17 22:10:02 EDT
James Livingston <jlivings@redhat.com> made a comment on jira JBWEB-276

This is a potential patch which adds a new error message (is using one past the last number correct?) and uses it when a RejectedExceutionException is caught instead of the normal message.

It alters the NIO, APR and JIO endpoints
Comment 2 Rémy Maucherat 2013-07-18 12:33:54 EDT
It's a serious problem when it occurs, so I don't think I can agree with this change.
Comment 3 James Livingston 2013-07-19 00:44:02 EDT
For the default blocking thread pools, a RejectedExecutionException means a serious problem occurred, yes.

If you switch the web container over to using a non-blocking thread pool however, so that it drops requests rather than blocking, it is expected that you would receive these.


Currently if you want to make JBoss drop web requests when there are too many, it writes a large number of these errors out into the log, which would have performance implications reducing the usefulness of dropping request under excessive load.

If you don't want to hide the message since it is serious for blocking thread pools, is there an alternative which would let people using non-blocking thread pools hide it?  It would be possible to move it to a different logging category, which could be turned off, but adding a new logging category for one message seems excessive.
Comment 4 Rémy Maucherat 2013-07-22 03:06:47 EDT
I agree it is obviously the intended design in that case, but I don't know how to differentiate between the two situations since the executor is a black box. So using a dedicated subcategory for these few messages would allow filtering them out, but they should remain at error level.
Comment 5 JBoss JIRA Server 2013-07-31 01:24:03 EDT
James Livingston <jlivings@redhat.com> made a comment on jira JBWEB-276

This patch is similar, except it creates a new logging category (org.apache.tomcat.util.executor) and leaves the level at ERROR.
Comment 7 JBoss JIRA Server 2013-08-26 03:16:28 EDT
Remy Maucherat <rmaucher@redhat.com> updated the status of jira JBWEB-276 to Resolved
Comment 8 JBoss JIRA Server 2013-08-26 03:16:28 EDT
Remy Maucherat <rmaucher@redhat.com> made a comment on jira JBWEB-276

The logging now occurs on a different category.
Comment 9 James Livingston 2013-08-27 22:41:01 EDT
The logging change was committed upstream.
Comment 10 Paul Gier 2013-09-17 18:11:56 EDT
Can this issue be closed?
Comment 11 James Livingston 2013-09-17 19:21:37 EDT
The fix should be in 7.2.2, which according to bug 995432 is in 6.1.1, so I would say yes.

Note You need to log in before you can comment on or make changes to this bug.