Bug 1001872 - Non-blocking executors are not safe to use in web container
Summary: Non-blocking executors are not safe to use in web container
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Other, Web
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: Pending
: One-off release
Assignee: James Livingston
QA Contact: Radim Hatlapatka
URL:
Whiteboard:
Depends On: 985191 985204
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-28 02:48 UTC by James Livingston
Modified: 2018-12-06 15:15 UTC (History)
4 users (show)

Fixed In Version:
Clone Of: 985204
Environment:
Last Closed: 2013-09-16 16:03:28 UTC
Type: Support Patch
Embargoed:


Attachments (Terms of Use)
jboss-threads-2.1.1.Final-redhat-0-bz-1001872.jar (117.14 KB, application/x-java-archive)
2013-08-28 02:52 UTC, James Livingston
no flags Details
bz-1001872.zip (2.21 MB, application/x-zip)
2013-08-30 03:19 UTC, James Livingston
no flags Details
BZ1001872.zip (2.22 MB, application/zip)
2013-08-30 21:26 UTC, Jimmy Wilson
no flags Details
BZ1001872.zip (2.21 MB, application/octet-stream)
2013-08-30 21:40 UTC, Brad Maxwell
no flags Details
BZ1001872.zip (2.21 MB, application/zip)
2013-09-03 13:46 UTC, Jimmy Wilson
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 513183 0 None None None Never

Description James Livingston 2013-08-28 02:48:12 UTC
+++ This bug was initially created as a clone of Bug #985204 +++

When org.apache.tomcat.util.net.JIoEndpoint catches Throwable and write a log message, it does not do anything with the Socket. This means it will be left open until the client times out and closes the connection.


This causes problems when non-blocking thread pools are used by the web container, since a RejectedExecutionException is thrown.

--- Additional comment from James Livingston on 2013-07-22 11:00:14 EST ---

This is actually caused by QueueExecutor not emitting RejectedExecutionExceptions when there is no handoff executor, but QueuelessExecutor doing so.

Comment 1 James Livingston 2013-08-28 02:52:02 UTC
Created attachment 791224 [details]
jboss-threads-2.1.1.Final-redhat-0-bz-1001872.jar

Build of jboss-threads 2.1.1, which contains the two related fixes only (on top of standard EAP 6.1.0 release)

Comment 2 James Livingston 2013-08-30 03:13:26 UTC
PATCH NAME:
    BZ1001872
PRODUCT NAME:
     JBoss Enterprise Application Platform (EAP)
VERSION:
    6.1.0
SHORT DESCRIPTION:
    Correctly reject requests when using non-blocking web executor
LONG DESCRIPTION:
    When a non-blocking executor was used by the web container, tasks were
    not properly rejected. This would cause the connection to be left open
    and leaked until garbage collected. If the tasks were correctly rejected
    logging was emitted at ERROR level, and that is now moved to the logging
    category org.apache.tomcat.util.executor.
MANUAL INSTALL INSTRUCTIONS:
    Backup and remove the following files:
        $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/jbossweb-7.2.0.Final-redhat-1.jar
        $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml
        $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/jboss-threads-2.1.0.Final-redhat-1.jar
        $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/module.xml


    Extract the patched files by either:
        Using unzip:
            unzip -d $JBOSS_HOME/ bz-1001872.zip

        Or by extracting the files from the zip to the following locations:
            $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/jbossweb-7.2.0.Final-redhat-1-bz1001872.jar
            $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml
            $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/jboss-threads-2.1.1.Final-redhat-0-bz-1001872.jar
            $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/module.xml

    Instructions to uninstall:
        Restore the following files that were backed up before the patch was installed:
        $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/jbossweb-7.2.0.Final-redhat-1.jar
        $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml
        $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/jboss-threads-2.1.0.Final-redhat-1.jar
        $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/module.xml


COMPATIBILITY:
    No known compatibility issues
DEPENDENCIES:
    None
SUPERSEDES:
    None
SUPERSEDED BY:
    None
CREATOR:
    James Livingston
DATE:
    30 August 2013

Comment 3 James Livingston 2013-08-30 03:19:33 UTC
Created attachment 791992 [details]
bz-1001872.zip

Comment 4 James Livingston 2013-08-30 03:23:25 UTC
To test, you need to configure a non-blocking executor for the web subsystem, such as:

    <subsystem xmlns="urn:jboss:domain:threads:1.1">
        <thread-factory name="http-connector-factory-a" group-name="http-a" thread-name-pattern="HTTP-%t" />
        <bounded-queue-thread-pool name="web-a">
            <core-threads count="2"/>
            <queue-length count="2"/>
            <max-threads count="2"/>
            <thread-factory name="http-connector-factory-a"/>
        </bounded-queue-thread-pool>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:web:1.4" default-virtual-server="default-host" native="false">
        <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="web-a"/>
        ...
     </subsystem>


Then when sufficient load is applied (5 concurrent requests should be enough for the above configuration), connections should be dropped immediately and not left open for a long time.

There will also be error messages reported to the JBoss log, which should be in the "org.apache.tomcat.util.executor" category (not "org.apache.tomcat.util").



This patch also incorporates the fixed from the security errata of bug 873664. Please also run the tests for that security flaw to ensure there is no security regression.

Comment 5 Jimmy Wilson 2013-08-30 21:26:08 UTC
Created attachment 792319 [details]
BZ1001872.zip

Fixed format per https://docspace.corp.redhat.com/docs/DOC-151974

Comment 6 Brad Maxwell 2013-08-30 21:40:43 UTC
Created attachment 792322 [details]
BZ1001872.zip

Comment 8 Radim Hatlapatka 2013-09-02 11:04:34 UTC
The zip structure of the patch doesn't correspond to the patch instructions.

The inner zip file (with the actual patch) should start with modules in the root and not be in another directory (bz-1001872)

Comment 9 Jimmy Wilson 2013-09-03 13:46:17 UTC
Created attachment 793196 [details]
BZ1001872.zip

Corrected internal bz-1001872.zip which had an extra directory structure added inadvertently.

Comment 11 Radim Hatlapatka 2013-09-03 14:39:52 UTC
I have reproduced and verified that the patch [1] fixes the issue with non-blocking executors (I have used ab from httpd-tools for creating concurrent requests) + CVE-2012-5885, CVE-2012-5886, CVE-2012-5887 verified via reproducers

Regression tests passed

[1]
62d02ef721b0d2f019c1d12e93a492d7  BZ1001872.zip
93d43be993ae6c42209cdff10fb722d1  BZ1001872/bz-1001872.zip (inner zip)


Note You need to log in before you can comment on or make changes to this bug.