Bug 1001872

Summary: Non-blocking executors are not safe to use in web container
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: James Livingston <jlivings>
Component: Other, WebAssignee: James Livingston <jlivings>
Status: CLOSED CURRENTRELEASE QA Contact: Radim Hatlapatka <rhatlapa>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.1.0CC: bmaxwell, cobrien, jawilson, joallen
Target Milestone: Pending   
Target Release: One-off release   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
PATCH NAME: bz-1001872 PRODUCT NAME: JBoss Enterprise Application Platform (EAP) VERSION: 6.1.0 SHORT DESCRIPTION: Correctly reject requests when using non-blocking web executor LONG DESCRIPTION: When a non-blocking executor was used by the web container, tasks were not properly rejected. This would cause the connection to be left open and leaked until garbage collected. If the tasks were correctly rejected logging was emitted at ERROR level, and that is now moved to the logging category org.apache.tomcat.util.executor. MANUAL INSTALL INSTRUCTIONS: Backup and remove the following files: $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/jbossweb-7.2.0.Final-redhat-1.jar $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/jboss-threads-2.1.0.Final-redhat-1.jar $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/module.xml Extract the patched files by either: Using unzip: unzip -d $JBOSS_HOME/ bz-1001872.zip Or by extracting the files from the zip to the following locations: $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/jbossweb-7.2.0.Final-redhat-1-bz1001872.jar $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/jboss-threads-2.1.1.Final-redhat-0-bz-1001872.jar $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/module.xml Instructions to uninstall: Restore the following files that were backed up before the patch was installed: $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/jbossweb-7.2.0.Final-redhat-1.jar $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/jboss-threads-2.1.0.Final-redhat-1.jar $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/module.xml COMPATIBILITY: No known compatibility issues DEPENDENCIES: None SUPERSEDES: None SUPERSEDED BY: None CREATOR: James Livingston DATE: 30 August 2013
Story Points: ---
Clone Of: 985204 Environment:
Last Closed: 2013-09-16 16:03:28 UTC Type: Support Patch
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 985191, 985204    
Bug Blocks:    
Attachments:
Description Flags
jboss-threads-2.1.1.Final-redhat-0-bz-1001872.jar
none
bz-1001872.zip
none
BZ1001872.zip
none
BZ1001872.zip
none
BZ1001872.zip none

Description James Livingston 2013-08-28 02:48:12 UTC
+++ This bug was initially created as a clone of Bug #985204 +++

When org.apache.tomcat.util.net.JIoEndpoint catches Throwable and write a log message, it does not do anything with the Socket. This means it will be left open until the client times out and closes the connection.


This causes problems when non-blocking thread pools are used by the web container, since a RejectedExecutionException is thrown.

--- Additional comment from James Livingston on 2013-07-22 11:00:14 EST ---

This is actually caused by QueueExecutor not emitting RejectedExecutionExceptions when there is no handoff executor, but QueuelessExecutor doing so.

Comment 1 James Livingston 2013-08-28 02:52:02 UTC
Created attachment 791224 [details]
jboss-threads-2.1.1.Final-redhat-0-bz-1001872.jar

Build of jboss-threads 2.1.1, which contains the two related fixes only (on top of standard EAP 6.1.0 release)

Comment 2 James Livingston 2013-08-30 03:13:26 UTC
PATCH NAME:
    BZ1001872
PRODUCT NAME:
     JBoss Enterprise Application Platform (EAP)
VERSION:
    6.1.0
SHORT DESCRIPTION:
    Correctly reject requests when using non-blocking web executor
LONG DESCRIPTION:
    When a non-blocking executor was used by the web container, tasks were
    not properly rejected. This would cause the connection to be left open
    and leaked until garbage collected. If the tasks were correctly rejected
    logging was emitted at ERROR level, and that is now moved to the logging
    category org.apache.tomcat.util.executor.
MANUAL INSTALL INSTRUCTIONS:
    Backup and remove the following files:
        $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/jbossweb-7.2.0.Final-redhat-1.jar
        $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml
        $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/jboss-threads-2.1.0.Final-redhat-1.jar
        $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/module.xml


    Extract the patched files by either:
        Using unzip:
            unzip -d $JBOSS_HOME/ bz-1001872.zip

        Or by extracting the files from the zip to the following locations:
            $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/jbossweb-7.2.0.Final-redhat-1-bz1001872.jar
            $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml
            $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/jboss-threads-2.1.1.Final-redhat-0-bz-1001872.jar
            $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/module.xml

    Instructions to uninstall:
        Restore the following files that were backed up before the patch was installed:
        $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/jbossweb-7.2.0.Final-redhat-1.jar
        $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml
        $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/jboss-threads-2.1.0.Final-redhat-1.jar
        $JBOSS_HOME/modules/system/layers/base/org/jboss/threads/main/module.xml


COMPATIBILITY:
    No known compatibility issues
DEPENDENCIES:
    None
SUPERSEDES:
    None
SUPERSEDED BY:
    None
CREATOR:
    James Livingston
DATE:
    30 August 2013

Comment 3 James Livingston 2013-08-30 03:19:33 UTC
Created attachment 791992 [details]
bz-1001872.zip

Comment 4 James Livingston 2013-08-30 03:23:25 UTC
To test, you need to configure a non-blocking executor for the web subsystem, such as:

    <subsystem xmlns="urn:jboss:domain:threads:1.1">
        <thread-factory name="http-connector-factory-a" group-name="http-a" thread-name-pattern="HTTP-%t" />
        <bounded-queue-thread-pool name="web-a">
            <core-threads count="2"/>
            <queue-length count="2"/>
            <max-threads count="2"/>
            <thread-factory name="http-connector-factory-a"/>
        </bounded-queue-thread-pool>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:web:1.4" default-virtual-server="default-host" native="false">
        <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="web-a"/>
        ...
     </subsystem>


Then when sufficient load is applied (5 concurrent requests should be enough for the above configuration), connections should be dropped immediately and not left open for a long time.

There will also be error messages reported to the JBoss log, which should be in the "org.apache.tomcat.util.executor" category (not "org.apache.tomcat.util").



This patch also incorporates the fixed from the security errata of bug 873664. Please also run the tests for that security flaw to ensure there is no security regression.

Comment 5 Jimmy Wilson 2013-08-30 21:26:08 UTC
Created attachment 792319 [details]
BZ1001872.zip

Fixed format per https://docspace.corp.redhat.com/docs/DOC-151974

Comment 6 Brad Maxwell 2013-08-30 21:40:43 UTC
Created attachment 792322 [details]
BZ1001872.zip

Comment 8 Radim Hatlapatka 2013-09-02 11:04:34 UTC
The zip structure of the patch doesn't correspond to the patch instructions.

The inner zip file (with the actual patch) should start with modules in the root and not be in another directory (bz-1001872)

Comment 9 Jimmy Wilson 2013-09-03 13:46:17 UTC
Created attachment 793196 [details]
BZ1001872.zip

Corrected internal bz-1001872.zip which had an extra directory structure added inadvertently.

Comment 11 Radim Hatlapatka 2013-09-03 14:39:52 UTC
I have reproduced and verified that the patch [1] fixes the issue with non-blocking executors (I have used ab from httpd-tools for creating concurrent requests) + CVE-2012-5885, CVE-2012-5886, CVE-2012-5887 verified via reproducers

Regression tests passed

[1]
62d02ef721b0d2f019c1d12e93a492d7  BZ1001872.zip
93d43be993ae6c42209cdff10fb722d1  BZ1001872/bz-1001872.zip (inner zip)