Bug 1410869

Summary: [GSS](6.4.z) AsyncContext.complete() blocks sometimes as long as the DEFAULT_CONNECTION_TIMEOUT
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Aaron Ogburn <aogburn>
Component: WebAssignee: Fedor Gavrilov <fgavrilo>
Status: CLOSED CURRENTRELEASE QA Contact: Radim Hatlapatka <rhatlapa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.4.11, 6.4.12CC: bmaxwell, fgavrilo, jclere, leiyu, mcada, msochure, pjurak, rmaucher
Target Milestone: CR1   
Target Release: EAP 6.4.15   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-19 08:06:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1419353, 1423507, 1427196    

Description Aaron Ogburn 2017-01-06 16:23:45 UTC
Description of problem:

For a small number of asynchronous http requests, invoking AsyncContext.complete() blocks the current thread, sometimes for as long as the timeout set in "org.apache.coyote.http11.DEFAULT_CONNECTION_TIMEOUT". For instance:

"pool-5-thread-3" prio=6 tid=0x000000000b3f9800 nid=0x11f0 waiting for monitor entry [0x000000000e5bf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:1149)
        - waiting to lock <0x00000000eb88a8b8> (a org.apache.coyote.http11.Http11Processor)
        at org.apache.coyote.Request.action(Request.java:360)
        at org.apache.catalina.connector.Request.wakeup(Request.java:2601)
        at org.apache.catalina.connector.Request$AsyncContextImpl.complete(Request.java:3419)
        at com.cgi.dsp.async.servlet.AsyncReadServlet$1.run(AsyncReadServlet.java:69)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
        - <0x00000000eda09618> (a java.util.concurrent.ThreadPoolExecutor$Worker)

The thread which holds the Http11Processor lock 0x00000000eb88a8b8,

"http-127.0.0.1:8080-2" daemon prio=6 tid=0x000000000b3f3800 nid=0x22b0 runnable [0x000000000df3f000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:713)
        at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:351)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:819)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:656)
        - locked <0x00000000eb88a8b8> (a org.apache.coyote.http11.Http11Processor)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:921)
        at java.lang.Thread.run(Unknown Source)


Version-Release number of selected component (if applicable):

EAP 6.4.11 & 6.4.12


Actual results:

AsyncContext.complete() may block for long periods due to pending Http11Processor.process calls on the same processor.


Expected results:

AsyncContext.complete() is not blocked by InternalInputBuffer.parseRequestLine.


Additional info:

This is introduced by the synchronization added to fix bz-1370182.  The issue is avoided when using the NIO connector.

Comment 25 Michael Cada 2017-02-23 11:23:07 UTC
Failed QA.

r2682 did not make it to JBossWeb 7.5.21.Final-redhat-1

Comment 29 Michael Cada 2017-04-26 11:43:44 UTC
Verified with EAP 6.4.15.CP.CR2

Comment 30 Petr Penicka 2017-05-19 08:06:36 UTC
Released on May 18 as part of EAP 6.4.15.