Bug 882048 - Shutdown process hangs using thread pool executor
Summary: Shutdown process hangs using thread pool executor
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Web
Version: 6.0.0
Hardware: All
OS: All
unspecified
medium
Target Milestone: ---
: EAP 6.4.0
Assignee: Rémy Maucherat
QA Contact: Radim Hatlapatka
Russell Dickenson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-11-30 02:07 UTC by Eiichi Nagai
Modified: 2018-12-04 14:53 UTC (History)
7 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-06-23 15:10:55 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker AS7-6105 0 Major Closed Shutdown process hangs using thread pool executor 2018-12-03 21:33:17 UTC

Description Eiichi Nagai 2012-11-30 02:07:12 UTC
Description of problem:
When AJP connecotr uses thread pool executor configuration[1], AJP's worker thread is waited by AjpProcessor.read()[2]. QueueExecuter recognizes it as active thread. Therefor, unless httpd server shut down, EAP server shutdown process does not finish forever. 

[1] standalone.xml
        <subsystem xmlns="urn:jboss:domain:threads:1.1">
            <bounded-queue-thread-pool name="http-executor">
                <queue-length count="1"/>
                <max-threads count="1"/>
            </bounded-queue-thread-pool>
        </subsystem>
--- snip ---
        <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
            <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
            <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" executor="http-executor"/>
            <virtual-server name="default-host" enable-welcome-root="true">
                <alias name="localhost"/>
                <alias name="example.com"/>
            </virtual-server>
        </subsystem>

[2]
"http-executor-threads - 1" prio=6 tid=0x55515000 nid=0x20c0 runnable [0x5590f000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1131)
   at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1213)
   at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:451)
   at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:452)
   at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:519)
   at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
   at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801)
   at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
   at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:821)
   at java.lang.Thread.run(Thread.java:662)
   at org.jboss.threads.JBossThread.run(JBossThread.java:122)


Version-Release number of selected component (if applicable):
6.0 GA
6.0.1 (AS 7.1.3.Final-redhat-4)

How reproducible:
Always (when using thread pool executor)

Steps to Reproduce:
1. Configure thread pool executor to AJP connector like [1].
2. Connect to AJP connector in EAP6 via httpd. 
3. You can confirm AJP worker thread likea like [2].
4. Shut down EAP server
5. You can confirm that shutdown process does not finish.
  
Actual results:
EAP shutdown process hangs unless httpd server shut down.

Expected results:
A graceful shutdown of EAP server even if httpd server does not shut down.

Additional info:
N/A

Comment 1 Aaron Ogburn 2012-12-04 22:06:24 UTC
I've checked a few things in debugging this:

1) QueueExecutor is indeed being shutdown after confirming its stop variable value in a heap dump.  Since shutdown is called, the worker threads are interrupted.

2) The interrupts do nothing to the keepalive worker threads on the socket read.  They do not break out of the read in any way and so shutdown just waits on the long, uninterruptible reads.

Comment 2 JBoss JIRA Server 2012-12-07 09:44:20 UTC
Remy Maucherat <rmaucher> updated the status of jira AS7-6105 to Resolved

Comment 3 JBoss JIRA Server 2012-12-07 09:44:20 UTC
Remy Maucherat <rmaucher> made a comment on jira AS7-6105

It is normal behavior if the threads are not daemon threads. There's no benefit to using an executor with the java.io AJP connector (it will almost never give any thread back).

Comment 4 JBoss JIRA Server 2012-12-07 14:55:04 UTC
David Lloyd <david.lloyd> made a comment on jira AS7-6105

It would be nice if we could forbid such a configuration...

Comment 5 JBoss JIRA Server 2012-12-07 15:05:22 UTC
Remy Maucherat <rmaucher> made a comment on jira AS7-6105

Do you think it should go as far as check if the executor configuration has a daemon flag using a simple runnable ? I don't see any other way to do it.

Comment 6 JBoss JIRA Server 2012-12-10 16:24:44 UTC
Mathieu Lachance <mathieu> made a comment on jira AS7-6105

jboss thread subsystem do not expose the daemon flag

Comment 7 JBoss JIRA Server 2012-12-10 16:25:26 UTC
Mathieu Lachance <mathieu> made a comment on jira AS7-6105

jboss thread subsystem do not expose the daemon flag, I think the issue should be re-opened.

Comment 8 JBoss JIRA Server 2012-12-10 16:36:16 UTC
David Lloyd <david.lloyd> made a comment on jira AS7-6105

To elaborate....

- The daemon flag is not sufficient as the connector can be shut down without shutting down the container; to do it "right" the thread pool has to be shut down after the connector is shut down.
- If this is not possible to do with an external executor-based thread pool, then we simply shouldn't allow users to configure an executor for AJP within the subsystem configuration.
- If this *generally* not possible to shut down the thread pool after shutting down the connector, then it's a real bug and we have to address that.

Comment 9 JBoss JIRA Server 2012-12-10 18:15:24 UTC
Mathieu Lachance <mathieu> made a comment on jira AS7-6105

I guess this is the same concern as expressed here : https://issues.jboss.org/browse/AS7-5771

Comment 10 JBoss JIRA Server 2012-12-13 01:03:44 UTC
Eiichi Nagai <enagai> updated the status of jira AS7-6105 to Reopened

Comment 11 JBoss JIRA Server 2012-12-13 01:03:44 UTC
Eiichi Nagai <enagai> made a comment on jira AS7-6105

I have reopened this JIRA. I am interested in David's comment. I believe that you tackle again regarding this behavior.

Comment 13 Fernando Nasser 2013-10-17 21:17:18 UTC
Can someone chose the right component please?

Comment 14 Martin Velas 2014-08-11 08:17:23 UTC
Issue is still valid for EAP 6.3.0.ER10.

Comment 15 Rémy Maucherat 2014-08-26 13:34:59 UTC
This executor configuration looks obviously broken (same as 1042754). I don't see a real way to handle this.

Comment 16 Dimitris Andreadis 2014-11-05 16:59:39 UTC
It should probably change to WONT_FIX.

Comment 17 Ivo Studensky 2015-06-23 15:10:55 UTC
based on comment 15 and comment 16


Note You need to log in before you can comment on or make changes to this bug.