Bug 998779

Hi, 
I don't reproduce your 'lock'. It looks like the Catalina connector is responsible for the time it takes to shutdown the server properly but the underemployment in itself is quite fast.
Using the -Dorg.apache.coyote.MAX_PAUSE_WAIT=5 I can reduce te timeout duration for the server to shutdown.
I'm on EAP 6.2.0 current dev status.

Created attachment 796224 [details]
reproducer

Attached is the reproducer from the case.

1) Deploy it to EAP 6 and start the server
2) Go to http://localhost:8080/casesupport/ and click the submit button
3) trigger a redeployment of the app (e.g. touch standalone/deploy/00922614.war)
4) Look at the stack output which includes:

--
Full thread dump OpenJDK 64-Bit Server VM (24.0-b56 mixed mode):

"ServerService Thread Pool -- 57" prio=10 tid=0x00007fabcc1d4000 nid=0x12c1 waiting on condition [0x00007fac11b72000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ed133320> (a java.util.concurrent.Semaphore$FairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
	at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:416)
	at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3916)
	- locked <0x00000000eb436a40> (a org.apache.catalina.core.StandardContext)
	at org.jboss.as.web.deployment.WebDeploymentService.doStop(WebDeploymentService.java:171)
	at org.jboss.as.web.deployment.WebDeploymentService.access$100(WebDeploymentService.java:60)
	at org.jboss.as.web.deployment.WebDeploymentService$2.run(WebDeploymentService.java:113)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

   Locked ownable synchronizers:
	- <0x00000000e7632c40> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"http-/127.0.0.1:8080-1" daemon prio=10 tid=0x00007fac000ed000 nid=0x12b2 waiting on condition [0x00007fac110c9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at com.redhat.gss.servlet.CaseSupportReproduceServlet.doPost(CaseSupportReproduceServlet.java:35)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
	at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:134)
	at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99)
	at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92)
	at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
	at java.lang.Thread.run(Thread.java:724)

   Locked ownable synchronizers:
	- None

...
--

The missing piece of info was that you have to have a web.xml with the distribuable element set.
EAP PR : https://github.com/jbossas/jboss-eap/pull/403

This issue is still valid even for the 6.2.0.CR1 preview bits.

Now when you start the server with -Djboss.web.session.manager.stop.timeout=15000 and tries to undeploy the application it shows after 15s [1]. I still see in deployments directory for quite some time isundeploying for the web app

After this message I am trying to shutdown the EAP server and it still takes about 5 minutes


[1]
13:39:31,227 WARN  [org.jboss.as.web.session] (ServerService Thread Pool -- 51) JBAS018228: Shutting down the session manager while there are still requests being processed after waiting for 15000 milliseconds
13:40:01,274 WARN  [org.infinispan.transaction.TransactionTable] (ServerService Thread Pool -- 58) ISPN000100: Stopping, but there are 1 local transactions and 0 remote transactions that did not finish in time.

Created attachment 822422 [details]
Jstack when shutting down the EAP server after reproducer web app being undeployed

Created attachment 822423 [details]
server jstack when undeployment of distributable app should be done

jstack output after the undeployment shows the warn message regarding stopping session

I looked a little more deeply and the duration shutdown process is caused by Catalina connector as was already written in Comment #1. The time can be then reduced using -Dorg.apache.coyote.MAX_PAUSE_WAIT=5 as was suggested.

The undeployment works with possibility to specify undeployment timeout via -Djboss.web.session.manager.stop.timeout=15000 

=> verified

I looked a little more deeply and the duration shutdown process is caused by Catalina connector as was already written in Comment #1. The time can be then reduced using -Dorg.apache.coyote.MAX_PAUSE_WAIT=5 as was suggested.

The undeployment works with possibility to specify undeployment timeout via -Djboss.web.session.manager.stop.timeout=15000 

=> verified using the 6.2.0.CR1 preview bits.