Bug 998779 - DistributableSessionManager should have a timeout on permit acquisition in stop()
DistributableSessionManager should have a timeout on permit acquisition in st...
Status: CLOSED CURRENTRELEASE
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Web (Show other bugs)
6.1.0
Unspecified Unspecified
unspecified Severity unspecified
: CR1
: EAP 6.2.0
Assigned To: Emmanuel Hugonnet (ehsavoie)
Radim Hatlapatka
Russell Dickenson
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-20 00:44 EDT by James Livingston
Modified: 2016-02-21 19:56 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-15 11:55:13 EST
Type: Enhancement
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
reproducer (5.31 KB, application/zip)
2013-09-10 23:10 EDT, James Livingston
no flags Details
Jstack when shutting down the EAP server after reproducer web app being undeployed (18.55 KB, text/plain)
2013-11-11 08:40 EST, Radim Hatlapatka
no flags Details
server jstack when undeployment of distributable app should be done (23.45 KB, text/plain)
2013-11-11 08:42 EST, Radim Hatlapatka
no flags Details

  None (edit)
Description James Livingston 2013-08-20 00:44:09 EDT
DistributableSessionManager.stop() does not use a timeout for acquiring the semaphore permits, so if there is a worker thread processing a request it will not finish undeploying until all requests have finished being handled.

It should have a timeout or mechanism to force undeployment even when there is slow application code.


To reproduce:
1) create a trivial servlet/JSP that does Thread.sleep(1000000)
2) load the server/JSP in a browser
3) Attempt to undeploy the application


The application will not be undeployed until the thread exits application code.
Comment 1 Emmanuel Hugonnet (ehsavoie) 2013-09-05 08:49:34 EDT
Hi, 
I don't reproduce your 'lock'. It looks like the Catalina connector is responsible for the time it takes to shutdown the server properly but the underemployment in itself is quite fast.
Using the -Dorg.apache.coyote.MAX_PAUSE_WAIT=5 I can reduce te timeout duration for the server to shutdown.
I'm on EAP 6.2.0 current dev status.
Comment 2 James Livingston 2013-09-10 23:10:30 EDT
Created attachment 796224 [details]
reproducer

Attached is the reproducer from the case.

1) Deploy it to EAP 6 and start the server
2) Go to http://localhost:8080/casesupport/ and click the submit button
3) trigger a redeployment of the app (e.g. touch standalone/deploy/00922614.war)
4) Look at the stack output which includes:

--
Full thread dump OpenJDK 64-Bit Server VM (24.0-b56 mixed mode):

"ServerService Thread Pool -- 57" prio=10 tid=0x00007fabcc1d4000 nid=0x12c1 waiting on condition [0x00007fac11b72000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ed133320> (a java.util.concurrent.Semaphore$FairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
	at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:416)
	at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3916)
	- locked <0x00000000eb436a40> (a org.apache.catalina.core.StandardContext)
	at org.jboss.as.web.deployment.WebDeploymentService.doStop(WebDeploymentService.java:171)
	at org.jboss.as.web.deployment.WebDeploymentService.access$100(WebDeploymentService.java:60)
	at org.jboss.as.web.deployment.WebDeploymentService$2.run(WebDeploymentService.java:113)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

   Locked ownable synchronizers:
	- <0x00000000e7632c40> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"http-/127.0.0.1:8080-1" daemon prio=10 tid=0x00007fac000ed000 nid=0x12b2 waiting on condition [0x00007fac110c9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at com.redhat.gss.servlet.CaseSupportReproduceServlet.doPost(CaseSupportReproduceServlet.java:35)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
	at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:134)
	at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99)
	at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92)
	at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64)
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
	at java.lang.Thread.run(Thread.java:724)

   Locked ownable synchronizers:
	- None

...
--
Comment 3 Emmanuel Hugonnet (ehsavoie) 2013-09-13 08:52:56 EDT
The missing piece of info was that you have to have a web.xml with the distribuable element set.
EAP PR : https://github.com/jbossas/jboss-eap/pull/403
Comment 7 Radim Hatlapatka 2013-11-11 08:38:19 EST
This issue is still valid even for the 6.2.0.CR1 preview bits.

Now when you start the server with -Djboss.web.session.manager.stop.timeout=15000 and tries to undeploy the application it shows after 15s [1]. I still see in deployments directory for quite some time isundeploying for the web app

After this message I am trying to shutdown the EAP server and it still takes about 5 minutes


[1]
13:39:31,227 WARN  [org.jboss.as.web.session] (ServerService Thread Pool -- 51) JBAS018228: Shutting down the session manager while there are still requests being processed after waiting for 15000 milliseconds
13:40:01,274 WARN  [org.infinispan.transaction.TransactionTable] (ServerService Thread Pool -- 58) ISPN000100: Stopping, but there are 1 local transactions and 0 remote transactions that did not finish in time.
Comment 8 Radim Hatlapatka 2013-11-11 08:40:39 EST
Created attachment 822422 [details]
Jstack when shutting down the EAP server after reproducer web app being undeployed
Comment 9 Radim Hatlapatka 2013-11-11 08:42:48 EST
Created attachment 822423 [details]
server jstack when undeployment of distributable app should be done

jstack output after the undeployment shows the warn message regarding stopping session
Comment 10 Radim Hatlapatka 2013-11-11 09:18:38 EST
I looked a little more deeply and the duration shutdown process is caused by Catalina connector as was already written in Comment #1. The time can be then reduced using -Dorg.apache.coyote.MAX_PAUSE_WAIT=5 as was suggested.

The undeployment works with possibility to specify undeployment timeout via -Djboss.web.session.manager.stop.timeout=15000 

=> verified
Comment 11 Radim Hatlapatka 2013-11-11 09:19:09 EST
I looked a little more deeply and the duration shutdown process is caused by Catalina connector as was already written in Comment #1. The time can be then reduced using -Dorg.apache.coyote.MAX_PAUSE_WAIT=5 as was suggested.

The undeployment works with possibility to specify undeployment timeout via -Djboss.web.session.manager.stop.timeout=15000 

=> verified using the 6.2.0.CR1 preview bits.

Note You need to log in before you can comment on or make changes to this bug.