Bug 1426471

Summary:	[GSS] (6.4.z) deadlock in org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible
Product:	[JBoss] JBoss Enterprise Application Platform 6	Reporter:	Masanobu Hatanaka <mhatanak>
Component:	Web	Assignee:	Radovan Netuka <rnetuka>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Radim Hatlapatka <rhatlapa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	6.4.12	CC:	aogburn, bmaxwell, fnasser, jason.greene, mcada, rmaucher, rnetuka
Target Milestone:	CR1
Target Release:	EAP 6.4.15
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-05-19 08:08:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1419353, 1427196, 1427753

Description Masanobu Hatanaka 2017-02-24 01:51:22 UTC

Description of problem:
The deadlock is defected in WebSocket on EAP 6.4.12. 

Found one Java-level deadlock:
=============================
"WebSocketServer-/asahi/i-smart-54":
  waiting to lock monitor 0x00007fb16819df98 (object 0x00000004127d7d00, a java.lang.Object),
  which is held by "WebSocketServer-/asahi/i-smart-53"
"WebSocketServer-/asahi/i-smart-53":
  waiting to lock monitor 0x00007fb26c1f9e68 (object 0x000000040631dd00, a org.apache.coyote.http11.Http11NioProcessor),
  which is held by "HTTP-1831"
"HTTP-1831":
  waiting to lock monitor 0x00007fb16819df98 (object 0x00000004127d7d00, a java.lang.Object),
  which is held by "WebSocketServer-/asahi/i-smart-53"

Java stack information for the threads listed above:
===================================================
"WebSocketServer-/asahi/i-smart-54":
	at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible(WsRemoteEndpointImplServer.java:81)
	- waiting to lock <0x00000004127d7d00> (a java.lang.Object)
	at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:75)
	at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:445)
	at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.endMessage(WsRemoteEndpointImplBase.java:363)
	at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$EndMessageHandler.onResult(WsRemoteEndpointImplBase.java:483)
	at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer$OnResultRunnable.run(WsRemoteEndpointImplServer.java:247)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
"WebSocketServer-/asahi/i-smart-53":
	at org.apache.coyote.http11.Http11NioProcessor.writeEvent(Http11NioProcessor.java:760)
	- waiting to lock <0x000000040631dd00> (a org.apache.coyote.http11.Http11NioProcessor)
	at org.apache.coyote.http11.Http11NioProcessor.action(Http11NioProcessor.java:859)
	at org.apache.coyote.Response.action(Response.java:188)
	at org.apache.catalina.connector.OutputBuffer.lastWrite(OutputBuffer.java:392)
	at org.apache.catalina.connector.CoyoteOutputStream.isReady(CoyoteOutputStream.java:112)
	at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible(WsRemoteEndpointImplServer.java:89)
	- locked <0x00000004127d7d00> (a java.lang.Object)
	at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:75)
	at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:445)
	at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessage(WsRemoteEndpointImplBase.java:335)
	at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$TextMessageSendHandler.write(WsRemoteEndpointImplBase.java:741)
	- locked <0x00000004127d7d00> (a java.lang.Object)
	at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$TextMessageSendHandler.onResult(WsRemoteEndpointImplBase.java:757)
	at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.endMessage(WsRemoteEndpointImplBase.java:371)
	at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$EndMessageHandler.onResult(WsRemoteEndpointImplBase.java:483)
	at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer$OnResultRunnable.run(WsRemoteEndpointImplServer.java:245)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
"HTTP-1831":
	at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible(WsRemoteEndpointImplServer.java:81)
	- waiting to lock <0x00000004127d7d00> (a java.lang.Object)
	at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsWriteListener.onWritePossible(WsHttpUpgradeHandler.java:243)
	at org.apache.catalina.core.StandardWrapperValve.async(StandardWrapperValve.java:605)
	at org.apache.catalina.core.StandardWrapperValve.event(StandardWrapperValve.java:350)
	at org.apache.catalina.core.StandardContextValve.event(StandardContextValve.java:171)
	at org.apache.catalina.valves.ValveBase.event(ValveBase.java:185)
	at org.apache.catalina.core.StandardHostValve.event(StandardHostValve.java:252)
	at org.apache.catalina.valves.ValveBase.event(ValveBase.java:185)
	at org.apache.catalina.core.StandardEngineValve.event(StandardEngineValve.java:121)
	at org.apache.catalina.connector.CoyoteAdapter.event(CoyoteAdapter.java:228)
	at org.apache.coyote.http11.Http11NioProcessor.event(Http11NioProcessor.java:232)
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.event(Http11NioProtocol.java:819)
	- locked <0x000000040631dd00> (a org.apache.coyote.http11.Http11NioProcessor)
	at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:940)
	- locked <0x00000004127c9560> (a java.lang.Object)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Found 1 deadlock.


How reproducible:
No reproducible steps but the customer told that this deadlock can be found a few times per day.

I thought this was similar but no related and it seems that Bug#1410869 would resolve the issue.
[GSS](6.4.z) Deadlock in WsRemoteEndpointImplServer.onWritePossible
https://bugzilla.redhat.com/show_bug.cgi?id=1299057

Comment 4 Michael Cada 2017-04-26 12:53:49 UTC

As we do not have reproducer, I only verified that commit which is removing unnecessary synchronization is present. No regressions were found, marking as verified. EAP 6.4.15.CP.CR2 was used for verification.

Comment 5 Petr Penicka 2017-05-19 08:08:12 UTC

Released on May 18 as part of EAP 6.4.15.