Bug 1248299

Summary: [GSS] (6.4.z) Deadlock in remoting during channel shutdown
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: James Livingston <jlivings>
Component: RemotingAssignee: James Livingston <jlivings>
Status: CLOSED DUPLICATE QA Contact: Jitka Kozana <jkudrnac>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.0CC: david.lloyd, joallen
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-07-30 04:28:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description James Livingston 2015-07-30 04:20:00 UTC
There is a deadlock risk in remoting which produces thread like:
"Remoting "host:MANAGEMENT" read-1" #149 prio=5 os_prio=0 tid=0x00007fbecc038800 nid=0x3f12 waiting for monitor entry [0x00007fbe136b0000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jboss.remoting3.remote.OutboundMessage.cancel(OutboundMessage.java:289)
	- waiting to lock <0x00000007ae29fc60> (a org.xnio.streams.BufferPipeOutputStream)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.closeMessages(RemoteConnectionChannel.java:560)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:542)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)
	at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:423)
	- locked <0x00000007ad9c3030> (a java.util.ArrayDeque)
	at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:114)
	at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:81)
	- locked <0x00000007ad9c3030> (a java.util.ArrayDeque)
	at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
	at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
	at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
	at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
	at org.xnio.ssl.JsseConnectedSslStreamChannel.handleReadable(JsseConnectedSslStreamChannel.java:183)
	at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
	at org.xnio.nio.NioHandle.run(NioHandle.java:90)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)


"Remoting "host:MANAGEMENT" task-8" #336 prio=5 os_prio=0 tid=0x00007fbdec01c800 nid=0x4105 waiting for monitor entry [0x00007fbcefd79000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:294)
	- waiting to lock <0x00000007ad9c3030> (a java.util.ArrayDeque)
	at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:122)
	at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:154)
	at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:126)
	at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:114)
	at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:143)
	- locked <0x00000007ae29fc60> (a org.xnio.streams.BufferPipeOutputStream)
	at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:135)
	at org.jboss.remoting3.remote.OutboundMessage.flush(OutboundMessage.java:277)
	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
	at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
	at org.jboss.remotingjmx.DelegatingRemotingConnectorServer.writeVersionHeader(DelegatingRemotingConnectorServer.java:208)
	at org.jboss.remotingjmx.DelegatingRemotingConnectorServer.access$200(DelegatingRemotingConnectorServer.java:60)
	at org.jboss.remotingjmx.DelegatingRemotingConnectorServer$ChannelOpenListener.channelOpened(DelegatingRemotingConnectorServer.java:288)
	at org.jboss.remoting3.spi.SpiUtils$ServiceOpenTask.run(SpiUtils.java:126)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)



This was reported upstream in WFLY-4113, and resolved by upgrading to remoting 4.0.7 which incorporated the commit https://github.com/jboss-remoting/jboss-remoting/commit/afd95bbf3b27af677fc74908515972b4ddb4452a.

To prevent this deadlock, we need to backport the commit to remoting 3.3.x and pull in a fixed version.

Comment 1 James Livingston 2015-07-30 04:28:48 UTC
And never mind, this is already fixed in 6.4.2 due to the remoting 3.3.5 upgrade, but I didn't manage to find bug 1215714.

*** This bug has been marked as a duplicate of bug 1215714 ***