Bug 1151960

Description of problem:

When CLI is connected to EAP 6.4.0.DR4 instance which is interrupted in the middle of CLI <-> EAP communication, CLI exhibits a deadlock condition. This is reproducible deadlock, for steps see below:

The stack trace of the relevant threads is as follows:
Found one Java-level deadlock:
=============================
"Remoting "cli-client" task-4":
  waiting to lock monitor 0x00007fb77c004958 (object 0x00000007c5165d00, a java.util.ArrayDeque),
  which is held by "Remoting "cli-client" read-1"
"Remoting "cli-client" read-1":
  waiting to lock monitor 0x00007fb758002998 (object 0x00000007c3191958, a java.lang.String),
  which is held by "main"
"main":
  waiting to lock monitor 0x00007fb77c004958 (object 0x00000007c5165d00, a java.util.ArrayDeque),
  which is held by "Remoting "cli-client" read-1"

Java stack information for the threads listed above:
===================================================
"Remoting "cli-client" task-4":
        at org.jboss.remoting3.remote.RemoteConnectionHandler.closePendingChannels(RemoteConnectionHandler.java:383)
        - waiting to lock <0x00000007c5165d00> (a java.util.ArrayDeque)
        at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:108)
        at org.jboss.remoting3.remote.RemoteReadListener$1$1.run(RemoteReadListener.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
"Remoting "cli-client" read-1":
        at org.jboss.as.cli.impl.CLIModelControllerClient$ChannelCloseHandler.handleClose(CLIModelControllerClient.java:286)
        - waiting to lock <0x00000007c3191958> (a java.lang.String)
        at org.jboss.as.cli.impl.CLIModelControllerClient$ChannelCloseHandler.handleClose(CLIModelControllerClient.java:269)
        at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)
        at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:532)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)
        at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:392)
        - locked <0x00000007c5165d00> (a java.util.ArrayDeque)
        at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:109)
        at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:81)
        - locked <0x00000007c5165d00> (a java.util.ArrayDeque)
        at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
        at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
        at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
        at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
        at org.xnio.ssl.JsseConnectedSslStreamChannel.handleReadable(JsseConnectedSslStreamChannel.java:183)
        at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
        at org.xnio.nio.NioHandle.run(NioHandle.java:90)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)
"main":
        at org.jboss.remoting3.remote.RemoteConnectionChannel.receiveMessage(RemoteConnectionChannel.java:361)
        - waiting to lock <0x00000007c5165d00> (a java.util.ArrayDeque)
        at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.connectionOpened(FutureManagementChannel.java:221)
        at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:78)
        - locked <0x00000007c44b6328> (a org.jboss.as.protocol.ProtocolConnectionManager)
        at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:208)
        at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:160)
        - locked <0x00000007c3191958> (a java.lang.String)
        at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:120)
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:123)
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:98)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
        at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:998)
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:850)
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:826)
        at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:282)
        at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:250)
        at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jboss.modules.Module.run(Module.java:312)
        at org.jboss.modules.Main.main(Main.java:460)


Version-Release number of selected component (if applicable):
EAP: 6.4.0.DR4 (xnio: 3.0.11.GA-redhat-2, remoting: 3.3.3.Final-redhat-1)

How reproducible:
Everytime

Steps to Reproduce:
1. Prepare a running EAP instance with secured management port - optionally in VM
2. Execute export JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=y"
3. Execute bin/jboss-cli.sh --connect --controller=$EAP_IP --user=admin --password=password ':read-resource'
4. From another terminal, execute jdb -attach localhost:8787
5. In JDB: 
5.a. Create breakpoint: "stop in org.xnio.channels.FramedMessageChannel.receive(java.nio.ByteBuffer)"
5.b. Resume all threads: "resume"
5.c. [Execute eight times] Wait until breakpoint is hit and execute "resume". Either set timeout or be fast so that timeout does not occur first
6. In VM: "kill -9 $EAP_PID"
7. In JDB:
8.a. Remove breakpoint: "clear org.xnio.channels.FramedMessageChannel.receive(java.nio.ByteBuffer)"
8.b. Resume all threads: "resume"
9. Now dump the stack trace of jboss-cli.sh: "kill -3 $JBOSS_CLI_PID"

Actual results:
Deadlock is found

Expected results:
No deadlock found