Bug 1151960 - Deadlock when CLI accesses suddenly inaccessible EAP instance
Summary: Deadlock when CLI accesses suddenly inaccessible EAP instance
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: CLI
Version: 6.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: DR11
: EAP 6.4.0
Assignee: Alexey Loubyansky
QA Contact: Pavel Slavicek
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-13 07:55 UTC by Hynek Mlnarik
Modified: 2019-08-19 12:42 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-19 12:42:45 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker WFCORE-202 0 Major Resolved Deadlock when shutdown Wildfly server during CLI client connection 2016-11-23 14:26:36 UTC

Description Hynek Mlnarik 2014-10-13 07:55:18 UTC
Description of problem:

When CLI is connected to EAP 6.4.0.DR4 instance which is interrupted in the middle of CLI <-> EAP communication, CLI exhibits a deadlock condition. This is reproducible deadlock, for steps see below:

The stack trace of the relevant threads is as follows:
Found one Java-level deadlock:
=============================
"Remoting "cli-client" task-4":
  waiting to lock monitor 0x00007fb77c004958 (object 0x00000007c5165d00, a java.util.ArrayDeque),
  which is held by "Remoting "cli-client" read-1"
"Remoting "cli-client" read-1":
  waiting to lock monitor 0x00007fb758002998 (object 0x00000007c3191958, a java.lang.String),
  which is held by "main"
"main":
  waiting to lock monitor 0x00007fb77c004958 (object 0x00000007c5165d00, a java.util.ArrayDeque),
  which is held by "Remoting "cli-client" read-1"

Java stack information for the threads listed above:
===================================================
"Remoting "cli-client" task-4":
        at org.jboss.remoting3.remote.RemoteConnectionHandler.closePendingChannels(RemoteConnectionHandler.java:383)
        - waiting to lock <0x00000007c5165d00> (a java.util.ArrayDeque)
        at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:108)
        at org.jboss.remoting3.remote.RemoteReadListener$1$1.run(RemoteReadListener.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
"Remoting "cli-client" read-1":
        at org.jboss.as.cli.impl.CLIModelControllerClient$ChannelCloseHandler.handleClose(CLIModelControllerClient.java:286)
        - waiting to lock <0x00000007c3191958> (a java.lang.String)
        at org.jboss.as.cli.impl.CLIModelControllerClient$ChannelCloseHandler.handleClose(CLIModelControllerClient.java:269)
        at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)
        at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:532)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)
        at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:392)
        - locked <0x00000007c5165d00> (a java.util.ArrayDeque)
        at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:109)
        at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:81)
        - locked <0x00000007c5165d00> (a java.util.ArrayDeque)
        at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
        at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
        at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
        at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
        at org.xnio.ssl.JsseConnectedSslStreamChannel.handleReadable(JsseConnectedSslStreamChannel.java:183)
        at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
        at org.xnio.nio.NioHandle.run(NioHandle.java:90)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)
"main":
        at org.jboss.remoting3.remote.RemoteConnectionChannel.receiveMessage(RemoteConnectionChannel.java:361)
        - waiting to lock <0x00000007c5165d00> (a java.util.ArrayDeque)
        at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.connectionOpened(FutureManagementChannel.java:221)
        at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:78)
        - locked <0x00000007c44b6328> (a org.jboss.as.protocol.ProtocolConnectionManager)
        at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:208)
        at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:160)
        - locked <0x00000007c3191958> (a java.lang.String)
        at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:120)
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:123)
        at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:98)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
        at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
        at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:998)
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:850)
        at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:826)
        at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:282)
        at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:250)
        at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jboss.modules.Module.run(Module.java:312)
        at org.jboss.modules.Main.main(Main.java:460)


Version-Release number of selected component (if applicable):
EAP: 6.4.0.DR4 (xnio: 3.0.11.GA-redhat-2, remoting: 3.3.3.Final-redhat-1)

How reproducible:
Everytime

Steps to Reproduce:
1. Prepare a running EAP instance with secured management port - optionally in VM
2. Execute export JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=y"
3. Execute bin/jboss-cli.sh --connect --controller=$EAP_IP --user=admin --password=password ':read-resource'
4. From another terminal, execute jdb -attach localhost:8787
5. In JDB: 
5.a. Create breakpoint: "stop in org.xnio.channels.FramedMessageChannel.receive(java.nio.ByteBuffer)"
5.b. Resume all threads: "resume"
5.c. [Execute eight times] Wait until breakpoint is hit and execute "resume". Either set timeout or be fast so that timeout does not occur first
6. In VM: "kill -9 $EAP_PID"
7. In JDB:
8.a. Remove breakpoint: "clear org.xnio.channels.FramedMessageChannel.receive(java.nio.ByteBuffer)"
8.b. Resume all threads: "resume"
9. Now dump the stack trace of jboss-cli.sh: "kill -3 $JBOSS_CLI_PID"

Actual results:
Deadlock is found

Expected results:
No deadlock found

Comment 2 Chao Wang 2014-11-17 00:11:10 UTC
6.x backport PR https://github.com/jbossas/jboss-eap/pull/1971

Comment 3 Hynek Mlnarik 2014-11-25 15:36:47 UTC
Verified in EAP 6.4.0.DR11. No deadlock found in the critical areas of CLI.


Note You need to log in before you can comment on or make changes to this bug.