Bug 1175126

Summary: [GSS](6.4.z) Deadlock on org.xnio.AbstractIoFuture lock
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Osamu Nagano <onagano>
Component: RemotingAssignee: Enrique Gonzalez Martinez <egonzale>
Status: CLOSED CURRENTRELEASE QA Contact: Jitka Kozana <jkudrnac>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.2.3CC: bbaranow, bmaxwell, cdewolf, egonzale, emartins, jawilson, lthon
Target Milestone: CR2   
Target Release: EAP 6.4.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-17 09:58:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1198685, 1207953    
Attachments:
Description Flags
TestCase none

Description Osamu Nagano 2014-12-17 08:46:49 UTC 
Description of problem:
Deadlock happened when closing InitialContexts on server shutdonw.

~~~
"pool-86-thread-193":
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:299)
	- waiting to lock <0x00007f3bd4c8b690> (a java.lang.Object)
	at org.jboss.remoting3.remote.RemoteConnectionProvider$5.handleCancelled(RemoteConnectionProvider.java:220)
	at org.jboss.remoting3.remote.RemoteConnectionProvider$5.handleCancelled(RemoteConnectionProvider.java:217)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
	at org.xnio.AbstractIoFuture$1.run(AbstractIoFuture.java:200)
	at org.xnio.IoUtils$2.execute(IoUtils.java:72)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:343)
	at org.xnio.AbstractIoFuture.runAllNotifiers(AbstractIoFuture.java:222)
	at org.xnio.AbstractIoFuture.setCancelled(AbstractIoFuture.java:280)
	- locked <0x00007f3bd4c8b768> (a java.lang.Object)
	at org.xnio.FutureResult.setCancelled(FutureResult.java:102)
	at org.jboss.remoting3.remote.RemoteConnectionProvider$2.cancel(RemoteConnectionProvider.java:158)
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:306)
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:39)
	at org.jboss.remoting3.remote.RemoteConnectionProvider.closeAction(RemoteConnectionProvider.java:246)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)
	at org.jboss.remoting3.EndpointImpl.closeAction(EndpointImpl.java:204)
	- locked <0x00007f3bd9a261c8> (a java.lang.Object)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:153)
	at org.jboss.naming.remote.client.EndpointCache.release(EndpointCache.java:61)
	- locked <0x00007f3b516ea5e0> (a org.jboss.naming.remote.client.EndpointCache)
	at org.jboss.naming.remote.client.EndpointCache$EndpointWrapper.close(EndpointCache.java:177)
	at org.jboss.naming.remote.client.InitialContextFactory$1.close(InitialContextFactory.java:233)
	at org.jboss.naming.remote.client.RemoteContext.close(RemoteContext.java:191)
	at javax.naming.InitialContext.close(InitialContext.java:550)
	at javax.naming.InitialContext.close(InitialContext.java:550)
	at ...(snip)...
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

 "Remoting "10.9.64.3:4447-0" read-1":
	at org.xnio.AbstractIoFuture.setException(AbstractIoFuture.java:236)
	- waiting to lock <0x00007f3bd4c8b768> (a java.lang.Object)
	at org.xnio.FutureResult.setException(FutureResult.java:93)
	at org.jboss.remoting3.remote.RemoteConnectionProvider$4.handleFailed(RemoteConnectionProvider.java:209)
	at org.jboss.remoting3.remote.RemoteConnectionProvider$4.handleFailed(RemoteConnectionProvider.java:207)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:219)
	at org.xnio.AbstractIoFuture$1.run(AbstractIoFuture.java:200)
	at org.xnio.IoUtils$2.execute(IoUtils.java:72)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:343)
	at org.xnio.AbstractIoFuture.runAllNotifiers(AbstractIoFuture.java:222)
	at org.xnio.AbstractIoFuture.setException(AbstractIoFuture.java:240)
	- locked <0x00007f3bd4c8b690> (a java.lang.Object)
	at org.xnio.FutureResult.setException(FutureResult.java:93)
	at org.xnio.nio.NioXnioWorker$1.handleEvent(NioXnioWorker.java:343)
	at org.xnio.nio.NioXnioWorker$1.handleEvent(NioXnioWorker.java:325)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
	at org.xnio.nio.NioHandle.run(NioHandle.java:90)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)
~~~


Version-Release number of selected component (if applicable):
XNIO 3.0.9.GA
Comment 1 Osamu Nagano 2014-12-17 08:50:12 UTC 
I entered XNIO-241 to backport the fix into XNIO 3.0.x.  Then we need to upgrade XNIO to include the fix into EAP, just like Bug 1144598.
Comment 2 Enrique Gonzalez Martinez 2015-03-04 15:23:06 UTC 
PR sent/merged in xnio:
3.0.x: https://github.com/xnio/xnio/pull/79
No upstream required

it requires an upgrade to xnio 3.0.14
Comment 3 Enrique Gonzalez Martinez 2015-04-16 11:04:28 UTC 
Created attachment 1015134 [details]
TestCase

I cannot provide a integration unit test case because is quite difficult to create a byteman script for this case.

I'm providing a junit test based in byteman to reproduce always the deadlock with the current remoting 3.3.4 and xnio 3.0.13.
Comment 4 Enrique Gonzalez Martinez 2015-04-16 11:09:44 UTC 
PM flag reset. Not sure why bz did that.
Comment 5 Rostislav Svoboda 2015-04-21 11:28:36 UTC 
You changed component ==> BZ bot resets all acks :)

As Jimmy already acked it (2015-03-13) I expect he will put pm_ack back to +. Granting qa_ack.
Comment 6 Rostislav Svoboda 2015-04-23 12:52:23 UTC 
Re-acking after "qa_ack flag reset for and jboss-eap-6.4.0 and jboss-eap-6.4.z items" cleanup.
Comment 7 Ladislav Thon 2015-05-18 07:32:32 UTC 
Nice reproducer, thanks!

Verified with EAP 6.4.1.CP.CR2 (or actually, with component version XNIO 3.0.14.GA-redhat-1).
Comment 8 Petr Penicka 2017-01-17 09:58:46 UTC 
Retroactively bulk-closing issues from released EAP 6.4 cummulative patches.
Comment 9 Petr Penicka 2017-01-17 10:00:00 UTC 
Retroactively bulk-closing issues from released EAP 6.4 cummulative patches.