Bug 1454564

Summary: [GSS] (6.4.z) JMX client hangs when closing an unresponsive connection
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Hiroki Daicho <hdaicho>
Component: RemotingAssignee: Petr Jurak <pjurak>
Status: CLOSED CURRENTRELEASE QA Contact: Jiří Bílek <jbilek>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.15CC: bmaxwell, david.lloyd, fgavrilo, hdaicho, jbilek, mmiura, pjurak, tanabe.yoshimasa
Target Milestone: CR1   
Target Release: EAP 6.4.17   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-06 13:31:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1449266, 1450092    
Attachments:
Description Flags
reproducer
none
thread dump none

Description Hiroki Daicho 2017-05-23 04:49:41 UTC 
Created attachment 1281315 [details]
reproducer

Description of problem:
When JMX client close the connection via remoting socket, the client waits for a long time by unresponsive connection/server with following stacktrace.

"jboss.remoting_jmx.App.main()" #11 prio=5 os_prio=0 tid=0x00007f36e91e6000 nid=0x56f6 in Object.wait() [0x00007f36cf0f2000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000772d7aad8> (a java.lang.Object)
	at java.lang.Object.wait(Object.java:502)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:190)
	- locked <0x0000000772d7aad8> (a java.lang.Object)
	at org.jboss.remotingjmx.RemotingConnector.safeClose(RemotingConnector.java:313)
	at org.jboss.remotingjmx.RemotingConnector.close(RemotingConnector.java:296)
	at jboss.remoting_jmx.App.main(App.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
	at java.lang.Thread.run(Thread.java:745)


Version-Release number of selected component (if applicable):
This issue occurs on EAP 6.4.5 (remorting 3.3.6) and later. (6.4.5 - 6.4.15)

How reproducible:
reproducer.zip attached. It includes readme.txt, pom.xml, and client application.

Steps to Reproduce:
1. 
EAP server start

2. 
A client start with the following command.
$ mvn clean compile exec:java -Djmx.service.url=service:jmx:remoting-jmx://host:port

This client will sleep before closing connection.
$ Sleeping 10 sec...

3. 
During client sleep, disconnect a network between the server and the client. 
(or suspend the server process with kill -sigstop <PID>, you can see similarly result.)


Additional info:
This issue is not reproduced on EAP6.4.0 - 6.4.4. Because there version WFCORE-112 fix is included.

After the fix, the close logic is changed by following fixes and the issue reproduced again.
https://github.com/jboss-remoting/jboss-remoting/commit/e48d4744a588e2064258989a7695aaa52366e296
https://github.com/jboss-remoting/jboss-remoting/commit/b5bcb477e66f88c2919e453e5864cbafaa66c620

The behaviour is discussing in the following JIRA, this bugzilla is for EAP6.
https://issues.jboss.org/browse/JBEAP-10038
Comment 1 Hiroki Daicho 2017-05-23 04:50:13 UTC 
Created attachment 1281316 [details]
thread dump
Comment 2 Hiroki Daicho 2017-05-23 06:28:51 UTC 
(In reply to Hiroki Daicho from comment #0)

> After the fix, the close logic is changed by following fixes and the issue
> reproduced again.
> https://github.com/jboss-remoting/jboss-remoting/commit/
> e48d4744a588e2064258989a7695aaa52366e296
> https://github.com/jboss-remoting/jboss-remoting/commit/
> b5bcb477e66f88c2919e453e5864cbafaa66c620

Sorry. It looks that commit id e48d47 is NOT related. because it was just reverted the previous commit id 055f0c4 for REM3-210. 

So, actually, I meant:

The close logic is changed by following fix and the issue reproduced again:
https://github.com/jboss-remoting/jboss-remoting/commit/b5bcb477e66f88c2919e453e5864cbafaa66c620
Comment 7 Jiří Bílek 2017-08-02 11:36:06 UTC 
Verified with EAP 6.4.17.CP.CR3
Comment 8 Petr Penicka 2017-09-06 13:31:17 UTC 
Released on 2017-09-05 as part of the EAP 6.4.17 release.