Bug 1098879 – EJB client failed initially if a cluster should used for EJB invocation with Could not create a connection for cluster node ClusterNode{} -> Operation failed with status WAITING

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1098879 - EJB client failed initially if a cluster should used for EJB invocation with Could not create a connection for cluster node ClusterNode{} -> Operation failed with status WAITING

Summary:	EJB client failed initially if a cluster should used for EJB invocation with ...

Status:	CLOSED CURRENTRELEASE

Aliases:	None

Product:	JBoss Enterprise Application Platform 6
Classification:	JBoss
Component:	EJB, Remoting (Show other bugs)
Sub Component:
Version:	6.1.0
Hardware:	Unspecified Windows

Priority	unspecified Severity unspecified
Target Milestone:	CR1
Target Release:	EAP 6.3.0
Assigned To:	David M. Lloyd
QA Contact:	Jan Martiska
Docs Contact:	Russell Dickenson

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:	1109394 eap62-cp04-blockers 1099638 1104666
	Show dependency tree / graph

Reported:	2014-05-19 02:34 EDT by wfink
Modified:	2014-07-03 23:11 EDT (History)
CC List:	3 users (show)

See Also:
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	In previous versions of JBoss EAP 6, there was an issue with EJB clients connecting to a cluster if more than one cluster node was specified for the initial connection. This was an issue specifically on the Windows platform, causing the first EJB invocation to intermittently fail and was caused by wrong thread synchronization. This issue has been addressed in this release and no longer presents.
Story Points:	---
Clone Of:
Clones:	1104666 1109394 (view as bug list)
Environment:
Last Closed:	2014-06-28 11:32:07 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Client logfile and a wireshark TCP file until the client initialisation (58.40 KB, application/zip) 2014-05-19 02:34 EDT, wfink	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

Groups: None (edit)

Description wfink 2014-05-19 02:34:18 EDT

Created attachment 897023 [details]
Client logfile and a wireshark TCP file until the client initialisation

A correct configured EJB client run into problems if there are more (here 4) cluster members which are also listed in the remote.connections properties within the jboss-ejb-client.properties.

After the initial connection the cluster view is provided but the connect failed with the following message:

2014-05-02 09:12:02,919 SERVER INFO  !==ejb-client-cluster-node-connection-creation-3-thread-3==!  RemotingConnectionClusterNodeManager: Could not create a connection for cluster node ClusterNode{clus
terName='ejb', nodeName='prod3', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='10.0.22.43', destinationPort=4547}], resolvedDestinat
ion=[Destination address=10.0.22.43, destination port=4547]} in cluster ejb
java.lang.RuntimeException: Operation failed with status WAITING
        at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93)
        at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:75)
        at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
        at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77)
        at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:406)
        at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:380)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

This is a sporadical problem but reproducable if there are more members within the cluster.
If there is no cluster (app not clustered or no cluster), or the client only reference one of the members for the initial connection, this Exception will not happen.
Also it looks not reproducable on Linux at this moment.

After this initial problems the connections are working as expected.


Steps to Reproduce:
1. create >2 server as cluster on Windows (same host works)
2. use a client with all members in the connections list and a valid cluster configuration

Comment 1 wfink 2014-05-19 02:42:01 EDT

https://issues.jboss.org/browse/JBJMX-117
might be related

Comment 4 David M. Lloyd 2014-05-20 13:37:04 EDT

OK I have figured out the root cause.  The XNIO scaling task thread pool has a problem where tasks can get "lost" under some circumstances.  This had previously been fixed upstream, so I backported the fix by hand to the XNIO 3.0 branch, and tagged 3.0.10.GA.  This has been released, and I will submit a component upgrade request for EAP 6.3; this release can also be used for patches.

Comment 6 Jan Martiska 2014-06-20 03:23:44 EDT

Verified in EAP 6.3.0.ER7.

Comment 7 Jan Martiska 2014-06-20 03:24:36 EDT

Ups, sorry, I meant 6.2.4.CR1 :)

Comment 8 wfink 2014-06-23 10:11:52 EDT

Jan,
hope that explain the behaviour before fixing

Note You need to log in before you can comment on or make changes to this bug.