Bug 1098879 - EJB client failed initially if a cluster should used for EJB invocation with Could not create a connection for cluster node ClusterNode{} -> Operation failed with status WAITING
Summary: EJB client failed initially if a cluster should used for EJB invocation with ...
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: EJB, Remoting
Version: 6.1.0
Hardware: Unspecified
OS: Windows
Target Milestone: CR1
: EAP 6.3.0
Assignee: David M. Lloyd
QA Contact: Jan Martiska
Russell Dickenson
Depends On:
Blocks: eap62-cp04-blockers 1099638 1104666 1109394
TreeView+ depends on / blocked
Reported: 2014-05-19 06:34 UTC by wfink
Modified: 2018-12-06 16:32 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
In previous versions of JBoss EAP 6, there was an issue with EJB clients connecting to a cluster if more than one cluster node was specified for the initial connection. This was an issue specifically on the Windows platform, causing the first EJB invocation to intermittently fail and was caused by wrong thread synchronization. This issue has been addressed in this release and no longer presents.
Clone Of:
: 1104666 1109394 (view as bug list)
Last Closed: 2014-06-28 15:32:07 UTC
Type: Bug

Attachments (Terms of Use)
Client logfile and a wireshark TCP file until the client initialisation (58.40 KB, application/zip)
2014-05-19 06:34 UTC, wfink
no flags Details

Description wfink 2014-05-19 06:34:18 UTC
Created attachment 897023 [details]
Client logfile and a wireshark TCP file until the client initialisation

A correct configured EJB client run into problems if there are more (here 4) cluster members which are also listed in the remote.connections properties within the jboss-ejb-client.properties.

After the initial connection the cluster view is provided but the connect failed with the following message:

2014-05-02 09:12:02,919 SERVER INFO  !==ejb-client-cluster-node-connection-creation-3-thread-3==!  RemotingConnectionClusterNodeManager: Could not create a connection for cluster node ClusterNode{clus
terName='ejb', nodeName='prod3', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='', destinationPort=4547}], resolvedDestinat
ion=[Destination address=, destination port=4547]} in cluster ejb
java.lang.RuntimeException: Operation failed with status WAITING
        at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93)
        at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:75)
        at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
        at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77)
        at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:406)
        at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:380)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

This is a sporadical problem but reproducable if there are more members within the cluster.
If there is no cluster (app not clustered or no cluster), or the client only reference one of the members for the initial connection, this Exception will not happen.
Also it looks not reproducable on Linux at this moment.

After this initial problems the connections are working as expected.

Steps to Reproduce:
1. create >2 server as cluster on Windows (same host works)
2. use a client with all members in the connections list and a valid cluster configuration

Comment 1 wfink 2014-05-19 06:42:01 UTC
might be related

Comment 4 David M. Lloyd 2014-05-20 17:37:04 UTC
OK I have figured out the root cause.  The XNIO scaling task thread pool has a problem where tasks can get "lost" under some circumstances.  This had previously been fixed upstream, so I backported the fix by hand to the XNIO 3.0 branch, and tagged 3.0.10.GA.  This has been released, and I will submit a component upgrade request for EAP 6.3; this release can also be used for patches.

Comment 6 Jan Martiska 2014-06-20 07:23:44 UTC
Verified in EAP 6.3.0.ER7.

Comment 7 Jan Martiska 2014-06-20 07:24:36 UTC
Ups, sorry, I meant 6.2.4.CR1 :)

Comment 8 wfink 2014-06-23 14:11:52 UTC
hope that explain the behaviour before fixing

Note You need to log in before you can comment on or make changes to this bug.