Created attachment 897023 [details] Client logfile and a wireshark TCP file until the client initialisation A correct configured EJB client run into problems if there are more (here 4) cluster members which are also listed in the remote.connections properties within the jboss-ejb-client.properties. After the initial connection the cluster view is provided but the connect failed with the following message: 2014-05-02 09:12:02,919 SERVER INFO !==ejb-client-cluster-node-connection-creation-3-thread-3==! RemotingConnectionClusterNodeManager: Could not create a connection for cluster node ClusterNode{clus terName='ejb', nodeName='prod3', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='10.0.22.43', destinationPort=4547}], resolvedDestinat ion=[Destination address=10.0.22.43, destination port=4547]} in cluster ejb java.lang.RuntimeException: Operation failed with status WAITING at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93) at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:75) at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51) at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:406) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) This is a sporadical problem but reproducable if there are more members within the cluster. If there is no cluster (app not clustered or no cluster), or the client only reference one of the members for the initial connection, this Exception will not happen. Also it looks not reproducable on Linux at this moment. After this initial problems the connections are working as expected. Steps to Reproduce: 1. create >2 server as cluster on Windows (same host works) 2. use a client with all members in the connections list and a valid cluster configuration
https://issues.jboss.org/browse/JBJMX-117 might be related
OK I have figured out the root cause. The XNIO scaling task thread pool has a problem where tasks can get "lost" under some circumstances. This had previously been fixed upstream, so I backported the fix by hand to the XNIO 3.0 branch, and tagged 3.0.10.GA. This has been released, and I will submit a component upgrade request for EAP 6.3; this release can also be used for patches.
Verified in EAP 6.3.0.ER7.
Ups, sorry, I meant 6.2.4.CR1 :)
Jan, hope that explain the behaviour before fixing