project_key: JBPAPP6 When node bound on IPv6 address is switch down and then added to cluster again then the mentioned exception is written down to console of the remote client. This has no affect on function of the client and it's not a exception from cluster. The replication works and the client forwards requests to active node. But the exception is confusing. Steps to reproduce should be following: 1) Start server on IPv6 address and deploy a stateful bean on it. {code} bin/standalone.sh -c=standalone-ha.xml -Djboss.node.name=server1 -Djboss.bind.address=10::11 -Djboss.bind.address.management=10::11 -Djboss.bind.address.unsecure=10::11 -Djboss.default.multicast.address=FF0E::1 -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true -Djboss.socket.binding.port-offset=0 {code} 2) Run the remote client and search for the SFSB 3) Start second server and deploy the SFSB on it. The cluster from the both servers is created. {code} bin/standalone.sh -c=standalone-ha.xml -Djboss.node.name=server2 -Djboss.bind.address=10::12 -Djboss.bind.address.management=10::12 -Djboss.bind.address.unsecure=10::12 -Djboss.default.multicast.address=FF0E::1 -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true -Djboss.socket.binding.port-offset=0 {code} 4) Undeploy the bean from the first server. Cluster is now consisted just from one node which is the second server. 5) Deploy the bean to the first server Note: Servers and client are placed on their own machines. On that second deploy the client will freeze for a while and then the following exception will be printed out: {code} Jun 5, 2012 4:28:23 PM org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager getEJBReceiver INFO: Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='1011server1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='10:0:0:0:0:0:0:11%2', destinationPort=4447}], resolvedDestination=[Destination address=[10:0:0:0:0:0:0:11%2], destination port=4447]} in cluster ejb java.lang.RuntimeException: Operation failed with status WAITING at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93) at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:104) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:406) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Jun 5, 2012 4:28:23 PM org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager getEJBReceiver INFO: Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='1011server1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='10:0:0:0:0:0:0:11%2', destinationPort=4447}], resolvedDestination=[Destination address=[10:0:0:0:0:0:0:11%2], destination port=4447]} in cluster ejb java.lang.RuntimeException: Operation failed with status WAITING at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93) at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:104) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:406) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) 1012server1: Hello [38] at Tue Jun 05 16:28:18 CEST 2012 Jun 5, 2012 4:28:23 PM org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager getEJBReceiver INFO: Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='1011server1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='10:0:0:0:0:0:0:11%2', destinationPort=4447}], resolvedDestination=[Destination address=[10:0:0:0:0:0:0:11%2], destination port=4447]} in cluster ejb java.lang.RuntimeException: Operation failed with status WAITING at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93) at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:104) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:406) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Jun 5, 2012 4:28:23 PM org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager getEJBReceiver INFO: Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='1011server1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='10:0:0:0:0:0:0:11%2', destinationPort=4447}], resolvedDestination=[Destination address=[10:0:0:0:0:0:0:11%2], destination port=4447]} in cluster ejb java.lang.RuntimeException: Operation failed with status WAITING at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93) at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:104) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:406) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) {code}
Link: Added: This issue is a dependency of JBPAPP-9320
Docs QE Status: Removed: NEW
We are seeing this issue in IPv4 environment too and with stateless beans: 04:26:56,624 INFO [org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager] (Runner - 1) Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='perf20', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='10.16.90.58', destinationPort=4447}], resolvedDestination=[Destination address=10.16.90.58, destination port=4447]} in cluster ejb java.lang.RuntimeException: Operation failed with status WAITING at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93) at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:91) at org.jboss.ejb.client.ClusterContext.getEJBReceiverContext(ClusterContext.java:163) at org.jboss.ejb.client.ClusterContext.getEJBReceiverContext(ClusterContext.java:89) at org.jboss.ejb.client.EJBClientContext.getClusterEJBReceiverContext(EJBClientContext.java:701) at org.jboss.ejb.client.EJBClientContext.getClusterEJBReceiverContext(EJBClientContext.java:693) at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:111) at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:181) at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136) at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121) at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104) at $Proxy0.getNodeNameOfService(Unknown Source) at org.jboss.smartfrog.clustering.ejb3.SSAccessProcessorFactoryImpl$SSAccessRequestProcessor.processRequest(SSAccessProcessorFactoryImpl.java:89) at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52) at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87) at java.lang.Thread.run(Thread.java:662)
Seen again with 6.1.0 DR4 testing.
Also affects EAP 6.1.0.ER1. Seen in https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/9/console-perf17/
Update from EAP 6.1.0.ER8 testing: still seeing this issue. Link to client logs: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/18/console-perf17/
Still seeing this with EAP 6.1.1.ER7. For example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/25/ https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-jvmkill-repl-async/30/
For the record, this affects RHQ 4.9 and the soon to be released firs tJON 3.2 BETA. See bug #1003797
Assigning jpai EJB issues to david.lloyd. Please re-assign to Cheng or others as needed.
Ipv4 issue seeming related in EAP 6.3.0.DR5 - see https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-AS-Testsuite/job/eap-60-as-testsuite-RHEL-matrix-openJDK7/lastCompletedBuild/jdk=openjdk-1.7.0-local,label_exp=RHEL6%26%26x86_64/testReport/org.jboss.as.test.clustering.cluster.ejb3.descriptor.disable/DisableClusteredTestCase(SYNC-tcp)/testSetup/
Most of the links are not working. The most interesting would be #11 but it's an invalid link. Cannot reproduce the problem following the steps. I tested 6.0, 6.1, 6.2 and 6.3 in a single computer doing the following: 1) create IP aliases, ie: ifconfig em1:0 192.168.1.150 up ifconfig em1:1 192.168.1.151 up 2) unzip the jboss version and create two copies of the standalone directory (i.e.: standaloneA, standaloneB) 3) for starting both instances I use: a) /PATH_TO_JBOSS/bin/standalone.sh -c=standalone-ha.xml -Djboss.node.name=server1 -Djboss.bind.address=192.168.1.150 -Djboss.bind.address.management=192.168.1.150 -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -Djboss.socket.binding.port-offset=0 -Djboss.server.base.dir=/PATH_TO_JBOSS/standaloneA/ b) PATH_TO_JBOSS/bin/standalone.sh -c=standalone-ha.xml -Djboss.node.name=server2 -Djboss.bind.address=192.168.1.151 -Djboss.bind.address.management=192.168.1.151 -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -Djboss.socket.binding.port-offset=0 -Djboss.server.base.dir=/PATH_TO_JBOSS/standaloneB/ I followed the steps described in the BZ but it's not working for me. I created an ejb with @clustered and I can see how the cluster is created. But whenever I remove the sfsb and redeploy again the clusters keep working ok. I'm not able to reproduce the problem. Regarding the stacktrace, the real exception is hidden. At that point the ejb-client had a bug that makes not possible to know when the connection fails. This was fixed in https://issues.jboss.org/browse/REM3-175 in 3.2.18GA. JBoss EAP 6.1 uses 3.2.16.GA, that means is not possible to know what really happened there.