Description of problem: Test org.jboss.as.test.clustering.cluster.ejb3.stateless.RemoteStatelessFailoverTestCase.testLoadbalance randomly fails with error message: java.io.IOException: Channel Channel ID 973e1867 (outbound) of Remoting connection 6b306b30 to /127.0.0.1:4447 has been closed This was seen with TCPGOSSIP configuration (as in https://bugzilla.redhat.com/show_bug.cgi?id=922662), but is not specific for TCPGOSSIP. Stacktraces are in links (because they're long). (ASYNC-tcp) https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/40/jdk=ibm16,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28ASYNC-tcp%29/testLoadbalance/ (SYNC-tcp) https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/40/jdk=ibm17,label=RHEL6_x86/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28SYNC-tcp%29/testLoadbalance/ (SYNC-udp) https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/40/jdk=ibm16,label=RHEL6_x86_64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28SYNC-udp%29/testLoadbalance/
Paul, Im going to take over this issue. Bad run: [0m[0m14:04:40,958 INFO [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-3) JNDI bindings for session bean named ViewChangeListenerBean in deployment unit deployment "remote-ejb-client-stateless-bean-failover-test.jar" are as follows: java:global/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:app/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:module/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:jboss/exported/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:global/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean java:app/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean java:module/ViewChangeListenerBean [0m[0m14:04:42,151 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000078: Starting JGroups Channel [0m[0m14:04:42,172 INFO [stdout] (ServerService Thread Pool -- 52) [0m[0m14:04:42,172 INFO [stdout] (ServerService Thread Pool -- 52) ------------------------------------------------------------------- [0m[0m14:04:42,172 INFO [stdout] (ServerService Thread Pool -- 52) GMS: address=node-0/ejb, cluster=ejb, physical address=127.0.0.1:7600 [0m[0m14:04:42,172 INFO [stdout] (ServerService Thread Pool -- 52) ------------------------------------------------------------------- [0m[0m14:04:42,255 INFO [org.jboss.as.clustering] (Incoming-8,shared=tcp) JBAS010225: New cluster view for partition ejb (id: 5, delta: 1, merge: false) : [node-1/ejb, node-0/ejb] [0m[0m14:04:42,256 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-8,shared=tcp) ISPN000094: Received new cluster view: [node-1/ejb|5] [node-1/ejb, node-0/ejb] [0m[0m14:04:42,276 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000094: Received new cluster view: [node-1/ejb|5] [node-1/ejb, node-0/ejb] [0m[0m14:04:42,346 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000079: Cache local address is node-0/ejb, physical addresses are [127.0.0.1:7600] [0m[0m14:04:42,361 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 52) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.7.Final [0m[0m14:04:42,434 INFO [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 52) ISPN000161: Using a batchMode transaction manager [0m[0m14:04:42,431 INFO [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 53) ISPN000161: Using a batchMode transaction manager [0m[0m14:04:42,671 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 52) ISPN000031: MBeans were successfully registered to the platform MBean server. [0m[0m14:04:42,677 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 53) ISPN000031: MBeans were successfully registered to the platform MBean server. [0m[0m14:04:42,848 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 53) JBAS010281: Started repl cache from ejb container [0m[0m14:04:42,873 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 52) JBAS010281: Started remote-connector-client-mappings cache from ejb container [0m[0m14:04:42,879 INFO [org.jboss.as.clustering] (MSC service thread 1-8) JBAS010238: Number of cluster members: 2 [0m[0m14:04:42,975 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "remote-ejb-client-stateless-bean-failover-test.jar" (runtime-name : "remote-ejb-client-stateless-bean-failover-test.jar") [0m[0m14:04:43,063 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management [0m[0m14:04:43,064 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990 [0m[0m14:04:43,064 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.1.1.Alpha1 (AS 7.2.2-internal-SNAPSHOT) started in 6490ms - Started 184 of 299 services (114 services are passive or on-demand) [0m[0m14:04:43,235 INFO [stdout] (EJB default - 4) Cluster 'ejb' successfully established view [node-0/ejb, node-1/ejb] within 0 ms. [0m[0m14:04:47,896 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-1 [0m14:04:47,896 INFO [org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager] (ejb-client-cluster-node-connection-creation-3-thread-2) Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='node-0', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='127.0.0.1', destinationPort=4447}], resolvedDestination=[Destination address=127.0.0.1, destination port=4447]} in cluster ejb: java.lang.RuntimeException: Operation failed with status WAITING at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93) at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:77) at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51) at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:406) at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) 14:04:47,900 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.RemoteStatelessFailoverTestCase] (main) Node called: node-1 [0m14:04:47,908 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:47,916 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:47,920 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-1 [0m14:04:47,929 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000017: Received server version 2 and marshalling strategies [river] 14:04:47,931 INFO [org.jboss.ejb.client.remoting] (ejb-client-cluster-node-connection-creation-3-thread-4) EJBCLIENT000013: Su[0m14:04:47,932 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.ccessful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@698e8627, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection@17d3676a,channel=jboss.ejb,nodename=node-0]} on channel Channel ID fc42cff7 (outbound) of Remoting connection 612b7943 to /127.0.0.1:4447 getNodeName() was called on node: node-1 [0m[0m14:04:47,951 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 10) StatelessBean.getNodeName() was called on node: node-1 [0m14:04:47,960 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000017: Received server version 2 and marshalling strategies [river] [0m14:04:47,963 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-1 [0m14:04:47,966 INFO [org.jboss.ejb.client.remoting] (ejb-client-cluster-node-connection-creation-3-thread-3) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@19331f6e, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection@26190041,channel=jboss.ejb,nodename=node-0]} on channel Channel ID a859f52c (outbound) of Remoting connection 77a22fe3 to /127.0.0.1:4447 [0m14:04:47,970 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:47,976 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:47,981 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:47,989 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:47,995 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:47,999 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,004 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,008 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,012 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 10) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,020 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,023 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,026 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,029 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,032 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,035 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,038 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,042 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,045 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,047 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 10) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,050 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,053 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,058 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,063 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,066 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,069 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,073 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,076 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,079 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,082 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 10) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,085 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,088 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,094 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,097 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,100 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,109 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,113 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,116 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,119 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,122 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 10) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,126 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,129 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,131 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,134 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,137 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-1 [0m[0m14:04:48,216 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-5) ISPN000029: Passivating all entries to disk [0m[0m14:04:48,217 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-5) ISPN000030: Passivated 0 entries in 1 milliseconds [0m[0m14:04:48,225 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment remote-ejb-client-stateless-bean-failover-test.jar (runtime-name: remote-ejb-client-stateless-bean-failover-test.jar) in 52ms [0m[0m14:04:48,241 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010282: Stopped repl cache from ejb container [0m[0m14:04:48,246 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container [0m[0m14:04:48,249 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting and closing JGroups Channel [0m[0m14:04:48,607 INFO [org.jboss.as.clustering] (Incoming-10,shared=tcp) JBAS010225: New cluster view for partition ejb (id: 6, delta: -1, merge: false) : [node-1/ejb] [0m[0m14:04:48,609 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-10,shared=tcp) ISPN000094: Received new cluster view: [node-1/ejb|6] [node-1/ejb] [0m[0m14:04:48,619 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000082: Stopping the RpcDispatcher [0m[0m14:04:48,764 INFO [org.jboss.as.repository] (management-handler-thread - 3) JBAS014901: Content removed from location /home/baranowb/redhat/git/jboss-eap/testsuite/integration/clust/target/jbossas-clustering-SYNC-tcp-0/standalone/data/content/2b/5f3d5e2098514a8f08993fb9bb96eaf17ee525/content [0m[0m14:04:48,766 INFO [org.jboss.as.server] (management-handler-thread - 3) JBAS018558: Undeployed "remote-ejb-client-stateless-bean-failover-test.jar" (runtime-name: "remote-ejb-client-stateless-bean-failover-test.jar") [0m[0m14:04:48,815 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-8) ISPN000029: Passivating all entries to disk [0m[0m14:04:48,816 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-8) ISPN000030: Passivated 0 entries in 0 milliseconds [0m[0m14:04:48,820 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010282: Stopped repl cache from ejb container [0m[0m14:04:48,822 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container [0m[0m14:04:48,827 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting and closing JGroups Channel [0m[0m14:04:48,830 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015877: Stopped deployment remote-ejb-client-stateless-bean-failover-test.jar (runtime-name: remote-ejb-client-stateless-bean-failover-test.jar) in 52ms [0m[0m14:04:48,838 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher [0m[0m14:04:48,965 INFO [org.jboss.as.repository] (management-handler-thread - 2) JBAS014901: Content removed from location Good run: [0m[0m15:55:57,182 INFO [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-4) JNDI bindings for session bean named ViewChangeListenerBean in deployment unit deployment "remote-ejb-client-stateless-bean-failover-test.jar" are as follows: java:global/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:app/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:module/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:jboss/exported/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean!org.jboss.as.test.clustering.ViewChangeListener java:global/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean java:app/remote-ejb-client-stateless-bean-failover-test/ViewChangeListenerBean java:module/ViewChangeListenerBean [0m[0m15:55:58,359 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000078: Starting JGroups Channel [0m[0m15:55:58,378 INFO [stdout] (ServerService Thread Pool -- 52) [0m[0m15:55:58,379 INFO [stdout] (ServerService Thread Pool -- 52) ------------------------------------------------------------------- [0m[0m15:55:58,379 INFO [stdout] (ServerService Thread Pool -- 52) GMS: address=node-0/ejb, cluster=ejb, physical address=127.0.0.1:7600 [0m[0m15:55:58,379 INFO [stdout] (ServerService Thread Pool -- 52) ------------------------------------------------------------------- [0m[0m15:55:58,453 INFO [org.jboss.as.clustering] (Incoming-8,shared=tcp) JBAS010225: New cluster view for partition ejb (id: 5, delta: 1, merge: false) : [node-1/ejb, node-0/ejb] [0m[0m15:55:58,454 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-8,shared=tcp) ISPN000094: Received new cluster view: [node-1/ejb|5] [node-1/ejb, node-0/ejb] [0m[0m15:55:58,471 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000094: Received new cluster view: [node-1/ejb|5] [node-1/ejb, node-0/ejb] [0m[0m15:55:58,541 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 52) ISPN000079: Cache local address is node-0/ejb, physical addresses are [127.0.0.1:7600] [0m[0m15:55:58,549 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 52) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.7.Final [0m[0m15:55:58,624 INFO [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 52) ISPN000161: Using a batchMode transaction manager [0m[0m15:55:58,624 INFO [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 53) ISPN000161: Using a batchMode transaction manager [0m[0m15:55:58,822 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 52) ISPN000031: MBeans were successfully registered to the platform MBean server. [0m[0m15:55:58,829 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 53) ISPN000031: MBeans were successfully registered to the platform MBean server. [0m[0m15:55:59,013 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 53) JBAS010281: Started repl cache from ejb container [0m[0m15:55:59,030 INFO [org.jboss.as.clustering] (MSC service thread 1-5) JBAS010238: Number of cluster members: 2 [0m[0m15:55:59,052 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 52) JBAS010281: Started remote-connector-client-mappings cache from ejb container [0m15:55:59,119 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-3) EJBCLIENT000017: Received server version 2 and marshalling strategies [river] 15:55:59,121 INFO [org.jboss.ejb.client.remoting] (ejb-client-cluster-node-connection-creation-3-thread-2) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@7a6885e2, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection@251a135c,channel=jboss.ejb,nodename=node-0]} on channel Channel ID f16b6867 (outbound) of Remoting connection 247ec34e to /127.0.0.1:4447 15:55:59,143 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000017: Received server version 2 and marshalling strategies [river] 15:55:59,144 INFO [org.jboss.ejb.client.remoting] (ejb-client-cluster-node-connection-creation-3-thread-3) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@38054f37, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection@50f96cf6,channel=jboss.ejb,nodename=node-0]} on channel Channel ID b3ce2417 (outbound) of Remoting connection 3efc892d to /127.0.0.1:4447 15:55:59,172 INFO [org.jboss.ejb.client.remoting] (Remoting "config-based-ejb-client-endpoint" task-4) EJBCLIENT000017: Received server version 2 and marshalling strategies [river] 15:55:59,173 INFO [org.jboss.ejb.client.remoting] (ejb-client-cluster-node-connection-creation-3-thread-1) EJBCLIENT000013: Successful version handshake completed for receiver context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@6f146a55, receiver=Remoting connection EJB receiver [connection=org.jboss.ejb.client.remoting.ConnectionPool$PooledConnection@5b3cf82b,channel=jboss.ejb,nodename=node-0]} on channel Channel ID d37d679c (outbound) of Remoting connection 156b5ee4 to /127.0.0.1:4447 [0m15:55:59,191 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "remote-ejb-client-stateless-bean-failover-test.jar" (runtime-name : "remote-ejb-client-stateless-bean-failover-test.jar") [0m[0m15:55:59,267 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management [0m[0m15:55:59,268 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990 [0m[0m15:55:59,268 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.1.1.Alpha1 (AS 7.2.2-internal-SNAPSHOT) started in 6308ms - Started 184 of 299 services (114 services are passive or on-demand) [0m[0m15:55:59,734 INFO [stdout] (EJB default - 1) Cluster 'ejb' successfully established view [node-0/ejb, node-1/ejb] within 0 ms. [0m[0m15:55:59,747 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-0 [0m15:55:59,751 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.RemoteStatelessFailoverTestCase] (main) Node called: node-0 [0m15:55:59,754 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,758 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,762 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,766 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,770 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,775 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,778 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,782 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,787 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,797 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 10) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,802 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,805 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,809 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,812 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,815 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,818 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,821 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,825 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,828 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,831 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,834 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 10) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,838 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,844 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,849 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,853 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,856 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,860 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,863 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,867 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,870 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,876 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,882 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,886 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,889 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,892 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 10) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,895 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,899 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,902 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,906 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,909 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 4) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,912 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 5) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,915 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,919 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 10) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,922 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 6) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,925 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 7) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,931 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 8) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,936 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 1) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,939 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 2) StatelessBean.getNodeName() was called on node: node-1 [0m[0m15:55:59,942 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 9) StatelessBean.getNodeName() was called on node: node-0 [0m[0m15:55:59,947 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.bean.StatelessBean] (EJB default - 3) StatelessBean.getNodeName() was called on node: node-1 [0m15:55:59,948 INFO [org.jboss.as.test.clustering.cluster.ejb3.stateless.RemoteStatelessFailoverTestCase] (main) All 2 servers processed at least 10.000000 of calls [0m15:56:00,006 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-8) ISPN000029: Passivating all entries to disk [0m[0m15:56:00,007 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-8) ISPN000030: Passivated 0 entries in 1 milliseconds [0m[0m15:56:00,019 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010282: Stopped repl cache from ejb container [0m[0m15:56:00,026 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container [0m[0m15:56:00,027 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015877: Stopped deployment remote-ejb-client-stateless-bean-failover-test.jar (runtime-name: remote-ejb-client-stateless-bean-failover-test.jar) in 60ms [0m[0m15:56:00,030 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting and closing JGroups Channel [0m[0m15:56:00,339 INFO [org.jboss.as.clustering] (Incoming-10,shared=tcp) JBAS010225: New cluster view for partition ejb (id: 6, delta: -1, merge: false) : [node-1/ejb] [0m[0m15:56:00,340 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-10,shared=tcp) ISPN000094: Received new cluster view: [node-1/ejb|6] [node-1/ejb] [0m[0m15:56:00,348 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000082: Stopping the RpcDispatcher [0m[0m15:56:00,468 INFO [org.jboss.as.repository] (management-handler-thread - 3) JBAS014901: Content removed from location /home/baranowb/redhat/git/jboss-eap/testsuite/integration/clust/target/jbossas-clustering-SYNC-tcp-0/standalone/data/content/2f/5cf683bfd6bd58f14ae038432a782a3eb5a379/content [0m[0m15:56:00,468 INFO [org.jboss.as.server] (management-handler-thread - 3) JBAS018558: Undeployed "remote-ejb-client-stateless-bean-failover-test.jar" (runtime-name: "remote-ejb-client-stateless-bean-failover-test.jar") [0m[0m15:56:00,523 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-2) ISPN000029: Passivating all entries to disk [0m[0m15:56:00,526 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-2) ISPN000030: Passivated 0 entries in 2 milliseconds [0m[0m15:56:00,527 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container [0m[0m15:56:00,530 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010282: Stopped repl cache from ejb container [0m[0m15:56:00,531 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment remote-ejb-client-stateless-bean-failover-test.jar (runtime-name: remote-ejb-client-stateless-bean-failover-test.jar) in 52ms [0m[0m15:56:00,534 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting and closing JGroups Channel [0m[0m15:56:00,546 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher [0m[0m15:56:00,646 INFO [org.jboss.as.repository] (management-handler-thread - 2) JBAS014901: Content removed from location /home/baranowb/redhat/git/jboss-eap/testsuite/integration/clust/target/jbossas-clustering-SYNC-tcp-
Initial traces look like ibm jdk finalization bug: https://bugzilla.redhat.com/show_bug.cgi?id=901231 Recent traces are possibly from 6.x, not 6.1.x-proposed, Im trying to reproduce.
Update: also seen in EAP 6.2.0.ER1, though it is with a different stacktrace: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-JGroups/job/eap-6x-jgroups-tcpgossip-rhel-matrix/63/jdk=java16_default,label=RHEL6_x86_64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28SYNC-tcp%29/testLoadBalance/ Error Message expected:<2> but was:<1> Stacktrace java.lang.AssertionError: expected:<2> but was:<1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.jboss.as.test.clustering.cluster.ejb3.stateless.RemoteStatelessFailoverTestCase.validateBalancing(RemoteStatelessFailoverTestCase.java:297) at org.jboss.as.test.clustering.cluster.ejb3.stateless.RemoteStatelessFailoverTestCase.testLoadBalance(RemoteStatelessFailoverTestCase.java:275) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.jboss.arquillian.junit.Arquillian$6$1.invoke(Arquillian.java:270) at org.jboss.arquillian.container.test.impl.execution.LocalTestExecuter.execute(LocalTestExecuter.java:60) at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:115) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.test.impl.execution.ClientTestExecuter.execute(ClientTestExecuter.java:53) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createContext(ContainerEventController.java:142) at org.jboss.arquillian.container.test.impl.client.ContainerEventController.createTestContext(ContainerEventController.java:129) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createTestContext(TestContextHandler.java:89) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createClassContext(TestContextHandler.java:75) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.test.impl.TestContextHandler.createSuiteContext(TestContextHandler.java:60) at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:90) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135) at org.jboss.arquillian.test.impl.EventTestRunnerAdaptor.test(EventTestRunnerAdaptor.java:111) at org.jboss.arquillian.junit.Arquillian$6.evaluate(Arquillian.java:263) at org.jboss.arquillian.junit.Arquillian$4.evaluate(Arquillian.java:226) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314) at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46) at org.jboss.arquillian.junit.Arquillian$5.evaluate(Arquillian.java:240) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.jboss.arquillian.junit.Arquillian$2.evaluate(Arquillian.java:185) at org.jboss.arquillian.junit.Arquillian.multiExecute(Arquillian.java:314) at org.jboss.arquillian.junit.Arquillian.access$100(Arquillian.java:46) at org.jboss.arquillian.junit.Arquillian$3.evaluate(Arquillian.java:199) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.jboss.arquillian.junit.Arquillian.run(Arquillian.java:147) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
The addition of the GlobalComponentRegistryService should improve the reliability of this test. Please retest against EAP 6.2.0.ER2.
Still present in 6.2.0.ER3. https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/65/jdk=ibm17,label=RHEL6_x86_64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28ASYNC-tcp%29/testLoadBalance/
@Richard -- is this only IBM JDK related? It seems like that from the jobs.
I've checked ER3 runs and it is not: HPUX with oracle jdk 1.6 https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-hpux-matrix/9/BITS=-d64,jdk=jdk16_hpux,label=hpux11v3/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28ASYNC-tcp%29/testLoadBalance/ RHEL with openjdk 1.7 https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/65/jdk=openjdk-1.7.0-local,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28SYNC-tcp%29/testLoadBalance/ RHEL (EAP 6.2.0.ER1) with oracle jdk 1.7 https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/63/jdk=java17_default,label=RHEL5_x86_64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28SYNC-tcp%29/testLoadBalance/ RHEL (EAP 6.2.0.ER1) with oracle jdk 1.6 https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/63/jdk=java16_default,label=RHEL6_x86_64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28SYNC-tcp%29/testLoadBalance/ Solaris with oracle jdk 1.7 https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-solaris-matrix/36/jdk=java17_default,label=solaris10-x64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28SYNC-tcp%29/testLoadBalance/ Solaris with oracle jdk 1.7 - and this one has a different error message: "9" https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-solaris-matrix/36/jdk=java17_default,label=sol10_sparc64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28ASYNC-tcp%29/testLoadBalance/ Windows with java 1.7 - error message as above: "8" - assert fails on the same line https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-win-matrix/62/jdk=java17_default,label=Win2k8r2_x86_64/testReport/org.jboss.as.test.clustering.cluster.ejb3.stateless/RemoteStatelessFailoverTestCase%28ASYNC-tcp%29/testLoadBalance/
This is no longer able to be addressed in the time remaining for release.
This issue seems to be permanent on RHEL 7.
Stable fail on RHEL 7 !
I've been rerunning some configurations to see if the issue was really permanent on RHEL 7 (only x86-64, it isn't that common with ppc) and It seems the test fails very often but not always. See [1] and [2]. [1]: run from ER4 with udp-sync where testLoadbalance passed https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/90/jdk=java16_default,label_exp=EAP-RHEL7/consoleText [2]: run from ER5 with udp-async where testLoadbalance passed https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-jgroups-tcpgossip-rhel-matrix/95/jdk=ibm17,label_exp=EAP-RHEL7/consoleText
Looking again more closely at the logs all points to a remoting issue. Context: The test starts 2 nodes, then accesses the SLSB multiple times and makes sure the client did load-balance. It fails, because all the responses are from a single node. The clustering logic was executed correctly. The EJB client has the correct topology installed (node-0 and node-1) but it fails to connect to one of the nodes: 09:00:31,031 INFO [org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager] (ejb-client-cluster-node-connection-creation-3-thread-4) Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='node-0', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='127.0.0.1', destinationPort=4447}], resolvedDestination=[Destination address=127.0.0.1, destination port=4447]} in cluster ejb: java.lang.RuntimeException: Operation failed with status WAITING Fixing the component.
If this is really a remoting issue per comment 20, maybe it's a duplicate of bug 900595 (note that the summary is incorrent, 900595 is not tied to IPv6)?
*** This bug has been marked as a duplicate of bug 1090357 ***
After some discussions with baranowb I've been rerunning my Windows matrix with modified configuration (6.4.0.DR11, 2 runs at the time of writing this comment) and I'm not seeing the second failure anymore. Closing as NOTABUG.