Description of problem:

Seen with EAP 6.1.1.ER3, Infinispan 5.2.7

Test org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus
randomly fails (seen on Solaris and HP-UX).

Stacktraces:

HP-UX version

Error Message

Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!

Stacktrace

      java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest-NodeB-47391, expected member list is [ClusterTopologyManagerTest-NodeB-47391], current member list is [ClusterTopologyManagerTest-NodeB-47391, ClusterTopologyManagerTest-NodeC-55740]!
	at org.infinispan.test.TestingUtil.waitForRehashToComplete(TestingUtil.java:203)
	at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:353)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
	at org.testng.TestRunner.privateRun(TestRunner.java:767)
	at org.testng.TestRunner.run(TestRunner.java:617)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
	at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

also see standard output:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infinispan/job/eap-6x-infinispan-hpux-matrix-clone/3/BITS=-d64,jdk=jdk16_hpux,label=hpux11v3/testReport/org.infinispan.statetransfer/ClusterTopologyManagerTest/testAbruptLeaveAfterGetStatus/



Solaris version

Error Message

Thread already timed out waiting for event 3 left

Stacktrace

java.lang.IllegalStateException: Thread already timed out waiting for event 3 left
	at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:150)
	at org.infinispan.test.fwk.CheckPoint.trigger(CheckPoint.java:135)
	at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus(ClusterTopologyManagerTest.java:350)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
	at org.testng.TestRunner.privateRun(TestRunner.java:767)
	at org.testng.TestRunner.run(TestRunner.java:617)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
	at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)


also see standard output:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Infinispan/job/eap-6x-infinispan-solaris-matrix/38/jdk=java16_default,label=solaris10-x64/testReport/org.infinispan.statetransfer/ClusterTopologyManagerTest/testAbruptLeaveAfterGetStatus/