Created attachment 820312 [details]
testsuiteLog-3.txt

Description of problem:

EAP 6.2.0.ER7, but I've seen this in previous versions as well. It seems to me that it happens _slightly_ more often now.
JGroups 3.2.12.Final, also seen in previous versions.

There is an intermittent failure in configuration of tests of org.jgroups.blocks.executor.ExecutingServiceTest:

   [testng] OK:   [48] org.jgroups.blocks.executor.ExecutingServiceTest.testCoordinatorWentDownWhileSendingMessage()
   [testng] failed config: java.lang.AssertionError: Unfinished e2 futures should be empty!
   [testng] FAIL: [48] org.jgroups.blocks.executor.ExecutingServiceTest.makeSureClean()
   [testng] SKIP: [48] org.jgroups.blocks.executor.ExecutingServiceTest.testExecutionCompletionService()

This causes the rest of the tests from ExecutingServiceTest to be skipped, but then there are other tests, that to me seem unrelated, skipped as well. Parallel running of tests and methods is set to true. A snippet from a log would look like this (different log than the one above):

...
   [testng] OK:   [48] org.jgroups.tests.MessageBundlingTest.testLatencyWithMessageBundling()
   [testng] OK:   [51] org.jgroups.tests.DiscardTest.testFastProperties()
   [testng] OK:   [49] org.jgroups.blocks.executor.ExecutingServiceTest.testCoordinatorWentDownWhileSendingMessage()
   [testng] OK:   [49] org.jgroups.blocks.executor.ExecutingServiceTest.testExecutionCompletionService()
   [testng] OK:   [49] org.jgroups.blocks.executor.ExecutingServiceTest.testExecutorAwaitTerminationInterrupt()
   [testng] OK:   [49] org.jgroups.blocks.executor.ExecutingServiceTest.testExecutorAwaitTerminationNoInterrupt()
   [testng] FAIL: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testInterruptTaskRequestBeforeRunning()
   [testng] failed config: java.lang.AssertionError: Unfinished e1 futures should be empty!
   [testng] FAIL: [49] org.jgroups.blocks.executor.ExecutingServiceTest.makeSureClean()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testInterruptWhileRunningAlot()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testInvokeAnyCalls()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testNonSerializableCallable()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testSimpleSerializableCallableConcurrently()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testSimpleSerializableCallableSubmit()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testSubmittingNonSerializeCallable()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testSubmittingSerializeCallableWithNonSerializableComponent()
   [testng] SKIP: [49] org.jgroups.tests.Deadlock2Test.testOneChannel()
   [testng] SKIP: [49] org.jgroups.tests.Deadlock2Test.testTwoChannels()
   [testng] SKIP: [49] org.jgroups.tests.Deadlock2Test.testTwoChannelsWithInitialMulticast()
   [testng] SKIP: [49] org.jgroups.blocks.RpcDispatcherInterruptTest.testMethodCallWithTimeoutNoInterrupt()
   [testng] SKIP: [49] org.jgroups.blocks.MuxRpcDispatcherTest.testMulticastRPCs()
   [testng] SKIP: [49] org.jgroups.blocks.MuxRpcDispatcherTest.testUnicastRPCs()
   [testng] SKIP: [49] org.jgroups.tests.ChannelTest.testBasicOperations()
   [testng] SKIP: [49] org.jgroups.tests.ChannelTest.testIsConnectedOnFirstViewChange()
...

A note: it happens with udp as well as tcp config. Stacktraces of the failed tests would look like this:

org.jgroups.blocks.executor.ExecutingServiceTest.testInterruptTaskRequestBeforeRunning (from org.jgroups.blocks.executor.ExecutingServiceTest-udp)

Error Message

The object in queue doesn't match

Stacktrace

java.lang.AssertionError 
at org.jgroups.blocks.executor.ExecutingServiceTest.testInterruptTaskRequestBeforeRunning(ExecutingServiceTest.java:419) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:616) 
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80) 
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:46) 
at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:37) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) 
at java.util.concurrent.FutureTask.run(FutureTask.java:166) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:679) 


and for makeSureClean():


Error Message

Unfinished e1 futures should be empty!

Stacktrace

java.lang.AssertionError 
at org.jgroups.blocks.executor.ExecutingServiceTest.makeSureClean(ExecutingServiceTest.java:110) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:616) 
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80) 
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564) 
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213) 
at org.testng.internal.Invoker.invokeMethod(Invoker.java:786) 
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901) 
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231) 
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127) 
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:679) 


Links to test results for some of the configurations affected with this are below, testsuite logs are attached:
[1]: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-manu-jgroups-rhel/26/jdk=ibm16,label=RHEL5_x86/testReport/
[2]: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-manu-jgroups-rhel/26/jdk=openjdk-1.6.0-local,label=RHEL5_x86/testReport/
[3]: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-manu-jgroups-rhel/26/jdk=openjdk-1.7.0-local,label=RHEL6_x86/testReport/