Bug 1027253

Summary: JGroups: intermittent failures in ExecutingServiceTest
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Richard Janík <rjanik>
Component: ClusteringAssignee: Richard Achmatowicz <rachmato>
Status: CLOSED NOTABUG QA Contact: Jitka Kozana <jkudrnac>
Severity: medium Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.2.0CC: rachmato
Target Milestone: ---   
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: JGroups testsuite
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-03-27 14:36:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
testsuiteLog-3.txt
none
testsuiteLog-2.txt
none
testsuiteLog-1.txt none

Description Richard Janík 2013-11-06 12:36:13 UTC 
Created attachment 820312 [details]
testsuiteLog-3.txt

Description of problem:

EAP 6.2.0.ER7, but I've seen this in previous versions as well. It seems to me that it happens _slightly_ more often now.
JGroups 3.2.12.Final, also seen in previous versions.

There is an intermittent failure in configuration of tests of org.jgroups.blocks.executor.ExecutingServiceTest:

   [testng] OK:   [48] org.jgroups.blocks.executor.ExecutingServiceTest.testCoordinatorWentDownWhileSendingMessage()
   [testng] failed config: java.lang.AssertionError: Unfinished e2 futures should be empty!
   [testng] FAIL: [48] org.jgroups.blocks.executor.ExecutingServiceTest.makeSureClean()
   [testng] SKIP: [48] org.jgroups.blocks.executor.ExecutingServiceTest.testExecutionCompletionService()

This causes the rest of the tests from ExecutingServiceTest to be skipped, but then there are other tests, that to me seem unrelated, skipped as well. Parallel running of tests and methods is set to true. A snippet from a log would look like this (different log than the one above):

...
   [testng] OK:   [48] org.jgroups.tests.MessageBundlingTest.testLatencyWithMessageBundling()
   [testng] OK:   [51] org.jgroups.tests.DiscardTest.testFastProperties()
   [testng] OK:   [49] org.jgroups.blocks.executor.ExecutingServiceTest.testCoordinatorWentDownWhileSendingMessage()
   [testng] OK:   [49] org.jgroups.blocks.executor.ExecutingServiceTest.testExecutionCompletionService()
   [testng] OK:   [49] org.jgroups.blocks.executor.ExecutingServiceTest.testExecutorAwaitTerminationInterrupt()
   [testng] OK:   [49] org.jgroups.blocks.executor.ExecutingServiceTest.testExecutorAwaitTerminationNoInterrupt()
   [testng] FAIL: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testInterruptTaskRequestBeforeRunning()
   [testng] failed config: java.lang.AssertionError: Unfinished e1 futures should be empty!
   [testng] FAIL: [49] org.jgroups.blocks.executor.ExecutingServiceTest.makeSureClean()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testInterruptWhileRunningAlot()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testInvokeAnyCalls()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testNonSerializableCallable()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testSimpleSerializableCallableConcurrently()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testSimpleSerializableCallableSubmit()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testSubmittingNonSerializeCallable()
   [testng] SKIP: [49] org.jgroups.blocks.executor.ExecutingServiceTest.testSubmittingSerializeCallableWithNonSerializableComponent()
   [testng] SKIP: [49] org.jgroups.tests.Deadlock2Test.testOneChannel()
   [testng] SKIP: [49] org.jgroups.tests.Deadlock2Test.testTwoChannels()
   [testng] SKIP: [49] org.jgroups.tests.Deadlock2Test.testTwoChannelsWithInitialMulticast()
   [testng] SKIP: [49] org.jgroups.blocks.RpcDispatcherInterruptTest.testMethodCallWithTimeoutNoInterrupt()
   [testng] SKIP: [49] org.jgroups.blocks.MuxRpcDispatcherTest.testMulticastRPCs()
   [testng] SKIP: [49] org.jgroups.blocks.MuxRpcDispatcherTest.testUnicastRPCs()
   [testng] SKIP: [49] org.jgroups.tests.ChannelTest.testBasicOperations()
   [testng] SKIP: [49] org.jgroups.tests.ChannelTest.testIsConnectedOnFirstViewChange()
...

A note: it happens with udp as well as tcp config. Stacktraces of the failed tests would look like this:

org.jgroups.blocks.executor.ExecutingServiceTest.testInterruptTaskRequestBeforeRunning (from org.jgroups.blocks.executor.ExecutingServiceTest-udp)

Error Message

The object in queue doesn't match

Stacktrace

java.lang.AssertionError 
at org.jgroups.blocks.executor.ExecutingServiceTest.testInterruptTaskRequestBeforeRunning(ExecutingServiceTest.java:419) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:616) 
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80) 
at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:46) 
at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:37) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) 
at java.util.concurrent.FutureTask.run(FutureTask.java:166) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:679) 


and for makeSureClean():


Error Message

Unfinished e1 futures should be empty!

Stacktrace

java.lang.AssertionError 
at org.jgroups.blocks.executor.ExecutingServiceTest.makeSureClean(ExecutingServiceTest.java:110) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:616) 
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80) 
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564) 
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213) 
at org.testng.internal.Invoker.invokeMethod(Invoker.java:786) 
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901) 
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231) 
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127) 
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:679) 


Links to test results for some of the configurations affected with this are below, testsuite logs are attached:
[1]: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-manu-jgroups-rhel/26/jdk=ibm16,label=RHEL5_x86/testReport/
[2]: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-manu-jgroups-rhel/26/jdk=openjdk-1.6.0-local,label=RHEL5_x86/testReport/
[3]: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-manu-jgroups-rhel/26/jdk=openjdk-1.7.0-local,label=RHEL6_x86/testReport/
Comment 1 Richard Janík 2013-11-06 12:37:53 UTC 
Created attachment 820313 [details]
testsuiteLog-2.txt
Comment 2 Richard Janík 2013-11-06 12:38:32 UTC 
Created attachment 820314 [details]
testsuiteLog-1.txt
Comment 3 Richard Achmatowicz 2014-03-27 14:26:53 UTC 
NOTE: A number of EAP clustering issues relate to failures in the JGroups testsuite. Some of these failures relate to tests of features of JGroups which are not used by EAP.

As of EAP 6.3, a modified version of the JGroups testsuite (JGroups version 3.2.13) is being executed which excludes tests of JGroups features which are not used by EAP. Only the testsuite has been modified/fixed; the product is unchanged. The list of features can be found here: https://mojo.redhat.com/docs/DOC-938761

EAP 6.3 Clustering issues relating to failures in the JGroups testsuite are being closed if they relate to features not used by EAP.
Comment 4 Richard Achmatowicz 2014-03-27 14:36:53 UTC 
This test is not within the scope of EAP 6.3 testing.