Bug 855350 - org.infinispan.distribution.topologyaware.TopologyAwareStateTransferTest.testNodeDown3 fails randomly
org.infinispan.distribution.topologyaware.TopologyAwareStateTransferTest.test...
Status: ASSIGNED
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan (Show other bugs)
6.0.1
Unspecified Unspecified
unspecified Severity medium
: ER5
: 6.2.0
Assigned To: Tristan Tarrant
Martin Gencur
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-07 09:43 EDT by Anna Manukyan
Modified: 2016-03-31 09:39 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-2516 Major Resolved org.infinispan.distribution.topologyaware.TopologyAwareStateTransferTest.testNodeDown3 fails randomly 2016-03-31 09:39 EDT

  None (edit)
Comment 1 Anna Manukyan 2012-09-12 08:47:20 EDT
The bug doesn't appear when the test is run separately.
Comment 2 mark yarborough 2012-09-12 09:24:19 EDT
Per ttarrant and mgencur, these are related to tests running in non-representative loading environments and will likely not occur in actual customer deployments. These are not regressions or blockers so will be moved to 6.1. They do not need to be release noted for 6.0.
Comment 3 mark yarborough 2012-11-14 09:42:31 EST
ttarrant will add jira links as appropriate.
Comment 4 Anna Manukyan 2012-12-11 07:53:17 EST
For ER5, the testNodeDown has been failed, and just in case the provided links will be broken, the error message is:

Error Message

Timed out waiting for rebalancing to complete on node TopologyAwareStateTransferTest-NodeA-23408, current topology is CacheTopology{id=9, currentCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[TopologyAwareStateTransferTest-NodeA-23408, TopologyAwareStateTransferTest-NodeB-47156, TopologyAwareStateTransferTest-NodeC-50852, TopologyAwareStateTransferTest-NodeD-5971], owners={0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 1, 10: 0 1, 11: 0 1, 12: 0, 13: 0, 14: 0, 15: 3 0, 16: 3 0, 17: 3 0, 18: 3 0, 19: 3 0, 20: 2 0, 21: 2 0, 22: 2 0, 23: 2 0, 24: 2 0, 25: 2 0, 26: 2 0, 27: 2 0, 28: 2 0, 29: 2 0, 30: 1 0, 31: 1 0, 32: 1 0, 33: 1 0, 34: 1 0, 35: 1 0, 36: 1 0, 37: 1 0, 38: 1 0, 39: 1 0, 40: 1 0, 41: 1 0, 42: 1, 43: 1, 44: 1, 45: 3 1, 46: 3 1, 47: 3 1, 48: 3 1, 49: 3 1, 50: 2 1, 51: 2 1, 52: 2, 53: 2, 54: 2, 55: 3 2, 56: 3 2, 57: 3, 58: 3, 59: 3}, pendingCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[TopologyAwareStateTransferTest-NodeA-23408, TopologyAwareStateTransferTest-NodeB-47156, TopologyAwareStateTransferTest-NodeC-50852, TopologyAwareStateTransferTest-NodeD-5971], owners={0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 1, 10: 0 1, 11: 0 1, 12: 0 2, 13: 0 2, 14: 0 2, 15: 3 0, 16: 3 0, 17: 3 0, 18: 3 0, 19: 3 0, 20: 2 0, 21: 2 0, 22: 2 0, 23: 2 0, 24: 2 0, 25: 2 0, 26: 2 0, 27: 2 0, 28: 2 0, 29: 2 0, 30: 1 0, 31: 1 0, 32: 1 0, 33: 1 0, 34: 1 0, 35: 1 0, 36: 1 0, 37: 1 0, 38: 1 0, 39: 1 0, 40: 1 0, 41: 1 0, 42: 1 2, 43: 1 2, 44: 1 2, 45: 3 1, 46: 3 1, 47: 3 1, 48: 3 1, 49: 3 1, 50: 2 1, 51: 2 1, 52: 2 3, 53: 2 3, 54: 2 3, 55: 3 2, 56: 3 2, 57: 3 2, 58: 3 2, 59: 3 2}}

Stacktrace

java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node TopologyAwareStateTransferTest-NodeA-23408, current topology is CacheTopology{id=9, currentCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[TopologyAwareStateTransferTest-NodeA-23408, TopologyAwareStateTransferTest-NodeB-47156, TopologyAwareStateTransferTest-NodeC-50852, TopologyAwareStateTransferTest-NodeD-5971], owners={0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 1, 10: 0 1, 11: 0 1, 12: 0, 13: 0, 14: 0, 15: 3 0, 16: 3 0, 17: 3 0, 18: 3 0, 19: 3 0, 20: 2 0, 21: 2 0, 22: 2 0, 23: 2 0, 24: 2 0, 25: 2 0, 26: 2 0, 27: 2 0, 28: 2 0, 29: 2 0, 30: 1 0, 31: 1 0, 32: 1 0, 33: 1 0, 34: 1 0, 35: 1 0, 36: 1 0, 37: 1 0, 38: 1 0, 39: 1 0, 40: 1 0, 41: 1 0, 42: 1, 43: 1, 44: 1, 45: 3 1, 46: 3 1, 47: 3 1, 48: 3 1, 49: 3 1, 50: 2 1, 51: 2 1, 52: 2, 53: 2, 54: 2, 55: 3 2, 56: 3 2, 57: 3, 58: 3, 59: 3}, pendingCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[TopologyAwareStateTransferTest-NodeA-23408, TopologyAwareStateTransferTest-NodeB-47156, TopologyAwareStateTransferTest-NodeC-50852, TopologyAwareStateTransferTest-NodeD-5971], owners={0: 0 1, 1: 0 1, 2: 0 1, 3: 0 1, 4: 0 1, 5: 0 1, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 1, 10: 0 1, 11: 0 1, 12: 0 2, 13: 0 2, 14: 0 2, 15: 3 0, 16: 3 0, 17: 3 0, 18: 3 0, 19: 3 0, 20: 2 0, 21: 2 0, 22: 2 0, 23: 2 0, 24: 2 0, 25: 2 0, 26: 2 0, 27: 2 0, 28: 2 0, 29: 2 0, 30: 1 0, 31: 1 0, 32: 1 0, 33: 1 0, 34: 1 0, 35: 1 0, 36: 1 0, 37: 1 0, 38: 1 0, 39: 1 0, 40: 1 0, 41: 1 0, 42: 1 2, 43: 1 2, 44: 1 2, 45: 3 1, 46: 3 1, 47: 3 1, 48: 3 1, 49: 3 1, 50: 2 1, 51: 2 1, 52: 2 3, 53: 2 3, 54: 2 3, 55: 3 2, 56: 3 2, 57: 3 2, 58: 3 2, 59: 3 2}}
	at org.infinispan.test.TestingUtil.waitForRehashToComplete(TestingUtil.java:198)
	at org.infinispan.test.TestingUtil.waitForRehashToComplete(TestingUtil.java:208)
	at org.infinispan.distribution.topologyaware.TopologyAwareStateTransferTest.testNodeDown(TopologyAwareStateTransferTest.java:100)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
	at org.testng.TestRunner.privateRun(TestRunner.java:767)
	at org.testng.TestRunner.run(TestRunner.java:617)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
	at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
Comment 6 JBoss JIRA Server 2013-06-07 10:31:48 EDT
Anna Manukyan <amanukya@redhat.com> updated the status of jira ISPN-2516 to Resolved
Comment 7 JBoss JIRA Server 2013-06-07 10:31:48 EDT
Anna Manukyan <amanukya@redhat.com> made a comment on jira ISPN-2516

The issue doesn't appear any more. I've run the build for Infinispan 5.4.2.Final for several times, and the issue didn't appear.
Comment 9 Anna Manukyan 2013-10-07 05:14:34 EDT
Hi,

the failure reappears on java16_default,w2k8r2 && x86_64 environment. The failure is:

Error Message

Timed out before caches had complete views.  Expected 2 members in each view.  Views are as follows: [[TopologyAwareStateTransferTest-NodeA-65287], [TopologyAwareStateTransferTest-NodeA-65287, TopologyAwareStateTransferTest-NodeB-35578, TopologyAwareStateTransferTest-NodeD-57223]]

Stacktrace

java.lang.RuntimeException: Timed out before caches had complete views.  Expected 2 members in each view.  Views are as follows: [[TopologyAwareStateTransferTest-NodeA-65287], [TopologyAwareStateTransferTest-NodeA-65287, TopologyAwareStateTransferTest-NodeB-35578, TopologyAwareStateTransferTest-NodeD-57223]]
	at org.infinispan.test.TestingUtil.viewsTimedOut(TestingUtil.java:248)
	at org.infinispan.test.TestingUtil.viewsTimedOut(TestingUtil.java:238)
	at org.infinispan.test.TestingUtil.blockUntilViewsReceived(TestingUtil.java:333)
	at org.infinispan.test.TestingUtil.blockUntilViewsReceived(TestingUtil.java:907)
	at org.infinispan.distribution.topologyaware.TopologyAwareStateTransferTest.testNodeDown3(TopologyAwareStateTransferTest.java:122)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
	at org.testng.TestRunner.privateRun(TestRunner.java:767)
	at org.testng.TestRunner.run(TestRunner.java:617)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
	at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
	at java.lang.Thread.run(Thread.java:662)

Standard Output

[testng-TopologyAwareStateTransferTest] Test testNodeDown3(org.infinispan.distribution.topologyaware.TopologyAwareStateTransferTest) failed.
Test suite progress: tests succeeded: 1651, failed: 1, skipped: 0.
Comment 10 JBoss JIRA Server 2013-10-07 05:16:06 EDT
Anna Manukyan <amanukya@redhat.com> updated the status of jira ISPN-2516 to Reopened
Comment 11 JBoss JIRA Server 2013-10-07 05:16:06 EDT
Anna Manukyan <amanukya@redhat.com> made a comment on jira ISPN-2516

Hi,

the issue again appears on Windows 2008R2 64 bit machine with JDK6.

You can find the infinispan tracelog for core module attached.

Regards,
Anna.
Comment 12 JBoss JIRA Server 2016-03-31 09:39:42 EDT
Dan Berindei <dberinde@redhat.com> updated the status of jira ISPN-2516 to Resolved

Note You need to log in before you can comment on or make changes to this bug.