Bug 805238 - Elasticity tests in REPL mode don't finish
Elasticity tests in REPL mode don't finish
Status: CLOSED NOTABUG
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan (Show other bugs)
6.0.0
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Tristan Tarrant
Michal Linhard
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-20 13:25 EDT by Michal Linhard
Modified: 2014-03-17 00:02 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-26 06:55:55 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
view installation times (824 bytes, text/html)
2012-03-20 13:25 EDT, Michal Linhard
no flags Details
view installation times after framework fix (1.31 KB, text/html)
2012-03-26 06:57 EDT, Michal Linhard
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-1933 None Closed State transfer in REPL mode takes more than 10 min 2012-03-27 23:58:15 EDT

  None (edit)
Description Michal Linhard 2012-03-20 13:25:20 EDT
Created attachment 571487 [details]
view installation times

Elasticity test with JDG 6.0.0.ER4 in REPL clustering mode:

http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-elasticity-repl-basic/3

starting from 1 node going to 3 nodes everything is allright
and then when 4th node is joining (viewId=4) the view installation takes more than 10 min.

Client load is 500
Data load is 5% of total heap.
Comment 1 Michal Linhard 2012-03-20 14:15:57 EDT
Of course this is not easy to replicate with small load...
Again this will be tough to TRACE

Run with 10clients 10K dataload
http://www.qa.jboss.com/~mlinhard/hyperion/run34-elasticity4-repl/report/stats-throughput.png
Comment 2 Michal Linhard 2012-03-20 15:14:48 EDT
Run with 500 clients 5% dataload TRACE log:
http://www.qa.jboss.com/~mlinhard/hyperion/run35-elasticity4-repl-trace/
generated 7.8G of logs, didn't replicate the issue
Comment 3 Michal Linhard 2012-03-20 15:28:26 EDT
times to install views are around 40 secs:
http://www.qa.jboss.com/~mlinhard/hyperion/run35-elasticity4-repl-trace/table.html
Comment 5 JBoss JIRA Server 2012-03-20 16:47:10 EDT
Dan Berindei <dberinde@redhat.com> made a comment on jira ISPN-1933

Michal, it looks like your test is using standalone.xml from the master branch, which is outdated, and not the latest config in the prod-6.0.0 branch.
Comment 6 Michal Linhard 2012-03-22 08:08:33 EDT
I updated the config we're using for the REPL tests:
https://svn.devel.redhat.com/repos/jboss-qa/load-testing/etc/edg-60/configs/comparison/stress-repl-sync.xml

In hyperion I ran the 4 node elasticity test 3 times and couldn't reproduce this anymore.

In edg lab this problem is still there:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-elasticity-repl-basic/4/
and still only with REPL case, the DIST works with the same JGroups config:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-elasticity-dist-basic/46/
Comment 7 Michal Linhard 2012-03-22 11:14:26 EDT
Second run in edg lab didn't even get to 3 node cluster:
http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-elasticity-repl-basic/5/artifact/report/stats-throughput.png

I'll try to run with trace logging
Comment 9 Michal Linhard 2012-03-22 14:28:06 EDT
this might be a framework problem. The test might have been ended sooner because it didn't ignore some expected exceptions during join/leave.
Comment 10 Michal Linhard 2012-03-22 14:55:15 EDT
Consider this a false alarm until further notice
Comment 11 Dan Berindei 2012-03-23 03:02:45 EDT
Michal, I don't think it's a false alarm, but it's a different problem. I'm seeing these errors in your TRACE log:

08:10:29,004 ERROR [org.infinispan.statetransfer.StateTransferLockImpl] Trying to release state transfer shared lock without acquiring it first: java.lang.Exception
08:10:29,005 ERROR [org.infinispan.statetransfer.StateTransferLockImpl] Trying to release state transfer shared lock without acquiring it first: java.lang.Exception

They are certainly not expected, so I'm looking into it.
Comment 12 Michal Linhard 2012-03-23 03:57:21 EDT
Oh It's a new one you're right, I thought it's
https://issues.jboss.org/browse/ISPN-1754
therefore I didn't report it, but the stack trace is different there.

Did you create JIRA for that ?

It occurs only after I start stopping the servers (or killing them).
In the log you can see this event marked by "Test will now stop the server", that's written to the log output file right before I send the kill signal to JBoss.
Comment 13 Michal Linhard 2012-03-23 04:01:17 EDT
I think I meant this one: https://issues.jboss.org/browse/ISPN-1704
So now I don't know why I didn't reopen it :-)
For some reason I thought it's something expected/insignificant, it's good that you spotted that.
Comment 14 Dan Berindei 2012-03-23 04:04:31 EDT
It looks very much like ISPN-1704, but that one happened on the surviving nodes - this one seems to happen on the nodes that you're killing. So I think it might be worth a separate bug after all.
Comment 15 Michal Linhard 2012-03-26 06:55:55 EDT
After fixing a problem in test framework, tests run till the end, and all state tansfers complete under 5sec.

http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-elasticity-repl-basic/12/artifact/report/stats-throughput.png
Comment 16 Michal Linhard 2012-03-26 06:57:15 EDT
Created attachment 572732 [details]
view installation times after framework fix

adding new view installation times after framework fix
Comment 17 JBoss JIRA Server 2012-03-26 06:59:48 EDT
Michal Linhard <mlinhard@redhat.com> updated the status of jira ISPN-1933 to Closed
Comment 18 JBoss JIRA Server 2012-03-26 06:59:48 EDT
Michal Linhard <mlinhard@redhat.com> made a comment on jira ISPN-1933

This was a problem in the test itself. The state transfer didn't really last more than 10 min.

Note You need to log in before you can comment on or make changes to this bug.