Red Hat Bugzilla – Bug 805238
Elasticity tests in REPL mode don't finish
Last modified: 2014-03-17 00:02:13 EDT
Created attachment 571487 [details]
view installation times
Elasticity test with JDG 6.0.0.ER4 in REPL clustering mode:
starting from 1 node going to 3 nodes everything is allright
and then when 4th node is joining (viewId=4) the view installation takes more than 10 min.
Client load is 500
Data load is 5% of total heap.
Of course this is not easy to replicate with small load...
Again this will be tough to TRACE
Run with 10clients 10K dataload
Run with 500 clients 5% dataload TRACE log:
generated 7.8G of logs, didn't replicate the issue
times to install views are around 40 secs:
Another TRACE run that doesn't reproduce the problem:
Another non-trace run that doesn't reproduce the problem:
Dan Berindei <email@example.com> made a comment on jira ISPN-1933
Michal, it looks like your test is using standalone.xml from the master branch, which is outdated, and not the latest config in the prod-6.0.0 branch.
I updated the config we're using for the REPL tests:
In hyperion I ran the 4 node elasticity test 3 times and couldn't reproduce this anymore.
In edg lab this problem is still there:
and still only with REPL case, the DIST works with the same JGroups config:
Second run in edg lab didn't even get to 3 node cluster:
I'll try to run with trace logging
Reproduced with TRACE log:
this might be a framework problem. The test might have been ended sooner because it didn't ignore some expected exceptions during join/leave.
Consider this a false alarm until further notice
Michal, I don't think it's a false alarm, but it's a different problem. I'm seeing these errors in your TRACE log:
08:10:29,004 ERROR [org.infinispan.statetransfer.StateTransferLockImpl] Trying to release state transfer shared lock without acquiring it first: java.lang.Exception
08:10:29,005 ERROR [org.infinispan.statetransfer.StateTransferLockImpl] Trying to release state transfer shared lock without acquiring it first: java.lang.Exception
They are certainly not expected, so I'm looking into it.
Oh It's a new one you're right, I thought it's
therefore I didn't report it, but the stack trace is different there.
Did you create JIRA for that ?
It occurs only after I start stopping the servers (or killing them).
In the log you can see this event marked by "Test will now stop the server", that's written to the log output file right before I send the kill signal to JBoss.
I think I meant this one: https://issues.jboss.org/browse/ISPN-1704
So now I don't know why I didn't reopen it :-)
For some reason I thought it's something expected/insignificant, it's good that you spotted that.
It looks very much like ISPN-1704, but that one happened on the surviving nodes - this one seems to happen on the nodes that you're killing. So I think it might be worth a separate bug after all.
After fixing a problem in test framework, tests run till the end, and all state tansfers complete under 5sec.
Created attachment 572732 [details]
view installation times after framework fix
adding new view installation times after framework fix
Michal Linhard <firstname.lastname@example.org> updated the status of jira ISPN-1933 to Closed
Michal Linhard <email@example.com> made a comment on jira ISPN-1933
This was a problem in the test itself. The state transfer didn't really last more than 10 min.