Bug 900154 (JBPAPP6-493)

Ondrej ,

        I believe you and Paul are already aware of this.This is a tracker jira to verify testsuite stability in ER5

Hi Madhumita,

1) is it the same error as before?
2) if so, have you tried to increase the timeout?
3) if not, there are some other configurable timeouts, see the testsuite docs.

Either way, this is result of running the "prod" branch as it is now:
{code}
[INFO] JBoss Application Server Test Suite: Aggregator ... SUCCESS [22.884s]
[INFO] JBoss Application Server Test Suite: Integration .. SUCCESS [1.942s]
[INFO] JBoss Application Server Test Suite: Integration - Smoke  SUCCESS [1:38.630s]
[INFO] JBoss Application Server Test Suite: Integration - Basic  SUCCESS [12:18.438s]
[INFO] JBoss Application Server Test Suite: Integration - Clustering  SUCCESS [10:55.882s]
[INFO] JBoss Application Server Test Suite: Integration - IIOP  SUCCESS [21.210s]
[INFO] JBoss Application Server Test Suite: Integration - XTS  SUCCESS [17.163s]
[INFO] JBoss Application Server Test Suite: Integration - Multinode Tests  SUCCESS [23.226s]
[INFO] JBoss Application Server Test Suite: Integration - Manual Mode Tests  SUCCESS [3:54.841s]
[INFO] JBoss Application Server Test Suite: Compatibility Tests  SUCCESS [24.031s]
[INFO] JBoss Application Server Test Suite: Domain Mode Integration Tests  SUCCESS [2:33.664s]
[INFO] JBoss Application Server Test Suite: Benchmark Tests  SUCCESS [7.959s]
[INFO] JBoss Application Server Test Suite: Stress Tests . SUCCESS [1.210s]
{code}

Therefore, you need to increase timeout limit until it passes, or figure out what's wrong in QA lab.
In case you discover some error in the testsuite, please describe the issue and assign back to me.

I'm attaching an arror report I got while running testsuite sgainst ER4.1

Attachment: Added: org.jboss.as.test.smoke.messaging.client.messaging.MessagingClientTestCase.txt

Ondra changing timeouts with every ER cycle and going for trial and error with several automated builds does not seem like a sane idea.
Option should be to use a standard timeout value that works.

As for above stacktrace you printed I also did get to see clean runs as well but it was rare.
Are you getting stable runs everytime against prod repo used in ER4.1?If so what is the timeout value used?
Regarding QA lab the tests are still stable on ER3(apart from benchmark and stress failures).

We will be using this testsuite for one-off regression hence unstability is major concern and the reason i highlighted is it is alarming in ER4.1.

I am attaching detailed stacktrace for build failures (attached file unstablebuildER4.1)

I can see clustering tests are not failing for timeout but for certain test failures
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.10-redhat-1:test (ts.surefire.clust.multinode-manual-udp) on project jboss-as-ts-integ-clust: There are test failures...........................................................
..................................................................................................................................
Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures.

Ondra.....Surprising that the build is failing for test failures (instead of timeout as in basic target)

Attachment: Added: unstablebuildER4.1

1) Changing timeout limits is the only idea I have regarding builds hitting timing out.

2) What build are you referring to? I can quickly check if I can be of any help with failures you see.

3) Regarding MessagingClientTestCase. If test authors followed my advice and put some message into the exception, we would see that the problem is that queue was not removed after 
{code:java}
        op = new ModelNode();
        op.get("operation").set("remove");
        op.get("address").add("subsystem", "messaging");
        op.get("address").add("hornetq-server", "default");
        op.get("address").add("queue", queueName);
        applyUpdate(op, client);
{code}
I don't know whether that's supposed to be synchronous operation. Please refer to docs.
My guess is that it's async, and with low-performance NFS it causes race condition;  therefore the test needs to actively wait for the queue to be removed, with some timeout.

All right, credits for empty exception goes to Jeff, not QE's :)  I'm going to discuss with him.

please check stacktrace in attached file where clustering targets fail due to test failures.....
and exactly for your comment above this is where i want to stress....i never saw the same failure in several runs which Jan has pointed above.Infact I saw several test failures due to server already running or data source already exists etc....
referring https://issues.jboss.org/browse/JBPAPP-8448 ....
I believe proper test clean-up is real cause of unstability issues

Pull request for MessagingClientTestCase: https://github.com/jbossas/jboss-as/pull/1994

Link: Added: This issue depends JBPAPP-8368

This seems to have been resolved by JBPAPP-8368

I may have closed this one too soon, we had another timeout build after the successful build.
http://hudson.qa.jboss.com/hudson/job/JBoss-EAP-6.0.x-MEAD/347/

But that job didn't have `surefire.forked.process.timeout` at all.
I've added it, let's see.  Run #348.

Tadaa: https://hudson.qa.jboss.com/hudson/job/JBoss-EAP-6.0.x-MEAD/

I mean... Tadaa:  https://hudson.qa.jboss.com/hudson/job/JBoss-EAP-6.0.x-MEAD/348/

ER5 looks much better and I got couple of runs with no compilation failures ...closing this as of now will reopen if there is regression in future

Docs QE Status: Removed: NEW