Bug 1020953 - JTS qa test CrashRecovery12_Test03 fails running with HornetQ object store
JTS qa test CrashRecovery12_Test03 fails running with HornetQ object store
Status: VERIFIED
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Transaction Manager (Show other bugs)
6.2.0,6.2.4
Unspecified Unspecified
unspecified Severity medium
: GA
: EAP 6.3.0
Assigned To: Michael
Ondrej Chaloupka
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-18 11:15 EDT by Ondrej Chaloupka
Modified: 2017-10-09 20:22 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker JBTM-1980 Major Closed QA test suite failure on HQStore: CrashRecovery12_Test03 2017-03-06 02:56 EST

  None (edit)
Description Ondrej Chaloupka 2013-10-18 11:15:38 EDT
There are a failure at test crashrecovery12 on method CrashRecovery12_Test03. This was reported for narayana 5.0.0 but the error occurs in 4.17.12.Final as well (which is version for EAP 6.2.0.ER6).

Running like:
./build.sh clean install -Pall  -Dmaven.test.skip.exec=true
cd qa/
ant dist
ant -f run-tests.xml -Dtest.methods="CrashRecovery12_Test03" -Dtest=crashrecovery12 -Dprofile=hornetq test

Fail:
Testcase: CrashRecovery12_Test03 took 31.498 sec
 FAILED
task outcome0 printed Failed.
junit.framework.AssertionFailedError: task outcome0 printed Failed.
 at org.jboss.jbossts.qa.junit.TaskImpl$TaskReaderThread.checkPassFail(TaskImpl.java:671)
 at org.jboss.jbossts.qa.junit.TaskImpl.waitFor(TaskImpl.java:430)
 at org.jboss.jbossts.qa.junit.testgroup.TestGroup_crashrecovery12.CrashRecovery12_Test03(TestGroup_crashrecovery12.java:80)
 at org.jboss.jbossts.qa.junit.QATestNameRule$1.evaluate(QATestNameRule.java:89)
Comment 2 Michael 2013-10-25 06:29:15 EDT
Note that this failure is a bug in the test design itself and does not affect the functionality of narayana itself - my fix is to the test suite and I did not touch the code that goes into the product
Comment 3 JBoss JIRA Server 2013-11-07 08:49:28 EST
Gytis Trikleris <gytis@redhat.com> updated the status of jira JBTM-1980 to Closed
Comment 4 JBoss JIRA Server 2013-11-11 06:27:13 EST
Gytis Trikleris <gytis@redhat.com> updated the status of jira JBTM-1980 to Reopened
Comment 5 JBoss JIRA Server 2013-11-11 06:27:13 EST
Gytis Trikleris <gytis@redhat.com> made a comment on jira JBTM-1980

Looks like the same issue http://172.17.131.2/view/Narayana+BlackTie/job/narayana-hqstore-jacorb/22/TESTS=QA_JTS_JACORB,jdk=jdk7.latest,label=linux/
Comment 6 JBoss JIRA Server 2013-11-11 07:46:17 EST
Michael Musgrove <mmusgrov@redhat.com> updated the status of jira JBTM-1980 to Resolved
Comment 7 JBoss JIRA Server 2013-11-11 07:46:17 EST
Michael Musgrove <mmusgrov@redhat.com> made a comment on jira JBTM-1980

The referenced CI failure on the previous comment affects 5.0.0.CR1 only so I am resolving the JIRA again since the problem is fixed on the 4.17 branch.

I have removed 5.0.0.CR1 from the fix for field. See linked JIRA to track the problem on the 5.0.0.CR1 branch)
Comment 8 JBoss JIRA Server 2013-11-11 11:03:33 EST
Michael Musgrove <mmusgrov@redhat.com> updated the status of jira JBTM-1980 to Reopened
Comment 9 JBoss JIRA Server 2013-11-11 11:03:33 EST
Michael Musgrove <mmusgrov@redhat.com> made a comment on jira JBTM-1980

The fix is in both branches (4.17 and master) so even though the failure was seen against master it could occur in 4.17 also. 

From the logs it looks like the test task is being reaped after 58 seconds (although the test config says it should only be reaped afater 240 seconds). This means that my fix to overcome port recycling did not have sufficient time to take effect.

My line of investigation is to determine why the task was reaped early.
Comment 10 JBoss JIRA Server 2013-12-03 09:14:50 EST
Michael Musgrove <mmusgrov@redhat.com> updated the status of jira JBTM-1980 to Resolved
Comment 11 JBoss JIRA Server 2013-12-03 09:14:50 EST
Michael Musgrove <mmusgrov@redhat.com> made a comment on jira JBTM-1980

I am unable to reproduce this issue - the PR adds extra logging information to help diagnose if it does ruccur.
Comment 12 JBoss JIRA Server 2013-12-05 04:55:38 EST
Michael Musgrove <mmusgrov@redhat.com> updated the status of jira JBTM-1980 to Reopened
Comment 13 JBoss JIRA Server 2013-12-05 04:55:38 EST
Michael Musgrove <mmusgrov@redhat.com> made a comment on jira JBTM-1980

Recurrence of the same issue (with extra trace)
Comment 14 JBoss JIRA Server 2013-12-05 11:50:36 EST
Michael Musgrove <mmusgrov@redhat.com> made a comment on jira JBTM-1980

The task that checks for recovery runs before the task that generates the crash scenario has finished. I have added a fix that delays the second task until the first task is ready.
I'm testing it with a dedicated job: http://172.17.131.2/job/JBTM-1980-hqstore/ to test my fix
Comment 15 JBoss JIRA Server 2013-12-08 12:59:12 EST
Michael Musgrove <mmusgrov@redhat.com> updated the status of jira JBTM-1980 to Resolved
Comment 16 JBoss JIRA Server 2013-12-08 12:59:12 EST
Michael Musgrove <mmusgrov@redhat.com> made a comment on jira JBTM-1980

The latest failure is because the client task that creates the crash record takes a while to start (because of port recycling) and the checker task that validates recovery occurred runs before the client has had time to generate the crash rec.

The fix is to wait for the client task to become ready before launching the checker task. I have only applied the fix to CrashRecovery12_Test03. If we ever get similar failures for other tests then we should apply the same fix as necessary.
Comment 17 JBoss JIRA Server 2013-12-11 12:37:06 EST
Tom Jenkinson <tom.jenkinson@redhat.com> updated the status of jira JBTM-1980 to Closed
Comment 18 Michael 2014-08-21 06:36:48 EDT
Can we close this one yet or are you still seeing the failure?
Comment 19 Ondrej Chaloupka 2014-08-21 08:26:28 EDT
Hi, 
I can't see the problem for 4.17.21.Final which is under 6.3.0.GA. So setting as verified from QA point of view.

Thanks
Ondra

Note You need to log in before you can comment on or make changes to this bug.