Bug 1078297

Summary: Failing qa tests: crashrecovery12 CrashRecovery12_Test02 when run with jdbc object store
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Ondrej Chaloupka <ochaloup>
Component: Transaction ManagerAssignee: Michael <mmusgrov>
Status: CLOSED CURRENTRELEASE QA Contact: Ondrej Chaloupka <ochaloup>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.3.0CC: hhovsepy
Target Milestone: ---   
Target Release: EAP 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Test output
none
client0_output.txt
none
emptyObjectStore_output.txt
none
outcome0_output.txt
none
server0_output.txt
none
setup0_output.txt none

Description Ondrej Chaloupka 2014-03-19 14:48:10 UTC
Created attachment 876372 [details]
Test output

There are failures for qa tests when running with jdbc object store (failing consistently on all tested dbs):
crashrecovery12 CrashRecovery12_Test02
crashrecovery12 CrashRecovery12_Test03
crashrecovery12 CrashRecovery12_Test04
crashrecovery12 CrashRecovery12_Test05
crashrecovery12 CrashRecovery12_Test06
crashrecovery12 CrashRecovery12_Test07

Running on Narayana 4.17.17.Final.

Comment 1 Ondrej Chaloupka 2014-03-19 14:48:51 UTC
Created attachment 876376 [details]
client0_output.txt

Comment 2 Ondrej Chaloupka 2014-03-19 14:49:18 UTC
Created attachment 876377 [details]
emptyObjectStore_output.txt

Comment 3 Ondrej Chaloupka 2014-03-19 14:49:45 UTC
Created attachment 876379 [details]
outcome0_output.txt

Comment 4 Ondrej Chaloupka 2014-03-19 14:50:03 UTC
Created attachment 876380 [details]
server0_output.txt

Comment 5 Ondrej Chaloupka 2014-03-19 14:50:22 UTC
Created attachment 876382 [details]
setup0_output.txt

Comment 7 Michael 2014-03-20 17:38:57 UTC
The failure is because the test is timing out waiting for the recovery system to recover failed transactions. I have seen this before on very slow connections to the db hosting the transaction logs. 

Is this something that is reproducible. If not then I would be tempted to close as cannot reproduce. Note that this is safe because the logs are still in the db and the recovery system would eventually replay the pending transactions.

Comment 8 Ondrej Chaloupka 2014-03-21 07:52:29 UTC
Hi Mike,

ok, I see. So is there some way how to increase the timeout or something?
The a bit strange thing is that this was not happening in previous testing cycle what I can say. But there could be some changes in our networking infrastructure.

What I can say it's easily reproducible if you try to run the test in the way that's described in comment #6. There are connection to our testing oracle database. But this happening for any database in our lab. I've tested against my local postgres installation:
ant -f run-tests.xml -Dtest.name=crashrecovery12 -Dtest.methods="CrashRecovery12_Test02" -Dprofile=postgres -Djdbc.db.url="localhost" -Djdbc.db.name=crashrec -Djdbc.db.user=crashrec -Djdbc.db.password=crashrec -Djdbc.db.port=5432 onetest

And fails are consistent there too.

Comment 11 JBoss JIRA Server 2014-04-09 15:03:30 UTC
Tom Jenkinson <tom.jenkinson> updated the status of jira JBTM-2133 to Closed

Comment 12 JBoss JIRA Server 2014-04-09 15:03:33 UTC
Tom Jenkinson <tom.jenkinson> updated the status of jira JBTM-2130 to Closed

Comment 13 Ondrej Chaloupka 2014-05-28 13:50:09 UTC
I've tested this with EAP 6.3.0.ER5 with version 4.17.20.Final. The failures mentioned here are still valid.

Comment 14 Ondrej Chaloupka 2014-07-08 13:54:26 UTC
The same for EAP 6.3.0.ER9 on 4.17.21.Final.

Comment 15 Ondrej Chaloupka 2014-08-22 06:42:05 UTC
We discussed this with Mike and I found out that issue was already solved by JBTM-2130. 
The remaining fails were only test issue.

There were two things - as the test with jdbc object store was not working I set too big number of MFACTOR (about 40) and it causes that this test were failing. When I put it to something normal like 3 test started to pass.
Then the reproducer on my local machine was not working as I did set MFACTOR but I run it manually and I didn't consider that the system variable MFACTOR is recognized just when using narayana.sh script. When running by hand the TaskImpl.properties are needed to be change directly (e.g. COMMAND_LINE_12=-DCoreEnvironmentBean.timeoutFactor=2, COMMAND_LINE_13=-DCoordinatorEnvironmentBean.defaultTimeout=240)

Setting as verified on Naryana 4.17.21.Final (what is version of EAP 6.3.0.GA)

Thanks and sorry Mike
o.

Jenkins job tested here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-62-jbossts-qa-test-jdbc-store-bz1078297