Bug 1080035 - Inconsistency for recovery when db connection fails for Oracle database when running on JTS
Summary: Inconsistency for recovery when db connection fails for Oracle database when ...
Status: VERIFIED
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Transaction Manager
Version: 6.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: DR12
: EAP 6.4.0
Assignee: Gytis Trikleris
QA Contact: Hayk Hovsepyan
URL:
Whiteboard:
Keywords:
Depends On: 1143956
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-24 14:46 UTC by Ondrej Chaloupka
Modified: 2018-12-07 22:33 UTC (History)
5 users (show)

(edit)
Clone Of:
(edit)
Last Closed:


Attachments (Terms of Use)
server.log (1.42 MB, text/x-log)
2014-03-24 14:46 UTC, Ondrej Chaloupka
no flags Details
JPAProxyCrashRecoveryTestCase_prepareHalt_jts_server.log.html (2.66 MB, text/html)
2014-07-01 15:13 UTC, Ondrej Chaloupka
no flags Details
server.log with -Dno.recovery.scan (1.68 MB, text/x-log)
2014-08-13 12:37 UTC, Ondrej Chaloupka
no flags Details
server.db2-10.log (1.63 MB, text/plain)
2014-09-02 13:24 UTC, Ondrej Chaloupka
no flags Details
server.log for oracle as html page (2.30 MB, text/html)
2014-09-05 07:28 UTC, Ondrej Chaloupka
no flags Details
server.log for oracle as html page (2.30 MB, text/html)
2014-09-05 07:43 UTC, Ondrej Chaloupka
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker JBTM-2255 Major Closed Do not return StatusCommiting, if transaction was commited by the original transaction manager 2017-11-22 12:27 UTC
Red Hat Bugzilla 1080140 None None None Never

Internal Trackers: 1080140

Description Ondrej Chaloupka 2014-03-24 14:46:28 UTC
Created attachment 878088 [details]
server.log

I'm hitting inconsistency for crash recovery when JTS is used. When using Oracle database and connection fails before commit is done the recovery finishes with rollback but the first resource was in fact committed.

Scenario: prepareHalt
Steps:
a. enlistment jdbc xa resource
b. enlistment test xa resource
c. prepare jdbc xa resource
d. killing connection to database
e. prepare test xa resource (no db connection)
f. commit jdbc xa resource -> failing as connection is down
f-a. jdbc xa resource returns XAException.XAER_RMFAIL
g. commiting test xa resource
h. start the connection to database
i. start recovery
As it's mentioned under issue bz1077216 there is needed to run 3 rounds of recovery to go through and for all the xids being recovered. But during recovery the db resource is rollbacked instead of being commited as the test xa resource was after the connection fell down.

This has to be run with Oracle database as other ones that were tested (e.g. PostgreSQL) return XAException.XAER_RMERR which causes the transaction being aborted and so the test xa resource is rolled back which means correct behavior for following recovery process.

Comment 5 Ondrej Chaloupka 2014-07-01 15:13:27 UTC
Created attachment 913790 [details]
JPAProxyCrashRecoveryTestCase_prepareHalt_jts_server.log.html

Hi Mike,

I've checked this issue and it seems to me being trouble somewhere. I'm not sure whether it's problem of TM or jdbc driver but the fact is that when the scenario is run on Oracle then the result seems to bring inconsistency in data.

TM tries to do commit on XA resource during recovery but there is still returned finish error [1]. And at the end rollback is called.

The same testcase works fine with JTA.

I'm not sure about this issue but I think that it should be part of release notes.

Ondra
ps. adding server log as attachment. this time in html format as I discovered  vim feature of exporting syntax highlihting to html.


[1]
TRACE [com.arjuna.ats.jts] (Periodic Recovery) ExtendedResourceRecord: Successfully stringed to object, next try to narrow
TRACE [com.arjuna.ats.jts] (Periodic Recovery) ExtendedResourceRecord: Failed to narrow to ArjunaSubtranAwareResource
TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) BasicAction.doCommit for 0:ffff7f000001:6e82ae10:53b29ee8:3a received TwoPhaseOutcome.FINISH_ERROR from class com.arjuna.ats.internal.jts.resources.ExtendedResourceRecord
TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) RecordList::insert(RecordList: empty) : appending /StateManager/AbstractRecord/ExtendedResourceRecord for 0:ffff7f000001:6e82ae10:53b29ee8:42
TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) BasicAction::doCommit() result for action-id (0:ffff7f000001:6e82ae10:53b29ee8:3a) on record id: (0:ffff7f000001:6e82ae10:53b29ee8:42) is (TwoPhaseOutcome.FINISH_ERROR) node id: (1)

Comment 8 Ondrej Chaloupka 2014-08-13 12:37:48 UTC
Created attachment 926404 [details]
server.log with -Dno.recovery.scan

Comment 9 Ondrej Chaloupka 2014-09-02 13:24:37 UTC
Created attachment 933761 [details]
server.db2-10.log

I was trying this with DB2 database (db2 9.7 and 10) and it suffers with this issue as well. Just the returned XA exception (after connection is down) is XAException.XA_RETRY (Oracle throws XAException.XAER_RMFAIL)

Comment 11 Ondrej Chaloupka 2014-09-05 07:28:24 UTC
Created attachment 934693 [details]
server.log for oracle as html page

Hi Gytis,

I was thinking to add some more information on this issue. I enhanced the server log with some comments. The comments are introduced by >>>>>.

I hope that could help you to understand the flow of our test case.

Thanks 
Ondra

Comment 12 Ondrej Chaloupka 2014-09-05 07:43:28 UTC
Created attachment 934695 [details]
server.log for oracle as html page

Comment 13 Gytis Trikleris 2014-09-05 09:31:15 UTC
Thanks Ondra. Did you commit it to any branch for me to use?

Comment 14 Ondrej Chaloupka 2014-09-05 10:01:17 UTC
Hi Gytis,

if you mean the way how to debug then it's already available in 6.3.0.
If you mean the enhanced report then it's just notes manually added to the log and processed by VI.
If you mean something else then probably not.

:)
Ondra

Comment 15 Gytis Trikleris 2014-09-05 10:04:42 UTC
I mean the enhanced report :) thanks for it, it makes it easier to read the log.

Comment 16 Gytis Trikleris 2014-09-17 14:38:04 UTC
I've merged the fix for this to https://github.com/jbosstm/narayana/tree/4.17. Now need to wait for JBossTS 4.17.23.Final to be released.

Comment 17 JBoss JIRA Server 2014-09-18 08:15:52 UTC
Mark Little <mark.little@jboss.com> updated the status of jira JBTM-2255 to Reopened

Comment 18 Gytis Trikleris 2014-09-18 08:37:14 UTC
Fix mentioned in the comment 16 was reverted, since this isn't the best option to solve the problem as explained in the JIRA.

Comment 19 tom.jenkinson 2014-09-26 11:10:36 UTC
Re-fixed in upstream

Comment 22 tom.jenkinson 2014-12-05 12:37:39 UTC
Went into 4.17.23

Comment 23 Hayk Hovsepyan 2014-12-05 14:35:16 UTC
Verified on revision EAP 6.4.0.DR12

Comment 24 JBoss JIRA Server 2014-12-11 15:54:01 UTC
Tom Jenkinson <tom.jenkinson@redhat.com> updated the status of jira JBTM-2255 to Closed


Note You need to log in before you can comment on or make changes to this bug.