Bug 1107582

Summary: [RFE] EAP6-17 CMR recovery could result in infinite cycle for Sybase database
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Ondrej Chaloupka <ochaloup>
Component: Transaction ManagerAssignee: Michael <mmusgrov>
Status: CLOSED CURRENTRELEASE QA Contact: Ondrej Chaloupka <ochaloup>
Severity: urgent Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.3.0CC: hhovsepy, jkudrnac, kkhan, myarboro
Target Milestone: ER8   
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-28 15:38:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1108152, 1108251    
Bug Blocks:    

Description Ondrej Chaloupka 2014-06-10 09:21:19 UTC
In case that connection is down then Sybase database throws SQL connection for preparing statement which causes that CommitMarkableResourceRecordRecoveryModule could ends in infinite loop.

I refer to the line of method delete(String jndiName, List<Xid> completedXids)
PreparedStatement prepareStatement = connection.prepareStatement(sql);
at
https://github.com/jbosstm/narayana/blob/4.17/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/recovery/arjunacore/CommitMarkableResourceRecordRecoveryModule.java#L631

The test where I can hit this is:
We use mock XA resource and CMR JPA resource.
Test stops (halts) DB connection at commit phase on TestXAResource before commit was done on JPA CMR resource. This means that commit of JPA CMR resource fails. The recovery then tries to fix the troubles (in fact at the end there will be no fix as the in-doubt transaction is in heuristic state). During this fixing it tried to run sql query DELETE xid from CMR 'xids' table which ends in infinite loop.

The exception which floods the log is:
TRACE [com.arjuna.ats.jta] (Periodic Recovery) TransactionSynchronizationRegistryImple.getTransactionKey
TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) Attempting to delete number of entries: 4
WARN  [com.arjuna.ats.arjuna] (Periodic Recovery) Could not handle the connection: java.sql.SQLException: JZ0C0: Connection is already closed.
 at com.sybase.jdbc4.jdbc.ErrorMessage.raiseError(ErrorMessage.java:765)
 at com.sybase.jdbc4.jdbc.SybConnection.checkConnection(SybConnection.java:3352)
 at com.sybase.jdbc4.jdbc.SybConnection.prepareStatement(SybConnection.java:1396)
 at com.sybase.jdbc4.jdbc.SybConnection.prepareStatement(SybConnection.java:1307)
 at com.sybase.jdbc4.jdbc.SybConnection.prepareStatement(SybConnection.java:1278)
 at com.sybase.jdbc4.jdbc.SybConnection.prepareStatement(SybConnection.java:2946)
 at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.doPrepareStatement(BaseWrapperManagedConnection.java:748)
 at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.prepareStatement(BaseWrapperManagedConnection.java:734)
 at org.jboss.jca.adapters.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:405)
 at com.arjuna.ats.internal.jta.recovery.arjunacore.CommitMarkableResourceRecordRecoveryModule.delete(CommitMarkableResourceRecordRecoveryModule.java:624) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]
 at com.arjuna.ats.internal.jta.recovery.arjunacore.CommitMarkableResourceRecordRecoveryModule.periodicWorkFirstPass(CommitMarkableResourceRecordRecoveryModule.java:177) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]

Comment 5 JBoss JIRA Server 2014-06-11 13:20:40 UTC
Tom Jenkinson <tom.jenkinson> updated the status of jira JBTM-2191 to Closed

Comment 6 Kabir Khan 2014-06-12 08:58:12 UTC
Fixed by component upgrade https://bugzilla.redhat.com/show_bug.cgi?id=1108152

Comment 7 Ondrej Chaloupka 2014-06-24 12:23:20 UTC
Verified on EAP 6.3.0.ER8.