Bug 1104227 - Incorrent jndi of datasource provided to jdbc driver for recovery when more datasources exist
Summary: Incorrent jndi of datasource provided to jdbc driver for recovery when more d...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Transaction Manager
Version: 6.3.0,6.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: EAP 6.4.0
Assignee: tom.jenkinson
QA Contact: Ondrej Chaloupka
URL:
Whiteboard:
: 1091425 1146450 1152783 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-06-03 14:38 UTC by Ondrej Chaloupka
Modified: 2017-10-10 00:23 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
In this release of JBoss EAP 6, there is an issue during passing JNDI of a datasource during a recovery process when more than one datasource is defined. Those datasources belong to one physical machine but each of them points to different database (different user is used for connecting). This causes the recovery to fail with an exception and finishes with the following error: ---- WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: oracle.jdbc.xa.OracleXAException ---- This issue is expected to be resolved in a future release.
Clone Of:
Environment:
Last Closed: 2015-02-08 22:33:23 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
server.log with oracle trace switched on (1.23 MB, application/x-zip-compressed)
2014-06-03 14:38 UTC, Ondrej Chaloupka
no flags Details
server.log for postgresql (1.22 MB, text/plain)
2014-09-03 15:01 UTC, Ondrej Chaloupka
no flags Details
standalone-full.xml (29.25 KB, text/plain)
2014-09-03 15:44 UTC, Ondrej Chaloupka
no flags Details
JPAMultiXACMRCrashRecoveryTestCase_commitHaltRev_jta_server log+standalone conf (70.93 KB, application/zip)
2015-02-03 13:43 UTC, Ondrej Chaloupka
no flags Details
standalone-full.xml with 3xa datasources defined (30.74 KB, application/xml)
2015-02-03 14:31 UTC, Ondrej Chaloupka
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1091425 0 unspecified CLOSED 1 CMR and 2 XA crashrec: error in log while recovering 2021-02-22 00:41:40 UTC

Internal Links: 1091425

Description Ondrej Chaloupka 2014-06-03 14:38:34 UTC
Created attachment 901811 [details]
server.log with oracle trace switched on

There is an issue during passing jndi of datasource during recovery process when more than one datasource is defined.
Those datasources belongs to one physical machine but each of them points to different database (different user is used for connecting).

This causes that recovery ends with exception [1] and finishes with error. The log is from test running against Oracle 12c.

Scenario:
- I have defined 2 XA datasources
- I do an operation that causes XA commit where 1 of the datasource is part of the transaction
- Server is crashed at the end of prepare phase
- Transaction periodic Recovery is run
- Recovery tries to rollback the transaction based on info from database (TM does not have any info in its local log)
- TM passes the XID to be rollbacked to JCA and JCA then to jdbc driver 
- JCA switches jndi of first XA datasource for jndi of second XA datasource
- recovery fails and the transaction is not recovered

If you check the code the XAResourceWrapperImpl.rollback(Xid) method is used to rollback the Xid.
https://github.com/ironjacamar/ironjacamar/blob/1.0/core/src/main/java/org/jboss
/jca/core/tx/jbossts/XAResourceWrapperImpl.java#L185
Method convertXid(Xid) is used to a wrapper being created around it (XidWrapperImpl).
But at the time of wrapper creation:
https://github.com/ironjacamar/ironjacamar/blob/1.0/core/src/main/java/org/jboss/jca/core/tx/jbossts/XAResourceWrapperImpl.java#L251
there is taken variable 'this.jndiName' which points to second XA datasource rather than to first one.


[1]
WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: oracle.jdbc.xa.OracleXAException
 at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1213)
 at oracle.jdbc.xa.client.OracleXAResource.rollback(OracleXAResource.java:979)
 at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.rollback(XAManagedConnection.java:346)
 at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.rollback(XAResourceWrapperImpl.java:186)
 at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:755) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]
 at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:661) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]
 at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:431) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]
 at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:212) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.20.Final-redhat-1.jar:4.17.20.Final-redhat-1]
Caused by: java.sql.SQLException: ORA-24774: cannot switch to specified transaction
 at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
 at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:392)
 at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:385)
 at oracle.jdbc.driver.T4CTTIfun.processError(T4CTTIfun.java:938)
 at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:655)
 at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:249)
 at oracle.jdbc.driver.T4CTTIOtxen.doOTXEN(T4CTTIOtxen.java:166)
 at oracle.jdbc.driver.T4CXAResource.doTransaction(T4CXAResource.java:746)
 at oracle.jdbc.driver.T4CXAResource.doRollback(T4CXAResource.java:624)
 at oracle.jdbc.xa.client.OracleXAResource.rollback(OracleXAResource.java:974)
 ... 8 more

Comment 2 Ondrej Chaloupka 2014-06-03 14:59:11 UTC
Just a not to server.log file. As it's a bit bigger file for orientation it's good to start at:
15:58:07,117 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) XAResourceOrphanFilter
where the bottom-up recovery votes for transaction rollback.
And it could be seen that the XidWrapperImpl contains link to CrashRecoveryDS2 instead of correct CrashRecoveryDS1.

Comment 3 Ondrej Chaloupka 2014-09-03 12:02:33 UTC
Hi Jesper,

as this is a JCA issue I've assigned this issue back on you. 

If I can help you with investigation or explain how to run the reproducer, I'll be happy to help. But I do not have deep insight to JCA code to dare to work on a fix.

Ondra

Comment 4 Jesper Pedersen 2014-09-03 12:23:17 UTC
Yes, please provide a summery for all levels - note, that Oracle JDBC is buggy in the Xid layer, so you may want to verify with <wrap-xaresource>false</>

Comment 5 Ondrej Chaloupka 2014-09-03 15:01:30 UTC
Created attachment 934130 [details]
server.log for postgresql

We run this on most of the databases that we have in our lab and it behaves in the same way.
I tried to use wrap-xa-resource for all XA resources and the issue remains.

I'm adding log file from run with PostgreSQL. From log it could be seen that for recovery it's used user crashrec (time 16:56:34,210) but the transaction was created by different user (crashrec2 or crashrec3) and postgresql throws permission deny exception.
The idea about cause of the issue is put in #c0

Comment 7 Jesper Pedersen 2014-09-03 15:34:46 UTC
Attach full configuration used - crashrec2 and crashrec3 are used as recovery credentials

Comment 8 Ondrej Chaloupka 2014-09-03 15:44:36 UTC
Created attachment 934140 [details]
standalone-full.xml

You can get the used config by yourself if you run our recovery test case.

Comment 9 Jesper Pedersen 2014-09-03 15:54:00 UTC
CrashRecoveryDS1 => crashrec2 and CrashRecoveryDS2 => crashrec3 as configured.

Assign it to Martin Simka if you have further information.

Comment 10 Ondrej Chaloupka 2014-09-04 09:13:19 UTC
If you mind naming of resources and db account names then it's intentional. It depends on account names that are provided in databases in our lab.

I hope that it could be enough information to find the root of the issue.
Logs are attached and the analysis of the problem is explained in the first comment.

Comment 12 Hayk Hovsepyan 2014-09-05 07:24:22 UTC
*** Bug 1091425 has been marked as a duplicate of this bug. ***

Comment 13 Gytis Trikleris 2014-10-07 13:46:43 UTC
*** Bug 1146450 has been marked as a duplicate of this bug. ***

Comment 14 Ondrej Chaloupka 2014-10-15 14:10:25 UTC
*** Bug 1152783 has been marked as a duplicate of this bug. ***

Comment 17 Ondrej Chaloupka 2015-01-30 15:19:23 UTC
I'm reassigning this to Transaction Manager component as I spent some time on investigation of this issue and I think that this is a real issue but rather on layer of TM than the layer of JCA.

Our testcase:
- 1 CMR resource directed, 2 XA resources, 1 XA test resource
- CMR and XA resources are based on datasources where each one points to the same database server but with different db and credentials
- prepare CMR
- prepare test XA
- prepare db XA 1
- prepare db XA 2
- commit CMR
- crash server
- recovery commit of test XA
- crash server
- recovery commit of db XA 1
- crash server
- recovery commit of db XA 2

What's happening is that recovery XID from db XA is not correctly joit with XAResource that have permissions to finish prepared transaction.

(in postgresql it's like
- select * from pg_prepared_xacts
- commit prepared 'trx3'
ERROR:  permission denied to finish prepared transaction
HINT:  Must be superuser or the user that prepared the transaction.)

What I understand from the code the com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule at method getNewXAResource the resources that could be used for recovery is filtered
https://github.com/jbosstm/narayana/blob/d32f23f9265fe399c99302fb18647996a4a080e3/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/recovery/arjunacore/XARecoveryModule.java#L238
The call is passed to com.arjuna.ats.internal.jta.recovery.arjunacore.RecoveryXids at method updateIfEquivalentRM
https://github.com/jbosstm/narayana/blob/d32f23f9265fe399c99302fb18647996a4a080e3/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/recovery/arjunacore/RecoveryXids.java#L199
there is logic that verifies xid - contains(xids[i]). It verifies that xid for XAResource of db XA 1 contains the xid of db XA 2. This leads to the fact that true is returned (isSameRM method is overriden and using datasource parameter is-same-rm-override does not matter). 

Then the test experiences the error (for postgresql):
TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) BasicAction::doCommit (XAResourceRecord < resource:XAResourceWrapperImpl@4a44a3c2[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@605b3ba5 pad=false overrideRmValue=false productName=PostgreSQL productVersion=9.3.5 jndiName=java:jboss/xa-datasources/CrashRecoveryDS2], txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a000006:-c8b2006:54cb93cf:45, node_name=1, branch_uid=0:ffff0a000006:-c8b2006:54cb93cf:54, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS1 >, heuristic: TwoPhaseOutcome.FINISH_OK, product: PostgreSQL/9.3.5, jndiName: java:jboss/xa-datasources/CrashRecoveryDS1 com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@3355d719 >)
TRACE [com.arjuna.ats.jta] (Periodic Recovery) XAResourceRecord.topLevelCommit for XAResourceRecord < resource:XAResourceWrapperImpl@4a44a3c2[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@605b3ba5 pad=false overrideRmValue=false productName=PostgreSQL productVersion=9.3.5 jndiName=java:jboss/xa-datasources/CrashRecoveryDS2], txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a000006:-c8b2006:54cb93cf:45, node_name=1, branch_uid=0:ffff0a000006:-c8b2006:54cb93cf:54, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS1 >, heuristic: TwoPhaseOutcome.FINISH_OK, product: PostgreSQL/9.3.5, jndiName: java:jboss/xa-datasources/CrashRecoveryDS1 com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@3355d719 >, record id=0:ffff0a000006:-c8b2006:54cb93cf:55
WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016036: commit on < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a000006:-c8b2006:54cb93cf:45, node_name=1, branch_uid=0:ffff0a000006:-c8b2006:54cb93cf:54, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS1 > (XAResourceWrapperImpl@4a44a3c2[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@605b3ba5 pad=false overrideRmValue=false productName=PostgreSQL productVersion=9.3.5 jndiName=java:jboss/xa-datasources/CrashRecoveryDS2]) failed with exception $XAException.XAER_RMERR: org.postgresql.xa.PGXAException: Error committing prepared transaction
 at org.postgresql.xa.PGXAConnection.commitPrepared(PGXAConnection.java:511)
 at org.postgresql.xa.PGXAConnection.commit(PGXAConnection.java:434)
 at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.commit(XAManagedConnection.java:338)
 at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(XAResourceWrapperImpl.java:107)
 at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:464) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2754) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2670) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1828) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.arjuna.recovery.RecoverAtomicAction.replayPhase2(RecoverAtomicAction.java:71) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.doRecoverTransaction(AtomicActionRecoveryModule.java:152) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.processTransactionsStatus(AtomicActionRecoveryModule.java:251) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkSecondPass(AtomicActionRecoveryModule.java:109) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:793) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:375) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
Caused by: org.postgresql.util.PSQLException: ERROR: permission denied to finish prepared transaction
  Hint: Must be superuser or the user that prepared the transaction.
 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
 at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
 at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
 at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)
 at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)
 at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:333)
 at org.postgresql.xa.PGXAConnection.commitPrepared(PGXAConnection.java:501)
 ... 13 more

Comment 19 Michael 2015-02-02 12:08:34 UTC
Are you setting the recovery-credential parameter on the datasource you want us to use for recovery - I could not find it the the config file (attachment.cgi?id=934140). 

The setting is defined in https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.3/html/Administration_and_Configuration_Guide/sect-Datasource_Configuration.html#Datasource_Parameters1

Comment 20 Ondrej Chaloupka 2015-02-03 13:43:46 UTC
Created attachment 987579 [details]
JPAMultiXACMRCrashRecoveryTestCase_commitHaltRev_jta_server log+standalone conf

Hi Mike,

I've added the recovery credentials to all datasources expected to be recovered (DS1 and DS2) and it does not help.
You can see the standalone-full.xml as attachment here.

I spend a bit time to check the test and confirm if it isn't problem at test side. I'm convinced that's not. I've added new test where I get rid off all that server killing and there is just one crash.

1. prepare CMR
2. prepare TestXA
3. prepare DB1
4. prepare DB2
5. commit CMR
6. commit TestXA
7. kill JVM/crash server
8. restart of server
9. recovery of DB1 and DB2 - both resources should be commited at the end

The problem here occurs. The XAResource for commiting data to DB1 is used for commiting data to DB2 to (DB1 and DB2 stand for database names on one machine).

I've played a bit with code but I'm not too much familiar with it and I haven't done any movement forward.

I think that there could be a problem in XARecoveryModule and in list _xidScans where where data is filtered in different cases based on similarity of xid.

Comment 22 tom.jenkinson 2015-02-03 14:06:14 UTC
Does the same thing happen if you just have DB1 and DB2 in the tx or does it seem to need the CMR?

Comment 23 Ondrej Chaloupka 2015-02-03 14:31:36 UTC
Created attachment 987607 [details]
standalone-full.xml with 3xa datasources defined

The same happens for just XA datasources as well.

We have another test that works in this way:
1. prepare TestXA
2. prepare DB
6. commit TestXA
7. kill JVM/crash server
8. restart of server
9. recovery of DB - resource should be recovered
but there are defined 2 more datasources in standalone-full.xml

What's happen is that db resource should be recovered but it is taken different datasource configured in standalone conf (such datasource is not used in test execution) and the recovery run fails.

Comment 25 tom.jenkinson 2015-02-03 14:48:33 UTC
Thanks Ondra, that should simplify the reproduction of this one. That being said I think we are going to end up with the situation where an XAR returns an Xid that it does not have the permissions to complete which I expect to be a permissions issue on the configured recovery user. I would expect that user to have permissions to complete branches for any user. Lets wait for Mike to verify that is what is happening and then discuss what we can do.

Comment 26 Ondrej Chaloupka 2015-02-03 15:36:06 UTC
Yeap, you are right Tom. If it's meant that user defined to process recovery should have permissions to recover data of any user then it will work.
Just e.g. for postgresql, I think, there is no way how to set permissions to finish prepared transaction of other user if you are not a superuser.

I was now searching the web and I found out the thread where the same issue is discussed by Mike
https://developer.jboss.org/thread/242834

The conversation assumes that the issue is in the jdbc driver. I've checked it and it could be so. I need to check with changed jdbc driver by myself.

In fact I've tried the same scenario on Oracle DB and there it behaves differently.
The last mentioned case passes but the CMR scenario (where all datasources are involved in some action) fails with
ORA-24774: cannot switch to specified transaction

Here the thing is similar as for PostgreSQL. I was not successful to find way how to set permissions for user to finished transactions of another user.


WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016036: commit on < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a280468:-7dc13464:54d0e888:47, node_name=1, branch_uid=0:ffff0a280468:-7dc13464:54d0e888:5b, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS2 > (XAResourceWrapperImpl@5ed5b9e4[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@627589ae pad=false overrideRmValue=false productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/xa-datasources/CrashRecoveryDS1]) failed with exception $XAException.XAER_RMERR: oracle.jdbc.xa.OracleXAException
 at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1135)
 at oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:565)
 at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.commit(XAManagedConnection.java:338)
 at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(XAResourceWrapperImpl.java:107)
 at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit(XAResourceRecord.java:464) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2754) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(BasicAction.java:2670) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(BasicAction.java:1828) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.arjuna.recovery.RecoverAtomicAction.replayPhase2(RecoverAtomicAction.java:71) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.doRecoverTransaction(AtomicActionRecoveryModule.java:152) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.processTransactionsStatus(AtomicActionRecoveryModule.java:251) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkSecondPass(AtomicActionRecoveryModule.java:109) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:793) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:375) [jbossjts-jacorb-4.17.29.Final-redhat-1.jar:4.17.29.Final-redhat-1]
Caused by: java.sql.SQLException: ORA-24774: cannot switch to specified transaction
 at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
 at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:389)
 at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:382)
 at oracle.jdbc.driver.T4CTTIfun.processError(T4CTTIfun.java:600)
 at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
 at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
 at oracle.jdbc.driver.T4CTTIOtxen.doOTXEN(T4CTTIOtxen.java:168)
 at oracle.jdbc.driver.T4CXAResource.doTransaction(T4CXAResource.java:746)
 at oracle.jdbc.driver.T4CXAResource.doCommit(T4CXAResource.java:422)
 at oracle.jdbc.xa.client.OracleXAResource.commit(OracleXAResource.java:560)
 ... 12 more

Comment 27 Ondrej Chaloupka 2015-02-04 10:12:55 UTC
I've tried to compile the newest postgresql jdbc driver
https://github.com/pgjdbc/pgjdbc/blob/master/org/postgresql/xa/PGXAConnection.java#L366
where the fix mentioned in discussion is included. That fixes the trouble of using incorrect datasource for recovery when one datasource is used in transaction and then some different one is used for recovery (despite the fact that datasource was not used - reproduced by JPAMultiXACrashRecoveryTestCase testcase).
But this does not fix the problem when more datasources are used to do some "real work" and then the both of them should be recovered (reproduced by JPAMultiXACMRCrashRecoveryTestCase#commitHaltRev, reproducer uses cmr resource but it's not the important point as mixing seems to happen in XARecoveryModule)

Comment 28 Michael 2015-02-04 11:02:46 UTC
The datasource used by the application (which looks like CrashRecoveryDS2 from your logs) may be, and often is, different from the one used during recovery (CrashRecoveryDS1). When we run recovery we use all the datasources that JCA registers with us and ask each of them for in doubt XIDs. Since DS1 and DS2 are connected to the same database they will return to us the same indoubt XIDs. So the recovery credentials on either datasource need to have the right permissions to manipulate the postgresql transaction tables (pg_prepared_xacts). Please will you verify whether or not this is the case?

If you need to disable recovery on a particular datasource use the "no-recovery" parameter.

Comment 29 Ondrej Chaloupka 2015-02-08 22:33:23 UTC
Hi Mike,

I wanted to  respond to you earilier but I needed a bit more time for a new round of investigation.

And thank you for pointing me to the right way. You are absolutely right.  I didn't recall that all xids are taken with recover call and that it could be the source of the problems.

I'm closing this as not a bug as it's
a) a problem of  jdbc driver (to return xids belong to the active database)
b) a problem of configuration (to set account for recovery that has privileges to commit/rollback prepared transactions).

And as I did a bit more investigation I focused mainly on PostgreSQL and Oracle.
I haven't found way how to set privilege to commit transaction of different user in PosgreSQL. But the new sql driver and the change mentioned in comment #c27 should help in this.
Oracle does have such setting for privileges and it should be "FORCE ANY TRANSACTION"

In any case of problem I would create a new bugzilla.

Thanks for your patience
Ondra

Comment 30 Ondrej Chaloupka 2015-03-11 09:11:02 UTC
Hi Scott,

I think that this issue finally should not be part of the release notes. Sorry for confusion here. When I closed this issue as not a bug I forgot to clear the requires_doc_text flag.

Is possible to leave this out of the release notes, please?

Thank you and sorry for inconveniences
Ondra

Comment 31 Scott Mumford 2015-03-11 23:50:26 UTC
Not a problem at all Ondrej. I should have checked the status before I included it.

I've marked it for exclusion now.


Note You need to log in before you can comment on or make changes to this bug.