Bug 1075008 - 3 XA resources crashrec fails
Summary: 3 XA resources crashrec fails
Keywords:
Status: CLOSED DUPLICATE of bug 1091425
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Transaction Manager
Version: 6.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Hayk Hovsepyan
QA Contact: Hayk Hovsepyan
Russell Dickenson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-03-11 10:57 UTC by Hayk Hovsepyan
Modified: 2014-10-25 11:32 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-07-08 14:27:44 UTC
Type: Bug


Attachments (Terms of Use)
FirstScenario (539.74 KB, text/x-log)
2014-03-11 10:57 UTC, Hayk Hovsepyan
no flags Details
SecondScenario (446.31 KB, text/x-log)
2014-03-11 10:59 UTC, Hayk Hovsepyan
no flags Details
MySql55Driver (823.33 KB, application/x-java-archive)
2014-03-21 16:00 UTC, Hayk Hovsepyan
no flags Details
PGSql2Driver (566.20 KB, application/x-java-archive)
2014-03-21 16:03 UTC, Hayk Hovsepyan
no flags Details
pgsqlLog (461.33 KB, text/x-log)
2014-05-13 13:27 UTC, Hayk Hovsepyan
no flags Details


Links
System ID Priority Status Summary Last Updated
JBoss Issue Tracker JBTM-2183 Optional Open Need finer grained control over which XAResource is used for recovery 2014-09-03 12:52:34 UTC

Description Hayk Hovsepyan 2014-03-11 10:57:19 UTC
Created attachment 873054 [details]
FirstScenario

Description of problem:


We did testing of 2 crashrec scenarios with XA (non LRCO related) and they failed, I will describe both of them and will attach log files.
Could you please review them so we can discuss it there any bug or it is test problem?

1. Register 3 pairs of XA resources in standalone-full.xml. All XA resources are connected to Oracle12c with different credentials. The names of DS pairs are CrashRecoveryDS/VerificationDS, CrashRecoveryDS1/VerificationDS1, CrashRecoveryDS2/VerificationDS2. In persistence.xml file of deployed JAR archive, all 3 XA datasources are added as persistence unit. Via Byteman, register TestXAResource "beforeEntityUpdate" and make it to crash on "commit' method exit. Call transactional SLSB which will update 1 entity in DB by using only CrashRecoveryDS , and it will crash. Then reload the server and call recovery. You will see that real XA resource is not recovered and error is in log (attached FirstScenario_jta_server.log). Seems like it mixes datasources registered in standalone or in persistence xml files.


2. Again like in first one, register 3 pairs of XA resources in standalone-full.xml. All XA resources are connected to Oracle12c with different credentials. The names of DS pairs are CrashRecoveryDS/VerificationDS, CrashRecoveryDS1/VerificationDS1, CrashRecoveryDS2/VerificationDS2. But this time, in persistence.xml file of deployed JAR archive, only 1 XA datasources are added as persistence unit. Via Byteman, register TestXAResource "beforeEntityUpdate" and make it to crash on "commit' method exit. Call transactional SLSB which will update 1 entity in DB by using only CrashRecoveryDS , and it will crash. Then reload the server and call recovery. As in first case, you will see that real XA resource is not recovered and error is in log (attached SecondScenario_jta_server.log).

Comment 1 Hayk Hovsepyan 2014-03-11 10:59:35 UTC
Created attachment 873055 [details]
SecondScenario

Comment 4 tom.jenkinson 2014-03-19 15:38:29 UTC
FYI, when I try to run this:

{code}
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 97.134 sec <<< FAILURE! - in org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase
commitHaltRevAfterResourceCommited(org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase)  Time elapsed: 96.012 sec  <<< ERROR!
org.jboss.arquillian.container.spi.client.container.DeploymentException: Cannot deploy: crashrecovery-3xa.jar
	at org.jboss.as.controller.client.helpers.standalone.impl.ServerDeploymentPlanResultFuture.getActionResult(ServerDeploymentPlanResultFuture.java:134)
	at org.jboss.as.controller.client.helpers.standalone.impl.ServerDeploymentPlanResultFuture.getResultFromNode(ServerDeploymentPlanResultFuture.java:123)
	at org.jboss.as.controller.client.helpers.standalone.impl.ServerDeploymentPlanResultFuture.get(ServerDeploymentPlanResultFuture.java:85)
	at org.jboss.as.controller.client.helpers.standalone.impl.ServerDeploymentPlanResultFuture.get(ServerDeploymentPlanResultFuture.java:42)
	at org.jboss.as.controller.client.helpers.standalone.ServerDeploymentHelper.deploy(ServerDeploymentHelper.java:50)
	at org.jboss.as.arquillian.container.ArchiveDeployer.deployInternal(ArchiveDeployer.java:77)
	at org.jboss.as.arquillian.container.ArchiveDeployer.deployInternal(ArchiveDeployer.java:64)
	at org.jboss.as.arquillian.container.ArchiveDeployer.deploy(ArchiveDeployer.java:46)
	at org.jboss.as.arquillian.container.CommonDeployableContainer.deploy(CommonDeployableContainer.java:144)
	at org.jboss.arquillian.container.impl.client.container.ContainerDeployController$3.call(ContainerDeployController.java:161)
	at org.jboss.arquillian.container.impl.client.container.ContainerDeployController$3.call(ContainerDeployController.java:128)
	at org.jboss.arquillian.container.impl.client.container.ContainerDeployController.executeOperation(ContainerDeployController.java:271)
	at org.jboss.arquillian.container.impl.client.container.ContainerDeployController.deploy(ContainerDeployController.java:127)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94)
	at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81)
	at org.jboss.arquillian.container.impl.client.ContainerDeploymentContextHandler.createDeploymentContext(ContainerDeploymentContextHandler.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
	at org.jboss.arquillian.container.impl.client.ContainerDeploymentContextHandler.createContainerContext(ContainerDeploymentContextHandler.java:57)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
	at org.jboss.arquillian.container.impl.client.container.DeploymentExceptionHandler.verifyExpectedExceptionDuringDeploy(DeploymentExceptionHandler.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94)
	at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135)
	at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:115)
	at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67)
	at org.jboss.arquillian.container.test.impl.client.deployment.ClientDeployer.deploy(ClientDeployer.java:93)
	at org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase.doDeploy(JPAMultiXACrashRecoveryTestCase.java:124)

commitHaltRevAfterResourceCommited(org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase)  Time elapsed: 96.013 sec  <<< ERROR!
java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:crashrecovery-3xa, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@7a7cf1ed
	at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:693)
	at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183)
	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:177)
	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:161)
	at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:124)
	at com.sun.proxy.$Proxy24.checkXidsInDoubt(Unknown Source)
	at org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase.checkXidsInDoubt(JPAMultiXACrashRecoveryTestCase.java:166)
{code}

Comment 5 tom.jenkinson 2014-03-19 16:08:07 UTC
Also, I can't find a persistence.xml in the repo.

Comment 6 Hayk Hovsepyan 2014-03-20 13:51:58 UTC
Please specify the full path to "datasource.properties" file in parameter "-Dds.properties"

"persistence.xml" file is generated by "PersistenceUtils.generatePersistenceXML" method and put in deployment in each test class.

Comment 7 tom.jenkinson 2014-03-20 15:29:47 UTC
Hi Hayk,

Adding the full path gets me this:

Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 48.444 sec <<< FAILURE! - in org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase
commitHaltRevAfterResourceCommited(org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase)  Time elapsed: 47.245 sec  <<< ERROR!
org.jboss.as.test.integration.management.util.MgmtOperationException: Management operation failed: {"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {"JBAS014771: Services with missing/unavailable dependencies" => [
    "jboss.data-source.java:jboss/xa-datasources/CrashRecoveryDS is missing [jboss.jdbc-driver.ojdbc6_jar]",
    "jboss.driver-demander.java:jboss/xa-datasources/CrashRecoveryDS is missing [jboss.jdbc-driver.ojdbc6_jar]"
]}}}
	at org.jboss.as.test.jbossts.client.TMOperations.executeOperation(TMOperations.java:1223)


I guess it could be having an issue adding the oracle driver? I can't see instructions above for how to do that?

Tom

Comment 9 tom.jenkinson 2014-03-20 17:30:18 UTC
Hi Hayk,

I did some digging, it appears that this is an issue in Oracle:
https://github.com/tomjenkinson/narayana/commit/b9a578c3ff8275dbc729716a97bb2c3cde225437

Basically, XAResource::recover() returns all the Xids in the system, not just the ones that a specific authenticated XAResource could actually complete.

I think you should raise this with Oracle, feel free to pass on my test.

Thanks,
Tom

Comment 10 Hayk Hovsepyan 2014-03-21 15:57:55 UTC
This problem happens for other databases as well.
mysql55
postgresql92

Please find attached the database.properties files for them and also JDBC drivers I used.

Comment 11 Hayk Hovsepyan 2014-03-21 16:00:57 UTC
Created attachment 877345 [details]
MySql55Driver

Comment 12 Hayk Hovsepyan 2014-03-21 16:03:48 UTC
Created attachment 877347 [details]
PGSql2Driver

Comment 14 Hayk Hovsepyan 2014-03-21 16:05:18 UTC
Created attachment 877349 [details]
datasourcesmysql55.properties

Comment 15 tom.jenkinson 2014-03-21 16:17:34 UTC
I wonder if there is a config option on the databases to only return Xids that the user has permission to complete? I think it is worth someone contacting Oracle/postgres to confirm if this is expected behaviour or a mis-configuration of the database.

Comment 16 Ondrej Chaloupka 2014-03-24 09:26:02 UTC
Hi Tom,

I think that this is correct behavior of db drivers.
What I know the JTA api is not able to differentiate returned Xids by permissions or users. This does not know either database vendors. We struggle with this fact in our tests. As we run the tests on shared databases. And we want to check existence of prepared transactions after the test in DB. We have to handle the fact that recover() method returns all prepared transactions nevertheless who started it. 
And in fact some of the db vendors are not able to differentiate the xids as they have one shared transaction log for all prepared transactions started for the database (e.g. PostgreSQL works in that way, I was checking this once in the past: http://postgresql.1045698.n5.nabble.com/Different-transaction-log-for-database-schema-td5764604.html). They could theoretically somelike differentiate users but from the test experience they do not do such thing. They returns a bunch of xids via jta api and the application is expected to handle it.

I think that this issue should be addressed by TM in the similar way as TM handles recovery by node-id identifier for not recovering transactions from different TM instance. But maybe I am wrong...?

Ondra

Comment 17 tom.jenkinson 2014-05-07 14:58:49 UTC
Hi Ondra,

I understand what you are saying about how the DB is implemented. If you have a suggestion for how we can handle it "I think that this issue should be addressed by TM in the similar way as TM handles recovery by node-id identifier for not recovering transactions from different TM instance. But maybe I am wrong...?" please can you elaborate more on that as I didn't quite understand.

It sounds more like a feature request but one I would be interested in adding.

Thanks,
Tom

Comment 18 Hayk Hovsepyan 2014-05-09 09:16:37 UTC
Tom, Ondra,

My propositions with failures in other databases were wrong, as we have already logged issue with mysql55 and with local correctly configured postgresql92 it worked.

So the issue is only with Oracle only.
As Tom mentioned in his comment https://bugzilla.redhat.com/show_bug.cgi?id=1075008#c9 we will rise an issue with Oracle.

Thanks
/Hayk.

Comment 19 Hayk Hovsepyan 2014-05-09 09:20:59 UTC
requires_doc_text: It needs to be mentioned in the documentation that using more than one Oracle XADatasource referring to the same Oracle server, cause problems in crash recovery.

Comment 20 Hayk Hovsepyan 2014-05-13 13:27:57 UTC
Created attachment 895134 [details]
pgsqlLog

Still, I was able to reproduce the same bug on postgresql92 database. It happens not constantly on both databases, but still the problem exists.
During recovery process it must commit "java:jboss/xa-datasources/CrashRecoveryDS" which is the only resource used in transaction, but it fails as it tries to use "java:jboss/xa-datasources/CrashRecoveryDS2" datasource which is not used in transaction but exists in server configuration.

Comment 21 Michael 2014-05-27 10:43:56 UTC
The recommendation is make sure that the recovery credentials on a connection are configured correctly:

- if multiple datasources are configured against the same database make sure that only one of them is enabled for recovery (by setting the boolean attribute no-recovery="true|false");

- set the recovery-username and recovery-password attributes to have sufficient privileges to access the oracle transaction tables

Comment 22 Ondrej Chaloupka 2014-06-03 14:47:14 UTC
Hi Mike, Hi Tom,

I did a bit more investigation around this and for now (need to be confirmed) I think that's problem on JCA layer. I found out that incorrect JNDI is passed to Xid Wrapper which is provided to jdbc driver.

Bug that I refer to is filled here:
https://bugzilla.redhat.com/show_bug.cgi?id=1104227

Ondra

Comment 23 Ondrej Chaloupka 2014-07-08 14:27:44 UTC

*** This bug has been marked as a duplicate of bug 1091425 ***


Note You need to log in before you can comment on or make changes to this bug.