Created attachment 873054 [details] FirstScenario Description of problem: We did testing of 2 crashrec scenarios with XA (non LRCO related) and they failed, I will describe both of them and will attach log files. Could you please review them so we can discuss it there any bug or it is test problem? 1. Register 3 pairs of XA resources in standalone-full.xml. All XA resources are connected to Oracle12c with different credentials. The names of DS pairs are CrashRecoveryDS/VerificationDS, CrashRecoveryDS1/VerificationDS1, CrashRecoveryDS2/VerificationDS2. In persistence.xml file of deployed JAR archive, all 3 XA datasources are added as persistence unit. Via Byteman, register TestXAResource "beforeEntityUpdate" and make it to crash on "commit' method exit. Call transactional SLSB which will update 1 entity in DB by using only CrashRecoveryDS , and it will crash. Then reload the server and call recovery. You will see that real XA resource is not recovered and error is in log (attached FirstScenario_jta_server.log). Seems like it mixes datasources registered in standalone or in persistence xml files. 2. Again like in first one, register 3 pairs of XA resources in standalone-full.xml. All XA resources are connected to Oracle12c with different credentials. The names of DS pairs are CrashRecoveryDS/VerificationDS, CrashRecoveryDS1/VerificationDS1, CrashRecoveryDS2/VerificationDS2. But this time, in persistence.xml file of deployed JAR archive, only 1 XA datasources are added as persistence unit. Via Byteman, register TestXAResource "beforeEntityUpdate" and make it to crash on "commit' method exit. Call transactional SLSB which will update 1 entity in DB by using only CrashRecoveryDS , and it will crash. Then reload the server and call recovery. As in first case, you will see that real XA resource is not recovered and error is in log (attached SecondScenario_jta_server.log).
Created attachment 873055 [details] SecondScenario
FYI, when I try to run this: {code} Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 97.134 sec <<< FAILURE! - in org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase commitHaltRevAfterResourceCommited(org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase) Time elapsed: 96.012 sec <<< ERROR! org.jboss.arquillian.container.spi.client.container.DeploymentException: Cannot deploy: crashrecovery-3xa.jar at org.jboss.as.controller.client.helpers.standalone.impl.ServerDeploymentPlanResultFuture.getActionResult(ServerDeploymentPlanResultFuture.java:134) at org.jboss.as.controller.client.helpers.standalone.impl.ServerDeploymentPlanResultFuture.getResultFromNode(ServerDeploymentPlanResultFuture.java:123) at org.jboss.as.controller.client.helpers.standalone.impl.ServerDeploymentPlanResultFuture.get(ServerDeploymentPlanResultFuture.java:85) at org.jboss.as.controller.client.helpers.standalone.impl.ServerDeploymentPlanResultFuture.get(ServerDeploymentPlanResultFuture.java:42) at org.jboss.as.controller.client.helpers.standalone.ServerDeploymentHelper.deploy(ServerDeploymentHelper.java:50) at org.jboss.as.arquillian.container.ArchiveDeployer.deployInternal(ArchiveDeployer.java:77) at org.jboss.as.arquillian.container.ArchiveDeployer.deployInternal(ArchiveDeployer.java:64) at org.jboss.as.arquillian.container.ArchiveDeployer.deploy(ArchiveDeployer.java:46) at org.jboss.as.arquillian.container.CommonDeployableContainer.deploy(CommonDeployableContainer.java:144) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController$3.call(ContainerDeployController.java:161) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController$3.call(ContainerDeployController.java:128) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController.executeOperation(ContainerDeployController.java:271) at org.jboss.arquillian.container.impl.client.container.ContainerDeployController.deploy(ContainerDeployController.java:127) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:99) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:81) at org.jboss.arquillian.container.impl.client.ContainerDeploymentContextHandler.createDeploymentContext(ContainerDeploymentContextHandler.java:78) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.container.impl.client.ContainerDeploymentContextHandler.createContainerContext(ContainerDeploymentContextHandler.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.container.impl.client.container.DeploymentExceptionHandler.verifyExpectedExceptionDuringDeploy(DeploymentExceptionHandler.java:50) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:94) at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:88) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:135) at org.jboss.arquillian.core.impl.ManagerImpl.fire(ManagerImpl.java:115) at org.jboss.arquillian.core.impl.EventImpl.fire(EventImpl.java:67) at org.jboss.arquillian.container.test.impl.client.deployment.ClientDeployer.deploy(ClientDeployer.java:93) at org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase.doDeploy(JPAMultiXACrashRecoveryTestCase.java:124) commitHaltRevAfterResourceCommited(org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase) Time elapsed: 96.013 sec <<< ERROR! java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:crashrecovery-3xa, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@7a7cf1ed at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:693) at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116) at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183) at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:177) at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:161) at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:124) at com.sun.proxy.$Proxy24.checkXidsInDoubt(Unknown Source) at org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase.checkXidsInDoubt(JPAMultiXACrashRecoveryTestCase.java:166) {code}
Also, I can't find a persistence.xml in the repo.
Please specify the full path to "datasource.properties" file in parameter "-Dds.properties" "persistence.xml" file is generated by "PersistenceUtils.generatePersistenceXML" method and put in deployment in each test class.
Hi Hayk, Adding the full path gets me this: Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 48.444 sec <<< FAILURE! - in org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase commitHaltRevAfterResourceCommited(org.jboss.as.test.jbossts.crashrec.test.JPAMultiXACrashRecoveryTestCase) Time elapsed: 47.245 sec <<< ERROR! org.jboss.as.test.integration.management.util.MgmtOperationException: Management operation failed: {"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {"JBAS014771: Services with missing/unavailable dependencies" => [ "jboss.data-source.java:jboss/xa-datasources/CrashRecoveryDS is missing [jboss.jdbc-driver.ojdbc6_jar]", "jboss.driver-demander.java:jboss/xa-datasources/CrashRecoveryDS is missing [jboss.jdbc-driver.ojdbc6_jar]" ]}}} at org.jboss.as.test.jbossts.client.TMOperations.executeOperation(TMOperations.java:1223) I guess it could be having an issue adding the oracle driver? I can't see instructions above for how to do that? Tom
Hi Hayk, I did some digging, it appears that this is an issue in Oracle: https://github.com/tomjenkinson/narayana/commit/b9a578c3ff8275dbc729716a97bb2c3cde225437 Basically, XAResource::recover() returns all the Xids in the system, not just the ones that a specific authenticated XAResource could actually complete. I think you should raise this with Oracle, feel free to pass on my test. Thanks, Tom
This problem happens for other databases as well. mysql55 postgresql92 Please find attached the database.properties files for them and also JDBC drivers I used.
Created attachment 877345 [details] MySql55Driver
Created attachment 877347 [details] PGSql2Driver
Created attachment 877349 [details] datasourcesmysql55.properties
I wonder if there is a config option on the databases to only return Xids that the user has permission to complete? I think it is worth someone contacting Oracle/postgres to confirm if this is expected behaviour or a mis-configuration of the database.
Hi Tom, I think that this is correct behavior of db drivers. What I know the JTA api is not able to differentiate returned Xids by permissions or users. This does not know either database vendors. We struggle with this fact in our tests. As we run the tests on shared databases. And we want to check existence of prepared transactions after the test in DB. We have to handle the fact that recover() method returns all prepared transactions nevertheless who started it. And in fact some of the db vendors are not able to differentiate the xids as they have one shared transaction log for all prepared transactions started for the database (e.g. PostgreSQL works in that way, I was checking this once in the past: http://postgresql.1045698.n5.nabble.com/Different-transaction-log-for-database-schema-td5764604.html). They could theoretically somelike differentiate users but from the test experience they do not do such thing. They returns a bunch of xids via jta api and the application is expected to handle it. I think that this issue should be addressed by TM in the similar way as TM handles recovery by node-id identifier for not recovering transactions from different TM instance. But maybe I am wrong...? Ondra
Hi Ondra, I understand what you are saying about how the DB is implemented. If you have a suggestion for how we can handle it "I think that this issue should be addressed by TM in the similar way as TM handles recovery by node-id identifier for not recovering transactions from different TM instance. But maybe I am wrong...?" please can you elaborate more on that as I didn't quite understand. It sounds more like a feature request but one I would be interested in adding. Thanks, Tom
Tom, Ondra, My propositions with failures in other databases were wrong, as we have already logged issue with mysql55 and with local correctly configured postgresql92 it worked. So the issue is only with Oracle only. As Tom mentioned in his comment https://bugzilla.redhat.com/show_bug.cgi?id=1075008#c9 we will rise an issue with Oracle. Thanks /Hayk.
requires_doc_text: It needs to be mentioned in the documentation that using more than one Oracle XADatasource referring to the same Oracle server, cause problems in crash recovery.
Created attachment 895134 [details] pgsqlLog Still, I was able to reproduce the same bug on postgresql92 database. It happens not constantly on both databases, but still the problem exists. During recovery process it must commit "java:jboss/xa-datasources/CrashRecoveryDS" which is the only resource used in transaction, but it fails as it tries to use "java:jboss/xa-datasources/CrashRecoveryDS2" datasource which is not used in transaction but exists in server configuration.
The recommendation is make sure that the recovery credentials on a connection are configured correctly: - if multiple datasources are configured against the same database make sure that only one of them is enabled for recovery (by setting the boolean attribute no-recovery="true|false"); - set the recovery-username and recovery-password attributes to have sufficient privileges to access the oracle transaction tables
Hi Mike, Hi Tom, I did a bit more investigation around this and for now (need to be confirmed) I think that's problem on JCA layer. I found out that incorrect JNDI is passed to Xid Wrapper which is provided to jdbc driver. Bug that I refer to is filled here: https://bugzilla.redhat.com/show_bug.cgi?id=1104227 Ondra
*** This bug has been marked as a duplicate of bug 1091425 ***