Bug 998449

Summary: Incorrect return code on second rollback call from TM running with JTS for crash recovery
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Ondrej Chaloupka <ochaloup>
Component: JMSAssignee: Gary Tully <gtully>
Status: CLOSED EOL QA Contact: Miroslav Novak <mnovak>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.1.0   
Target Milestone: ---   
Target Release: EAP 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-19 12:46:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
JMSCrashRecoveryRemoteTestCase#prepareHalt jts server.log (in html format) none

Description Ondrej Chaloupka 2013-08-19 10:52:23 UTC 
It seems that ActiveMQ returns incorrect return code after second call of the rollback of transaction manager doing crash recovery. This seems to be a similar issue as we found for PostgreSQL (https://bugzilla.redhat.com/show_bug.cgi?id=988724).

There is exception thrown on the second rollback() call on XAResource [1].

The JTS specification defines that the rollback() can be called twice on the same XAResource. The XAException seems to carry value 0.
As I referred we found out the similar issue for driver postgresql and the return value was changed to be XAException.XAER_NOTA in such case.

More details on test setup could be found here:
https://docspace.corp.redhat.com/docs/DOC-146452#crash-recovery

In case technical details could be discussed with Tom Jenkinson from Narayana team.



[1]
12:15:57,770 WARN  [com.arjuna.ats.jtax] (RequestProcessor-10) ARJUNA024023: XAResourceRecord.rollback caused an XA error: ARJUNA016099: Unknown error code:0 from resource XAResourceWrapperImpl@2874fb7a[xaResource=org.apache.activemq.ra.ActiveMQManagedConnection$1@38665ee7 pad=false overrideRmValue=null productName=ActiveMQ productVersion=5.8.0.redhat-60024 jndiName=java:/JmsXA] in transaction < 131072, 29, 36, 0000000000-1-1127001-11735-31-518217-16460005149, 292929292929292929292828156292930-8864-2-22111461375292929807829292929292929 >: javax.transaction.xa.XAException: Transaction 'XID:[131072,globalId=0000000000ffffffffffffffff7f001ffffff8b23ffffffe1ffffffcd5211fffffff02e0003331,branchId=0000000000ffffffffffffffff7f001ffffff8b23ffffffe1ffffffcd5211fffffff02e0003800000000]' has not been started.
 at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:773)
 at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:526)
 at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
 at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.rollback(XAResourceWrapperImpl.java:186)
 at com.arjuna.ats.internal.jta.resources.jts.orbspecific.XAResourceRecord.rollback(XAResourceRecord.java:369) [jbossjts-jacorb-4.17.7.Final-redhat-3.jar:4.17.7.Final-redhat-3]
 at com.arjuna.ArjunaOTS.OTSAbstractRecordPOA._invoke(OTSAbstractRecordPOA.java:213) [jbossjts-jacorb-4.17.7.Final-redhat-3.jar:4.17.7.Final-redhat-3]
 at org.jacorb.poa.RequestProcessor.invokeOperation(RequestProcessor.java:306) [jacorb-2.3.2-redhat-4.jar:2.3.2-redhat-4]
 at org.jacorb.poa.RequestProcessor.process(RequestProcessor.java:626) [jacorb-2.3.2-redhat-4.jar:2.3.2-redhat-4]
 at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:769) [jacorb-2.3.2-redhat-4.jar:2.3.2-redhat-4]
Caused by: javax.transaction.xa.XAException: Transaction 'XID:[131072,globalId=0000000000ffffffffffffffff7f001ffffff8b23ffffffe1ffffffcd5211fffffff02e0003331,branchId=0000000000ffffffffffffffff7f001ffffff8b23ffffffe1ffffffcd5211fffffff02e0003800000000]' has not been started.
 at org.apache.activemq.broker.TransactionBroker.getTransaction(TransactionBroker.java:380)
 at org.apache.activemq.broker.TransactionBroker.rollbackTransaction(TransactionBroker.java:267)
 at org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:125)
 at org.apache.activemq.broker.BrokerFilter.rollbackTransaction(BrokerFilter.java:125)
 at org.apache.activemq.broker.MutableBrokerFilter.rollbackTransaction(MutableBrokerFilter.java:131)
 at org.apache.activemq.broker.TransportConnection.processRollbackTransaction(TransportConnection.java:474)
 at org.apache.activemq.command.TransactionInfo.visit(TransactionInfo.java:104)
 at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329)
 at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184)
 at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
 at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
 at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288)
 at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
 at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
 at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
 at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Comment 1 Gary Tully 2013-08-20 11:28:56 UTC 
fix on apache trunk - http://svn.apache.org/r1515783

the problem relates to sparse unmarshalling of exceptions in openwire, so the broker assigned xa_error is not visible to the client and uninitialised to 0 which is invalid xa_ok.
The fix replaces xa_ok with xa_rmerr (a generic failure notification)

this fix will make 5.9 and A-MQ 6.1
Comment 2 Ondrej Chaloupka 2013-09-17 12:38:30 UTC 
Hi Gary,

I've took the RAR from 
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/activemq-rar/5.9-SNAPSHOT/activemq-rar-5.9-20130905.131830-99.rar
and your fix seems to be inside.

But the error exception is still visible because of the incorrect return code.
ARJUNA024023: XAResourceRecord.rollback caused an XA error: XAException.XAER_RMERR from resource XAResourceWrapperImpl@22f01dd3[xaResource=org.apache.activemq.ra.ActiveMQManagedConnection$1@4c848346 pad=false overrideRmValue=null productName=ActiveMQ productVersion=5.9-SNAPSHOT jndiName=java:/JmsXA] in transaction < 131072, 29, 36, 0000000000-1-112700184-20-5322825675-680003749, 2929292929292929292928281562929301139-245111185104-39292929667829292929292929 >: javax.transaction.xa.XAException: Transaction 'XID:[131072,globalId=0000000000ffffffffffffffff7f00154ffffffecffffffcb1652384bffffffbc0002531,branchId=0000000000ffffffffffffffff7f00154ffffffecffffffcb1652384bffffffbc0002a00000000]' has not been started.

What I understand there should be returned return code XA_NOTA to be consistent with Narayana behavior.
Comment 3 Miroslav Novak 2014-07-04 13:04:11 UTC 
Hi Ondra,

can you check this one whether it's still a problem.

Thanks,
Mirek
Comment 4 Ondrej Chaloupka 2014-07-07 08:36:04 UTC 
Created attachment 915999 [details]
JMSCrashRecoveryRemoteTestCase#prepareHalt jts server.log (in html format)

Hi Mirek,

I've rerun the test and pass through the test server log and the issue is still valid. There is returned incorrect return code (XAException.XAER_RMERR instead of XAException.XAER_NOTA) when JTS periodic recovery calls rollback on the same XID for second time.

Adding the server log with Narayana TRACE as attachment.

Ondra
Comment 5 Miroslav Novak 2014-07-07 08:47:00 UTC 
Thanks Ondra! I'm setting jboss-eap-6.4 ? flag and release as this should get fix to EAP 6.4 release. Creating jira ENTMQ-717 for JBoss A-MQ dev to notice this issue.
Comment 6 JBoss JIRA Server 2014-08-07 13:02:16 UTC 
Gary Tully <gary.tully> updated the status of jira ENTMQ-717 to Resolved