Bug 979058

Summary: IBM MQ crash recovery - shown error messages on crash at prepare phase
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Ondrej Chaloupka <ochaloup>
Component: Transaction ManagerAssignee: tom.jenkinson
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.1.0CC: cdewolf, tom.jenkinson
Target Milestone: ER4   
Target Release: EAP 6.1.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-16 20:20:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 986047    

Description Ondrej Chaloupka 2013-06-27 13:52:07 UTC
When crash recovery is run against IBM MQ broker when the server is halted after prepare method being called on the IBM MQ XA resource then you can find these error messages [1] in the log.

But it seems that these error messages do not have influence the crash recovery functionality. I think that the messages are shown in case that TM calls recovery and there are no xids to recover. I've found the error messages in case that I just start the server with resource adapter configured and I waited some time to recovery was launched.

I think that that this is caused by some change in the transaction manager code base because these messages are not shown for version 4.17.3. For 4.17.4 and 5.0.0 could be caught.

How to reproduce:
1) Checkout crash recovery tests
git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-transactions.git
2) cd eap-tests-transactions/integration/jbossts
3) export JBOSS_HOME=.../jboss-eap-6.1.0.GA
(when you run this for AS7.2 or EAP6.1.0.ER5 you can find yourself not getting the error messages)
4) Run the IBM MQ crash recovery test
mvn clean verify -Djboss.dist=$JBOSS_HOME -Dtest=JMSIBMMQCrashRecoveryTestCase#prepareHalt -Dibmmq -Djbossts.noJTS
5) Check server log for errors
vim integration/jbossts/target/jbossas-jbossts/standalone/log/server.log
The test will download resource adapter and run the test against broker. For more details on app server configuration you can check the integration/jbossts/target/jbossas-jbossts/standalone/configuration/standalone-full.xml file after the test was run or take a look on document https://hurl.corp.redhat.com/56fd5bb under title "IBM WebSphere MQ transaction crash recovery testing for EAP6".




[1] Error messages shown in the log after crash recovery is done
INFO  [org.jboss.as.test.jbossts.common.TestXAResourceRecovered] (Periodic Recovery) TestXAResourceRecovered.recover(i=8388608)
INFO  [org.jboss.as.test.jbossts.common.TestXAResourceRecoveryHelper] (Periodic Recovery) recover()
INFO  [org.jboss.as.test.jbossts.common.TestXAResourceRecoveryHelper] (Periodic Recovery) returning 0 Xids
ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException: The method 'xa_rollback' has failed with errorCode '-7'.
ERROR [stderr] (Periodic Recovery)  at com.ibm.mq.jmqi.JmqiXAResource.rollback(JmqiXAResource.java:795)
ERROR [stderr] (Periodic Recovery)  at com.ibm.mq.connector.xa.XARWrapper.rollback(XARWrapper.java:494)
ERROR [stderr] (Periodic Recovery)  at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.rollback(XAResourceWrapperImpl.java:174)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:741)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:647)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:419)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)
WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMFAIL: javax.transaction.xa.XAException: The method 'xa_recover' has failed with errorCode '-7'.
 at com.ibm.mq.jmqi.JmqiXAResource.recover(JmqiXAResource.java:749)
 at com.ibm.mq.connector.xa.XARWrapper.recover(XARWrapper.java:461)
 at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.recover(XAResourceWrapperImpl.java:165)
 at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:695) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2]
 at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:419) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2]
 at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2]

Comment 1 tom.jenkinson 2013-06-27 18:46:05 UTC
https://issues.jboss.org/browse/JBTM-1810

Comment 2 Ondrej Chaloupka 2013-08-16 16:02:50 UTC
It seems that I put incorrect flags for this issue.
Verified for EAP 6.1.1.ER6. This issue was fixed in 4.17.6.

Tom, are you agree that I can set this jira to verified? I'm asking just because this is not in status ON_QA.
Thanks

Comment 3 tom.jenkinson 2013-08-19 07:54:26 UTC
Hi Ondra,
I tried to mark the bz as POST:

"For all JBoss bugs, it is mandatory to set the target milestone field to a non-empty value before a bug can be moved past the ASSIGNED state. i.e. to POST, MODIFIED, ON_DEV, ON_QA, VERIFIED or RELEASE_PENDING. "

Thanks,
Tom

Comment 4 Ondrej Chaloupka 2013-08-19 08:26:55 UTC
Hi Tom,

I hope that you will agree - I changed the milestone to 6.1.1.GA. What is true as the bug is fixed from 6.1.1.ER4.

Ondra

Comment 5 tom.jenkinson 2013-08-19 08:32:45 UTC
Thanks Ondra

Comment 6 Ondrej Chaloupka 2013-08-26 05:34:36 UTC
Verified  for EAP 6.1.1.ER6.
Thanks