Bug 923202
| Summary: | JCA XA transaction won't failover in clustered MRG-M <-> EAP5/6 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | ppecka <ppecka> | ||||
| Component: | qpid-jca | Assignee: | messaging-bugs <messaging-bugs> | ||||
| Status: | CLOSED UPSTREAM | QA Contact: | Michal Toth <mtoth> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 2.2 | CC: | jross, mtoth | ||||
| Target Milestone: | 2.2.4 | Flags: | mtoth:
needinfo+
|
||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2025-02-10 03:27:36 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 917988 | ||||||
| Attachments: |
|
||||||
Created attachment 712786 [details]
failover_exchange 5messages sent
attachment with logs, run with only 5 messages, failover_exchange
this ouput shows that MDB opens connection to remaining cluster node(
qpid-stat -b localhost:5673 -c)
Connections
client-addr cproc cpid auth connected idle msgIn msgOut
===================================================================================================
192.168.122.169:5673-192.168.122.1:34911 guest@QPID 24s 19s 38 25
192.168.122.169:5673-192.168.122.1:34910 mdb@QPID 24s 19s 0 0
[::1]:5673-[::1]:37031 anonymous 0s 0s 214 235
Quick update, I am still debugging this issue. It appears to be in the JMS client XASession/XAResource classes. As a result, this does not involve the adapter directly but is manifest in that we only really use XA in the context of JCA at this time. I will loop in Rajith as well as Alan to maybe get some more insight. QE is equipped to run tests, gather logs etc. I have a limited environment on my end, but I can use it to a certain extent. At this point, from dev it's simply a matter of continuing investigate and debugging. This scenario has been tested on the EAP 5 as well and the same issue occurred. as there is no bz for mrg-2.3.0, I d like to note that the issue is present in mrg-2.3.0 as well. I tested same scenario with Local Transactions on 2.3.0 packages: 1. Whet tests are run on EAP 5 -> Pass - failover completes successfully . 2. When tests are run on EAP 6 -> Fail - failover fails to complete. This product has been discontinued or is no longer tracked in Red Hat Bugzilla. |
Description of problem: Failover won't work under XAtransactions. Killing "active" qpidd cluster node, when message driven bean reads messages out of Queue under XA. Version-Release number of selected component (if applicable): qpid-cpp-server-0.14-22.el6_3.x86_64 qpid-cpp-client-devel-0.14-22.el6_3.x86_64 qpid-java-common-0.18-2.el6.noarch qpid-cpp-server-devel-0.14-22.el6_3.x86_64 qpid-cpp-server-cluster-0.14-22.el6_3.x86_64 python-qpid-qmf-0.14-14.el6_3.x86_64 qpid-cpp-server-store-0.14-22.el6_3.x86_64 python-qpid-0.14-11.el6_3.noarch qpid-cpp-server-xml-0.14-22.el6_3.x86_64 qpid-jca-0.18-2.2.el6.noarch qpid-cpp-client-ssl-0.14-22.el6_3.x86_64 qpid-java-client-0.18-2.el6.noarch qpid-qmf-devel-0.14-14.el6_3.x86_64 qpid-tools-0.14-6.el6_3.noarch qpid-java-example-0.18-2.el6.noarch rh-qpid-cpp-tests-0.14-22.el6_3.x86_64 qpid-qmf-0.14-14.el6_3.x86_64 qpid-jca-xarecovery-0.18-2.2.el6.noarch qpid-cpp-client-0.14-22.el6_3.x86_64 qpid-cpp-server-ssl-0.14-22.el6_3.x86_64 qpid-cpp-client-devel-docs-0.14-22.el6_3.noarch How reproducible: 100% Steps to Reproduce: 1. start 2 clustered qpidd nodes 2. run EAP6 with qpid JCA (configured to use XAtransactions) 3. fill queue with some number of messages in one transaction, and start reading from Queue with MDB one message per XA transaction 4. kill one active qpidd node 5. check if MDB failovers to remaining qpidd node Actual results: ... 13:19:20,320 INFO [com.redhat.mrg.MRGJCAMessageBean] (Dispatcher-0-Conn-1) MDB got message '1363609141926:499 13:19:20,324 INFO [com.redhat.mrg.MRGJCAMessageBean] (Dispatcher-0-Conn-1) MDB got message '1363609141926:500 13:19:20,367 INFO [com.redhat.mrg.MRGJCAMessageBean] (Dispatcher-0-Conn-1) MDB got message '1363609141926:501 13:19:20,371 INFO [com.redhat.mrg.MRGJCAMessageBean] (Dispatcher-0-Conn-1) MDB got message '1363609141926:502 13:19:20,413 INFO [com.redhat.mrg.MRGJCAMessageBean] (Dispatcher-0-Conn-1) MDB got message '1363609141926:503 13:19:20,419 INFO [com.redhat.mrg.MRGJCAMessageBean] (Dispatcher-0-Conn-1) MDB got message '1363609141926:504 13:19:20,423 INFO [com.redhat.mrg.MRGJCAMessageBean] (Dispatcher-0-Conn-1) MDB got message '1363609141926:505 13:19:20,449 INFO [org.apache.qpid.jms.failover.FailoverExchangeMethod] (IoReceiver - /192.168.122.169:5672) ============================================================ 13:19:20,450 INFO [org.apache.qpid.jms.failover.FailoverExchangeMethod] (IoReceiver - /192.168.122.169:5672) FailoverExchange: Current Broker Index:1 Failed Attempts:0 Orignal broker details:tcp://192.168.122.169:5672?max_prefetch='0'&sync_ack='true' -------- Broker List ----------- tcp://192.168.122.169:5673?max_prefetch='0'&sync_ack='true' >tcp://192.168.122.169:5672?max_prefetch='0'&sync_ack='true' -------------------------------- 13:19:20,450 INFO [org.apache.qpid.jms.failover.FailoverExchangeMethod] (IoReceiver - /192.168.122.169:5672) FailoverAllowed true 13:19:20,450 INFO [org.apache.qpid.jms.failover.FailoverExchangeMethod] (IoReceiver - /192.168.122.169:5672) ============================================================ 13:19:20,450 INFO [org.apache.qpid.jms.failover.FailoverExchangeMethod] (IoReceiver - /192.168.122.169:5672) No delay between connect retries, use tcp://host:port?connectdelay='value' to enable. 13:19:20,449 WARN [org.apache.qpid.ra.inflow.QpidExceptionHandler] (IoReceiver - /192.168.122.169:5672) Failure in Qpid activation org.apache.qpid.ra.inflow.QpidActivationSpec(ra=org.apache.qpid.ra.QpidResourceAdapter@245afead destination=java:jboss/exported/queue/mrg_mrg_jca_mdb_transactions_in destinationType=javax.jms.Queue ack=Auto-acknowledge durable=false clientID=null user=null maxSession=15 connectionPerHandler=true): javax.jms.JMSException: connection aborted at org.apache.qpid.client.AMQConnectionDelegate_0_10.closed(AMQConnectionDelegate_0_10.java:346) at org.apache.qpid.transport.Connection.closed(Connection.java:567) at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:112) at org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:204) at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:174) at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] Caused by: org.apache.qpid.transport.ConnectionException: connection aborted at org.apache.qpid.transport.Connection.closed(Connection.java:540) ... 4 more ... 13:19:20,501 INFO [org.apache.qpid.jms.failover.FailoverExchangeMethod] (Dispatcher-0-Conn-17) Failover exchange notified cluster membership change 13:19:20,501 INFO [org.apache.qpid.jms.failover.FailoverExchangeMethod] (Dispatcher-0-Conn-17) ============================================================ 13:19:20,502 INFO [org.apache.qpid.jms.failover.FailoverExchangeMethod] (Dispatcher-0-Conn-17) Updated cluster membership details amqp://guest:********@client_id/test?brokerlist='tcp://192.168.122.169:5673?max_prefetch='0'&sync_ack='true''&failover='failover_exchange' 13:19:20,502 INFO [org.apache.qpid.jms.failover.FailoverExchangeMethod] (Dispatcher-0-Conn-17) ============================================================ 13:19:20,505 ERROR [org.apache.qpid.client.BasicMessageConsumer] (Dispatcher-0-Conn-1) reNotification : Caught exception (dump follows) - ignoring...: javax.jms.IllegalStateException: Invalid Session at org.apache.qpid.client.BasicMessageConsumer.checkPreConditions(BasicMessageConsumer.java:890) at org.apache.qpid.client.BasicMessageConsumer.getMessageListener(BasicMessageConsumer.java:225) at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:751) at org.apache.qpid.client.BasicMessageConsumer_0_10.notifyMessage(BasicMessageConsumer_0_10.java:141) at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:725) at org.apache.qpid.client.BasicMessageConsumer_0_10.notifyMessage(BasicMessageConsumer_0_10.java:186) at org.apache.qpid.client.BasicMessageConsumer_0_10.notifyMessage(BasicMessageConsumer_0_10.java:54) at org.apache.qpid.client.AMQSession$Dispatcher.notifyConsumer(AMQSession.java:3479) at org.apache.qpid.client.AMQSession$Dispatcher.dispatchMessage(AMQSession.java:3418) at org.apache.qpid.client.AMQSession$Dispatcher.access$1000(AMQSession.java:3205) at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:3198) at org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:54) at org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:3341) at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] 13:19:20,508 ERROR [org.apache.qpid.client.BasicMessageConsumer] (Dispatcher-0-Conn-1) reNotification : Caught exception (dump follows) - ignoring...: javax.jms.IllegalStateException: Invalid Session at org.apache.qpid.client.BasicMessageConsumer.checkPreConditions(BasicMessageConsumer.java:890) at org.apache.qpid.client.BasicMessageConsumer.getMessageListener(BasicMessageConsumer.java:225) at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:751) at org.apache.qpid.client.BasicMessageConsumer_0_10.notifyMessage(BasicMessageConsumer_0_10.java:141) at org.apache.qpid.client.BasicMessageConsumer.notifyMessage(BasicMessageConsumer.java:725) at org.apache.qpid.client.BasicMessageConsumer_0_10.notifyMessage(BasicMessageConsumer_0_10.java:186) at org.apache.qpid.client.BasicMessageConsumer_0_10.notifyMessage(BasicMessageConsumer_0_10.java:54) at org.apache.qpid.client.AMQSession$Dispatcher.notifyConsumer(AMQSession.java:3479) at org.apache.qpid.client.AMQSession$Dispatcher.dispatchMessage(AMQSession.java:3418) at org.apache.qpid.client.AMQSession$Dispatcher.access$1000(AMQSession.java:3205) at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:3198) at org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:54) at org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:3341) at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] Expected results: Additional info: