Bug 697505 - Dynamically created bridge queues not always recreated after network outage
Summary: Dynamically created bridge queues not always recreated after network outage
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: 1.3
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: 3.0
: ---
Assignee: Jason Dillaman
QA Contact: Chuck Rolke
URL:
Whiteboard:
Depends On:
Blocks: 698367
TreeView+ depends on / blocked
 
Reported: 2011-04-18 13:23 UTC by Andy Goldstein
Modified: 2014-09-24 15:02 UTC (History)
6 users (show)

Fixed In Version: qpid-cpp-server-0.14
Doc Type: Bug Fix
Doc Text:
It was discovered that after a network outage, dynamic routes were not reestablished. Messages were not sent to the destination exchange, or to the bindings on the destination broker. The fix periodically attempts to reestablish the connection for the dynamic route. After a network outage the dynamic route is recovered and messages are relayed over it.
Clone Of:
Environment:
Last Closed: 2014-09-24 15:02:29 UTC
Target Upstream Version:


Attachments (Terms of Use)
shell reproducer (6.24 KB, application/x-shellscript)
2013-08-06 19:25 UTC, Chuck Rolke
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Apache JIRA QPID-3352 0 None None None 2012-11-14 19:49:44 UTC
Red Hat Product Errata RHEA-2014:1296 0 normal SHIPPED_LIVE Red Hat Enterprise MRG Messaging 3.0 Release 2014-09-24 19:00:06 UTC

Description Andy Goldstein 2011-04-18 13:23:01 UTC
Description of problem: When simulating a network failure, dynamically created bridge queues are not always recreated for dynamic federation routes.


Version-Release number of selected component (if applicable):qpid-cpp-server-0.7.946106-28_ptc_hotfix_5_v2.el5


How reproducible: 100%


Steps to Reproduce:
1. Set up 2 brokers on different IP addresses
2. Set up dynamic federation between the 2 brokers
3. Verify federation functions
3. Use iptables on the source broker to reset the existing federation link (e.g. iptables -I INPUT -p tcp -s <destination broker ip> --dport <source broker port> -j REJECT --reject-with tcp-reset)
4. Send a message to the source broker using a routing key that should result in the message being federated to the destination broker - this should fail due to iptables
4. Wait a short amount of time (maybe 10 seconds), then delete the iptables rule i.e. re-open the port
5. Attempt to send more messages to the source broker for federation
  
Actual results: dynamically created bridge is deleted and not recreated.  No messages go across the federation link.  qpid-route commands show the link is operational and the route exists.  qpid-stat -q shows no bridge queue and qpid-stat -u shows no subscription.


Expected results: dynamically created bridge queue should be recreated and messages should be able to flow across the link


Additional info: It appears that the destination broker (link owner) is attempting to call session.attach again with the same session id.  I saw this in the source broker's log:

Apr 18 05:50:32 agoldste qpidd[32576]: 2011-04-18 05:50:32 error Channel exception: session-busy: Session already attached: anonymous.f891f9df-f957-4079-b0b4-a59339d81592 (qpid/broker/SessionManager.cpp:55)
Apr 18 05:50:32 agoldste qpidd[32576]: 2011-04-18 05:50:32 error Channel exception: not-attached: Channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:39)

I'm guessing the destination broker was attempting to recreated the bridge queue and subscription, but because the session.attach command failed, it gave up.

Comment 1 Jason Dillaman 2011-07-12 15:54:45 UTC
See https://issues.apache.org/jira/browse/QPID-3352 for details and proposed patch

Comment 2 Ted Ross 2011-07-12 18:32:12 UTC
Fixed upstream in revision 1145706 by Jason Dillaman.

Comment 5 Chuck Rolke 2013-08-06 19:25:08 UTC
Created attachment 783472 [details]
shell reproducer

Comment 9 errata-xmlrpc 2014-09-24 15:02:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-1296.html


Note You need to log in before you can comment on or make changes to this bug.