Bug 697505

Summary: Dynamically created bridge queues not always recreated after network outage
Product: Red Hat Enterprise MRG Reporter: Andy Goldstein <agoldste>
Component: qpid-cppAssignee: Jason Dillaman <jdillama>
Status: CLOSED ERRATA QA Contact: Chuck Rolke <crolke>
Severity: unspecified Docs Contact:
Priority: high    
Version: 1.3CC: crolke, freznice, jdillama, jross, lzhaldyb, tross
Target Milestone: 3.0   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qpid-cpp-server-0.14 Doc Type: Bug Fix
Doc Text:
It was discovered that after a network outage, dynamic routes were not reestablished. Messages were not sent to the destination exchange, or to the bindings on the destination broker. The fix periodically attempts to reestablish the connection for the dynamic route. After a network outage the dynamic route is recovered and messages are relayed over it.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-24 15:02:29 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 698367    
Attachments:
Description Flags
shell reproducer none

Description Andy Goldstein 2011-04-18 13:23:01 UTC 
Description of problem: When simulating a network failure, dynamically created bridge queues are not always recreated for dynamic federation routes.


Version-Release number of selected component (if applicable):qpid-cpp-server-0.7.946106-28_ptc_hotfix_5_v2.el5


How reproducible: 100%


Steps to Reproduce:
1. Set up 2 brokers on different IP addresses
2. Set up dynamic federation between the 2 brokers
3. Verify federation functions
3. Use iptables on the source broker to reset the existing federation link (e.g. iptables -I INPUT -p tcp -s <destination broker ip> --dport <source broker port> -j REJECT --reject-with tcp-reset)
4. Send a message to the source broker using a routing key that should result in the message being federated to the destination broker - this should fail due to iptables
4. Wait a short amount of time (maybe 10 seconds), then delete the iptables rule i.e. re-open the port
5. Attempt to send more messages to the source broker for federation
  
Actual results: dynamically created bridge is deleted and not recreated.  No messages go across the federation link.  qpid-route commands show the link is operational and the route exists.  qpid-stat -q shows no bridge queue and qpid-stat -u shows no subscription.


Expected results: dynamically created bridge queue should be recreated and messages should be able to flow across the link


Additional info: It appears that the destination broker (link owner) is attempting to call session.attach again with the same session id.  I saw this in the source broker's log:

Apr 18 05:50:32 agoldste qpidd[32576]: 2011-04-18 05:50:32 error Channel exception: session-busy: Session already attached: anonymous.f891f9df-f957-4079-b0b4-a59339d81592 (qpid/broker/SessionManager.cpp:55)
Apr 18 05:50:32 agoldste qpidd[32576]: 2011-04-18 05:50:32 error Channel exception: not-attached: Channel 1 is not attached (qpid/amqp_0_10/SessionHandler.cpp:39)

I'm guessing the destination broker was attempting to recreated the bridge queue and subscription, but because the session.attach command failed, it gave up.
Comment 1 Jason Dillaman 2011-07-12 15:54:45 UTC 
See https://issues.apache.org/jira/browse/QPID-3352 for details and proposed patch
Comment 2 Ted Ross 2011-07-12 18:32:12 UTC 
Fixed upstream in revision 1145706 by Jason Dillaman.
Comment 5 Chuck Rolke 2013-08-06 19:25:08 UTC 
Created attachment 783472 [details]
shell reproducer
Comment 9 errata-xmlrpc 2014-09-24 15:02:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-1296.html