Bug 681715

Description of problem:

Ran into an issue today with qpid-route dynamic add/del .  Just to give you an idea of what we are trying to accomplish we have 4 brokers all using the same exchange.  I left it out of the explination below to keep it simple.  All of the routes are added via qpid-route.

src->dest represents a qpid-route.  The arrow represents the direction of the route.

Scenerio:

Broker  1 -> Static Route: 'Out.Key'  ->  Broker A
Broker  1 <- Static Route: '#.Key'      <-  Broker A

Broker A   <= Dynamic Route => Broker B  (Both ways)

Broker 2  -> Static Route: 'Out.Key'  -> Broker B
Broker 2  <- Static Route: '#.Key'       <- Broker B

Our map really looks like this  with the caveat that the static is really two separate keys one being #.Key the other Out.Key

Broker 1  <=Static=> Broker A<=Dynamic=> Broker B <=Static=> Broker 2

On first time creating the Dynamic link betweek Broker A and Broker B it works perfectly.   

Send Message(Out.Key)  ->  Broker 1  <=Static=> Broker A<=Dynamic=> Broker B<=Static=> Broker 2  -> (Message Received)

If we delete both sides of the dynamic route and re add them then the following fails:

Send Message(Out.Key)  ->  Broker 1  <=Static=> Broker A<=Dynamic=> Broker B<=Static=> Broker 2  -> (Message Failed)



What we have found is that the message will make it all they way to Broker B, but will never follow the static route to Broker2.

We have also found that if we drain on Broker B, it will then propagate to Broker 2.


It is reproduceable

Steps to Reproduce:
1. Create 4 Brokers
2. Run these commands:
qpid-route route add localhost:5671 localhost:5673 ExchangeName "#.Key"
qpid-route route add localhost:5673 localhost:5671 ExchangeName "Out.Key"
qpid-route route add localhost:5675 localhost:5674 ExchangeName "#.Key"
qpid-route route add localhost:5674 localhost:5675 ExchangeName "Out.Key"
qpid-route dynamic add localhost:5673 localhost:5674 ExchangeName
qpid-route dynamic add localhost:5674 localhost:5673 ExchangeName
#everything below this will cause the federation to break
#before these lines everything works perfectly 

qpid-route dynamic del localhost:5673 localhost:5674 ExchangeName
qpid-route dynamic del localhost:5674 localhost:5673 ExchangeName

qpid-route dynamic add localhost:5673 localhost:5674 ExchangeName
qpid-route dynamic add localhost:5674 localhost:5673 ExchangeName

3.  Create Receiver:
./drain -b localhost:5675 "ExchangeName/Out.Key" "Test"   
3.  Send message
./spout -b localhost:5671 "ExchangeName/Out.Key" "Test"

Actual results:
Noting, will make it to 5674 or 5673 but never to 5675 or 5671.



Expected results:
Messages to arrive at 5675 or 5671

Additional info:

We have found that if you add a drain to 5674 or 5675 will make the messages arrive at 5675 or 5671.

The first time you test, do not run