Bug 568838
| Summary: | Dynamic federation duplicates messages | ||
|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Gordon Sim <gsim> |
| Component: | qpid-cpp | Assignee: | Ken Giusti <kgiusti> |
| Status: | CLOSED ERRATA | QA Contact: | Jiri Kolar <jkolar> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 1.2 | CC: | jkolar, kgiusti, tross |
| Target Milestone: | 1.3 | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
When dynamic federation of a topic exchange was in use, sending a message with a routing key that matched more than one binding may have caused such message to be delivered to a matching queue multiple times. To prevent this behavior, the broker now tracks all queues to which the message is routed, and no longer sends it to the same queue more than once.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2010-10-14 16:07:55 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 579681 | ||
| Bug Blocks: | |||
(In reply to comment #0) > Description of problem: > > A unidirectional dynamic federation of a topic exchange results in unexpected > message duplication. > > Version-Release number of selected component (if applicable): > > 1.2 and qpid trunk as of r915773 > > How reproducible: > > 100% > > Steps to Reproduce: > 1. start two brokers (I used ports 5672 for src and 5673 for dest on same box) > 2. establish dynamic federation link between them for amq.topic > (e.g. qpid-route dynamic add localhost:5673 localhost:5672 amq.topic) > 3. create a queue on the destination broker and bind it to amq.topic with two > binding patterns > (e.g. qpid-config -a localhost:5673 add queue test-queue > qpid-config -a localhost:5673 bind amq.topic test-queue '*.hat' > qpid-config -a localhost:5673 bind amq.topic test-queue 'red.*') > > 4. send a single message to the source broker with a routing key that would > match both those bindings > (e.g. echo "My Message" | ./src/tests/sender --exchange amq.topic > --routing-key 'red.hat') > > 5. check queue for that message > (e.g. ./src/tests/receiver --port 5673) > > Actual results: > > Find four copies of the message > > Expected results: > > Find one copy of the message > > Additional info: Do we have any rules about which pattern to match against? i.e. first fit or best fit (if that's even possible) I've not looked at the code yet but it looks like we are matching multiple times, that is, not stopping on the first match, and then the messages are being federated and matched again, hence the 4 messages you are seeing. I can't remember anything in the AMQP spec about this so I'm guessing its up to each vendor to make a choice. Sam's observation is correct. I've spent some time looking at the code under debug and it doesn't appear to be a federation problem per se - more of a topic exchange issue. The problem appears in the TopicExchange::route() method. This code searches all bindings for matches against the routing key. Each matched binding is stored in a temporary vector. Once all matching bindings are found, the message is enqueued to all queues specified by the bindings in the temp vector list. The problem is that this ends up putting the same queue on that temp binding list multiple times. Hence, multiple messages are sent to the same queue. In the test case above the "red.hat" routing key matches the "red.*" binding, which points to the federated bridge queue. It also matches the "*.hat" binding, which points to the same queue. Two messages are enqueued to the destination broker. The same process applies at the destination broker, resulting in 4 duplicate messages. A potential solution: ensure that the same queue is never added to that temp list more than once for a topic Exchange. However, this visibly changes the behaviour of the broker, so I'm reluctant to implement this change.. Opinions/Options? The desired solution is as you describe, Ken. I've created bug 579681 to track that root cause (this depends on that). The 0-10 spec isn't as clear on the subject, but this was very explicit in earlier specs and the intention is that at most one copy of a message is ever enqueued when routing a message through an exchange, no matter how many bindings match. Fixed upstream & test added to federation tests: http://svn.apache.org/viewcvs?view=rev&rev=931257 Tested: on 752581 bug appears on 946106 does not. It has been fixed validated on RHEL 5.5 i386 / x86_64 packages: # rpm -qa | grep -E '(qpid|openais|rhm)' | sort -u openais-0.80.6-16.el5_5.1 openais-debuginfo-0.80.6-16.el5_5.1 python-qpid-0.7.946106-1.el5 qpid-cpp-client-0.7.946106-1.el5 qpid-cpp-client-devel-0.7.946106-1.el5 qpid-cpp-client-devel-docs-0.7.946106-1.el5 qpid-cpp-client-ssl-0.7.946106-1.el5 qpid-cpp-mrg-debuginfo-0.7.935473-1.el5 qpid-cpp-server-0.7.946106-1.el5 qpid-cpp-server-cluster-0.7.946106-1.el5 qpid-cpp-server-devel-0.7.946106-1.el5 qpid-cpp-server-ssl-0.7.946106-1.el5 qpid-cpp-server-store-0.7.946106-1.el5 qpid-cpp-server-xml-0.7.946106-1.el5 qpid-java-client-0.7.946106-3.el5 qpid-java-common-0.7.946106-3.el5 qpid-tools-0.7.946106-4.el5 rhm-docs-0.7.946106-1.el5 ->VERIFIED also verified on on RHEL 4.8 i386 / x86_64
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
* Cause: Federating brokers using a topic exchange and a message with a routing key that matches multiple distinct bindings.
* Consequence: The message would be delivered multiple times to each matching queue.
* Fix: The broker now tracks queues to which the message is routed, and prevents the message from being forwarded on the same queue twice.
* Result: A single copy of the message appears on each matching queue.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
Diffed Contents:
@@ -1,4 +1 @@
-* Cause: Federating brokers using a topic exchange and a message with a routing key that matches multiple distinct bindings.
+When dynamic federation of a topic exchange was in use, sending a message with a routing key that matched more than one binding may have caused such message to be delivered to a matching queue multiple times. To prevent this behavior, the broker now tracks all queues to which the message is routed, and no longer sends it to the same queue more than once.-* Consequence: The message would be delivered multiple times to each matching queue.
-* Fix: The broker now tracks queues to which the message is routed, and prevents the message from being forwarded on the same queue twice.
-* Result: A single copy of the message appears on each matching queue.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0773.html |
Description of problem: A unidirectional dynamic federation of a topic exchange results in unexpected message duplication. Version-Release number of selected component (if applicable): 1.2 and qpid trunk as of r915773 How reproducible: 100% Steps to Reproduce: 1. start two brokers (I used ports 5672 for src and 5673 for dest on same box) 2. establish dynamic federation link between them for amq.topic (e.g. qpid-route dynamic add localhost:5673 localhost:5672 amq.topic) 3. create a queue on the destination broker and bind it to amq.topic with two binding patterns (e.g. qpid-config -a localhost:5673 add queue test-queue qpid-config -a localhost:5673 bind amq.topic test-queue '*.hat' qpid-config -a localhost:5673 bind amq.topic test-queue 'red.*') 4. send a single message to the source broker with a routing key that would match both those bindings (e.g. echo "My Message" | ./src/tests/sender --exchange amq.topic --routing-key 'red.hat') 5. check queue for that message (e.g. ./src/tests/receiver --port 5673) Actual results: Find four copies of the message Expected results: Find one copy of the message Additional info: