Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 916975

Summary: Journal can not manage enqueues matching two bindings from headers exchange to the same queue
Product: Red Hat Enterprise MRG Reporter: Pavel Moravec <pmoravec>
Component: qpid-cppAssignee: Pavel Moravec <pmoravec>
Status: CLOSED ERRATA QA Contact: Zdenek Kraus <zkraus>
Severity: high Docs Contact:
Priority: medium    
Version: 2.3CC: esammons, freznice, jkocnova, jross, pmoravec, sauchter, zkraus
Target Milestone: 3.1Keywords: TestCaseProvided, WorkAround
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: qpid-cpp-0.30-2 Doc Type: Bug Fix
Doc Text:
It was discovered that if a message sent to a headers exchange matched two bindings to the same queue, the queue attempted to enqueue the message twice. This caused a journal exception when the queue attempted to store the message. The headers exchange implementation has been improved to ensure there is just one enqueue made in these circumstances. This prevents the journal exception from occurring, and fixes the originally reported issue.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2015-04-14 13:46:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Patch proposal none

Description Pavel Moravec 2013-03-01 10:32:34 UTC 
Description of problem:
Assume there are two bindings from a header exchange towards the same durable queue. Sending a durable message to the exchange and matching both the bindings raises below exception in queue's journal.


Version-Release number of selected component (if applicable):
qpid-cpp-server-0.18-14
qpid-cpp-server-store-0.18-14


How reproducible:
100%


Steps to Reproduce:
rm -rf /var/lib/qpidd/* /var/lib/qpidd/.* 2> /dev/null
service qpidd restart
qpid-config add queue MyQueue --durable
qpid-config bind amq.match MyQueue SomeKey any property1=value1
qpid-config bind amq.match MyQueue OtherKey all property2=value2
qpid-send -a "amq.match" -m 1 -P property1=value1 -P property2=value2 --durable=true

  
Actual results:
2013-03-01 10:15:54 [Client] warning Broker closed connection: 501, Queue MyQueue: MessageStoreImpl::store() failed: jexception 0x0b00 wmgr::enqueue() threw JERR_MAP_DUPLICATE: Attempted to insert record into map using duplicate key. (rid=0x1 _pfid=0x0) (MessageStoreImpl.cpp:1361)
qpid-send: framing-error: Queue MyQueue: MessageStoreImpl::store() failed: jexception 0x0b00 wmgr::enqueue() threw JERR_MAP_DUPLICATE: Attempted to insert record into map using duplicate key. (rid=0x1 _pfid=0x0) (MessageStoreImpl.cpp:1361)


Expected results:
No such exception.


Additional info:
Curiously, when sending transient message (or to transient queue), one message sent triggers _two_ messages stored in the queue. Is that intentional? As this is inconsistency. Doing equivalent example but for topic exchange instead of headers, only _one_ message ends in the queue.

Therefore it is a question what should be correct:
- two messages should be kept in the queue, as simply following mathematical / logical rules, the exchange gets a message, and for each binding matched, it sends the message to the destination queue
- the message should be kept in the queue only once, as there is no reason to enqueue there two copies of the same message - what a consumer would gain from two copies? That would be rather redundant than usefull..

Resolving the dilema above is prerequisite to resolving this bug.
Comment 1 Pavel Moravec 2013-05-16 10:24:38 UTC 
Effective workaround: prevent having more bindings from a headers exchange to the same queue such that a message can match more bindings.

Example: instead of having:

$ qpid-config bind amq.match MyQueue MyQueue.1 all prop1=1 prop2=1
$ qpid-config bind amq.match MyQueue MyQueue.2 all prop1=1 prop3=1

where a message with properties prop1=1 and prop2=1 and prop3=1 matches both bindings, change the bindings to:

$ qpid-config bind amq.match MyQueue MyQueue.1 all prop1=1 prop2=1
$ qpid-config bind amq.match MyQueue MyQueue.2 all prop1=1 prop3=1 prop2!=1

Now, with the new restriction on second binding, the bindings have mutually exclusive binding keys.
Comment 3 Pavel Moravec 2013-12-10 15:21:51 UTC 
Created attachment 834835 [details]
Patch proposal

Patch proposal - simple but maybe not ideal.

Root cause of the bug: HeadersExchange::route method generates BindingList that may contain multiple bindings to the same queue. That causes a message to be enqueued to the same queue more than once.

The patch sorts+makes unique the BindingList by using std::map < Queue , Binding > (boost pointers in fact). Maybe there is more elegant way of doing so, anyway this patch fixes the problem (no automated tests run so far).
Comment 4 Pavel Moravec 2013-12-10 15:24:32 UTC 
(In reply to Pavel Moravec from comment #3)
> Created attachment 834835 [details]
> Patch proposal
> 
> Patch proposal - simple but maybe not ideal.
> 
> Root cause of the bug: HeadersExchange::route method generates BindingList
> that may contain multiple bindings to the same queue. That causes a message
> to be enqueued to the same queue more than once.
> 
> The patch sorts+makes unique the BindingList by using std::map < Queue ,
> Binding > (boost pointers in fact). Maybe there is more elegant way of doing
> so, anyway this patch fixes the problem (no automated tests run so far).

FYI the bug is _not_ in store but in broker itself. Per gsim, a message matching multiple bindings to same queue should be enqueued just once to the queue. That is now fixed by the patch for headers exchange.
Comment 5 Pavel Moravec 2014-02-02 12:19:51 UTC 
Upstream review request: https://reviews.apache.org/r/17641/
Comment 6 Pavel Moravec 2014-02-03 12:51:24 UTC 
Committed to upstream: r1563863.
Comment 10 Jitka Kocnova 2015-03-20 09:23:35 UTC 
This bug is UNABLE TO REPRODUCE with:
qpid-cpp-server-0.18-14 and qpid-cpp-server-store-0.18-14 on RHEL x86_64

There is no such exception.
Comment 11 Zdenek Kraus 2015-04-02 15:43:19 UTC 
I managed to reproduce the issue with qpid-cpp-0.18-36 with use of our testing clients. Unfortunatelly I cannot find a reason why qpid-send do not cause the issue.

Issue was tested on RHEL 6.6, RHEL 7.1 with following packages:
python-qpid-0.30-6
python-qpid-qmf-0.30-6
qpid-tools-0.30-5
qpid-cpp-server-linearstore-0.30-8
qpid-proton-c-0.9-2
qpid-cpp-server-0.30-8
qpid-cpp-client-devel-0.30-8
qpid-qmf-0.30-6
qpid-java-common-0.30-5
qpid-java-example-0.30-5
qpid-cpp-server-rdma-0.30-8
qpid-cpp-server-ha-0.30-8
qpid-cpp-server-xml-0.30-8
qpid-cpp-client-devel-docs-0.30-8
qpid-cpp-client-0.30-8
qpid-cpp-client-rdma-0.30-8
qpid-java-client-0.30-5
qpid-cpp-server-devel-0.30-8
qpid-cpp-debuginfo-0.30-8

Bindings work as expected (Comment 9).

-> VERIFIED
Comment 14 errata-xmlrpc 2015-04-14 13:46:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2015-0805.html