Description of problem: There are two similar but not identical scenarios, where playing with bindings result in memory growth of qpidd process due to a queue's binding list growing forever. In one scenario, even with identical binding items. The bug is applicable to different exchange types (at least direct and topic ones are affected). Version-Release number of selected component (if applicable): qpid-cpp-server-1.36.0-22.el6.x86_64 (reproduced also in 0.34-21) How reproducible: 100% Steps to Reproduce: First scenario: repetitively create a link that creates the same binding every time: 1. qpid-config add queue Q1 while true; do qpid-receive -a "Q1; {link:{x-bindings:[{exchange:'amq.direct', key: 'some.key', queue:'Q1'}]}}"; done 2. time to time, check memory usage and also check in gdb: thread 1 frame 4 p brokerPtr->px->queues p ((qpid::broker::Queue *) 0x7f9024018a50)->bindings (where the pointer is the "px" of the queue Q1) Second scenario: create and delete different bindings: 1. qpid-config add queue Q1 j=0 while true; do j=$((j+1)) for i in $(seq 1 100); do for act in bind unbind; do qpid-config $act amq.direct Q1 some.key.${j}.${i} done done done 2. time to time, check memory usage and also check the same stuff in gdb. Actual results: 2. in both scenarios, memory grows over time as well as the "bindings = std::vector of length 46303, capacity 65536 .." grows over time. In the 1st scenario, _identical_ bindings are in the vector (that is one problem). In the 2nd scenario, _different_ bindings are in the vector (that is the _other_ bug/problem). Expected results: 2. no mem.leak, bindings vector stable over time Additional info: interestingly, "qpid-config bind/unbind" of the same binding does _not_ trigger the scenario 1.
Trial fix in https://github.com/cliffjansen/qpid-cpp (QPID-8397 branch).
I can confirm that qpid-cpp-1.36.0-22+hf1.el6_10 package fixes both the reproducers from #c0.
.. and also 1.36.0-22+hf2.el6_10 fixes both reproducers well.
There is yet at least one another scenario: deleting an exchange with bound queues does not remove the entries from bindings objects of the queues. Reproducer: qpid-config add queue Q while true; do qpid-config add exchange direct DirectEx for i in $(seq 1 100); do qpid-receive -a "Q; {create:always, node:{ type:queue, x-bindings:[{exchange:'DirectEx', queue:'Q', key:'k$i'}]}}" & done wait qpid-config del exchange DirectEx sleep 0.1 done Each iteration creates an exchange and 100 bindings to it, and deletes the exchange. Any time, qpid-config shows at most the 100 bindings between the DirectEx and Q, while internally the "bindings" vector in the Queue object of Q grows over time.
Yet another one (tricky, not hit by the customer): As n Broker.cpp , unbind method: } else { if (exchange->unbind(queue, key, 0)) { queue->unbound(exchange->getName(), key); might unbind from exchange but not from queue, when the exchange->unbind returns false. That is possible e.g. under scenario where routing key is unified in TopicExchange::Normalizer like: qpid-config add queue A qpid-receive -a "A; { link:{x-bindings:[{exchange:amq.topic, key:'*.#', queue:A}]}}" --timeout=2 qpid-receive -a "A; { link:{x-bindings:[{exchange:amq.topic, key:'#.*', queue:A}]}}" --timeout=1 This leaves one binding object in Queue's vector, like: {exchange = "amq.topic", key = "*.#", .. } Neither this or "delete exchange" from previous post is hit by the customer (though at least the "delete exchange" scenario is worth fixing, I think).
OK, the reproducer from #c7 is 99% the one hit by the customer: start_broker 5001 # starts qpidd on localhost:5001 start_broker 6001 # starts qpidd on localhost:6001 qpid-route dynamic add localhost:6001 localhost:5001 amq.topic qpid-config add queue BindQueue -b localhost:5001 qpid-config add queue BindQueue -b localhost:6001 for i in $(seq 1 100); do qpid-receive -b localhost:6001 -a "BindQueue; {create:always, link:{x-bindings:[{exchange:amq.topic, key:some.key, queue:BindQueue}]}}" done Either repeat the for cycle long time (for any key or even queue), and monitor memory usage on source broker (port 5001). Optionally, gdb the broker, find Queue object of a queue like qpid.bridge_queue__qpid.tcp:localhost:5001!amq.topic!amq.topic!_5390860c-f550-47a5-95af-99130595ed35 , and print that object's "bindings"
(In reply to Pavel Moravec from comment #9) > OK, the reproducer from #c7 is 99% the one hit by the customer: > > start_broker 5001 # starts qpidd on localhost:5001 > start_broker 6001 # starts qpidd on localhost:6001 > > qpid-route dynamic add localhost:6001 localhost:5001 amq.topic > > qpid-config add queue BindQueue -b localhost:5001 > qpid-config add queue BindQueue -b localhost:6001 > > for i in $(seq 1 100); do > qpid-receive -b localhost:6001 -a "BindQueue; {create:always, > link:{x-bindings:[{exchange:amq.topic, key:some.key, queue:BindQueue}]}}" > done > > > Either repeat the for cycle long time (for any key or even queue), and > monitor memory usage on source broker (port 5001). > > Optionally, gdb the broker, find Queue object of a queue like > qpid.bridge_queue__qpid.tcp:localhost:5001!amq.topic!amq.topic!_5390860c- > f550-47a5-95af-99130595ed35 , and print that object's "bindings" .. and this is fixed in qpid-cpp-server-1.36.0-22+hf3.el6_10.x86_64 . No other scenario tested against this build.