| Summary: | Reaching queue capacity limit causes delivery failure on other queues with same binding | ||
|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Petr Matousek <pematous> |
| Component: | qpid-cpp | Assignee: | Ken Giusti <kgiusti> |
| Status: | NEW --- | QA Contact: | MRG Quality Engineering <mrgqe-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | medium | ||
| Version: | Development | CC: | iboverma, jross |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | --- | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Description of problem: 1. start broker with producer workflow disabled 2. create several queues all or some of which have a limit configured, bind these queues to e.g. amq.topic all with the same key 3. send some messages through the exchange to those queues; do not hit the limit yet 4. drain messages from all queues except one of them (say q1). 5. send some more messages (3..N) through the exchange to those queues, cause the non-empty queue q1 to reach its capacity limit (more messages must be sent at once, sending message by message works well) 6. use qpid-stat -q or similar tool to list messages count on each queue. Independently on messages count sent step 5, the result is that the q1 has reached the capacity limit and other queues contains the messages sent before the q1 capacity limit was reached and just one additional message, other messages are lost. notice: when c++ sender is used to send the messages, there is one extra message, this message is already contained on all the queues, but is for some reason redelivered. When using python or java sender this message won't appear. Version-Release number of selected component (if applicable): python-qpid-0.10-1.el5 python-qpid-qmf-0.10-8.el5 qpid-cpp-client-0.10-7.el5 qpid-cpp-client-devel-0.10-7.el5 qpid-cpp-client-devel-docs-0.10-7.el5 qpid-cpp-client-rdma-0.10-7.el5 qpid-cpp-client-ssl-0.10-7.el5 qpid-cpp-mrg-debuginfo-0.10-7.el5 qpid-cpp-server-0.10-7.el5 qpid-cpp-server-cluster-0.10-7.el5 qpid-cpp-server-devel-0.10-7.el5 qpid-cpp-server-rdma-0.10-7.el5 qpid-cpp-server-ssl-0.10-7.el5 qpid-cpp-server-store-0.10-7.el5 qpid-cpp-server-xml-0.10-7.el5 qpid-java-client-0.10-6.el5 qpid-java-common-0.10-6.el5 qpid-java-example-0.10-6.el5 qpid-qmf-0.10-8.el5 qpid-qmf-devel-0.10-8.el5 qpid-tools-0.10-5.el5 rh-qpid-cpp-tests-0.10-7.el5 How reproducible: 100% Steps to Reproduce: 1. qpidd --default-flow-stop-threshold 0 --default-flow-resume-threshold 0 & 2. for q in q1 q2 ; do qpid-config add queue $q --max-queue-count 5; qpid-config bind amq.topic $q key; done 3. /usr/share/doc/python-qpid-0.10/examples/api/spout -c 4 "amq.topic/key" 4. /usr/share/doc/python-qpid-0.10/examples/api/drain "q2" 5. /usr/share/doc/python-qpid-0.10/examples/api/spout -c 4 "amq.topic/key" # OR /opt/rh-qpid/clients/qpid-send --messages 100 --address amq.topic/key # OR /usr/share/doc/qpid-java-0.10/examples/run_example.sh org.apache.qpid.example.Spout -c 10 "amq.topic/key" 6. qpid-stat -q 7. for q in q1 q2; do echo $q; /usr/share/doc/python-qpid-0.10/examples/api/drain "$q;{mode: browse}"; done Please see the Additional info. Actual results: Only one additional messages is queued after any of the queues hits its capacity, other messages are lost. Expected results: All messages sent by the sender are queued to all relevant queues that didn't hit the capacity limit Additional info: See this example, on the end of this code the queue q2 shall contain 4 messages and messagesIn count shall be equal to 8, but the queue contains just 2 messages, messagesIn=6. # qpidd --default-flow-stop-threshold 0 --default-flow-resume-threshold 0 & [1] 14212 # for q in q1 q2 ; do qpid-config add queue $q --max-queue-count 5; qpid-config bind amq.topic $q key; done # /usr/share/doc/python-qpid-0.10/examples/api/spout -c 4 "amq.topic/key" Message(properties={'qpid.subject': 'key', 'spout-id': 'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:0'}) Message(properties={'qpid.subject': 'key', 'spout-id': 'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:1'}) Message(properties={'qpid.subject': 'key', 'spout-id': 'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:2'}) Message(properties={'qpid.subject': 'key', 'spout-id': 'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:3'}) # /usr/share/doc/python-qpid-0.10/examples/api/drain "q2" Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:0'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:1'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:2'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:3'}) # /usr/share/doc/python-qpid-0.10/examples/api/spout -c 4 "amq.topic/key" Message(properties={'qpid.subject': 'key', 'spout-id': 'f7e0cf42-11ed-6245-b927-9b95583c9cd0:0'}) 2011-05-20 16:25:31 warning Exchange amq.topic cannot deliver to queue q1: resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87) 2011-05-20 16:25:31 error Execution exception: resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87) resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87)(506) Traceback (most recent call last): File "/usr/share/doc/python-qpid-0.10/examples/api/spout", line 123, in ? conn.close() File "<string>", line 6, in close File "/usr/lib/python2.4/site-packages/qpid/messaging/endpoints.py", line 304, in close ssn.close(timeout=timeout) File "<string>", line 6, in close File "/usr/lib/python2.4/site-packages/qpid/messaging/endpoints.py", line 727, in close self.sync(timeout=timeout) File "<string>", line 6, in sync File "/usr/lib/python2.4/site-packages/qpid/messaging/endpoints.py", line 718, in sync snd.sync(timeout=timeout) File "<string>", line 6, in sync File "/usr/lib/python2.4/site-packages/qpid/messaging/endpoints.py", line 873, in sync if not self._ewait(lambda: self.acked >= mno, timeout=timeout): File "/usr/lib/python2.4/site-packages/qpid/messaging/endpoints.py", line 787, in _ewait result = self.session._ewait(lambda: self.error or predicate(), timeout) File "/usr/lib/python2.4/site-packages/qpid/messaging/endpoints.py", line 555, in _ewait self.check_error() File "/usr/lib/python2.4/site-packages/qpid/messaging/endpoints.py", line 544, in check_error raise self.error qpid.messaging.exceptions.TargetCapacityExceeded: resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87)(506) # qpid-stat -q Queues queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind ====================================================================================================================================== qmfc-v2-hb-dhcp-37-228.lab.eng.brq.redhat.com.14262.1 Y Y 0 0 0 0 0 0 1 2 q1 5 5 0 0 0 0 0 2 q2 2 6 4 0 0 0 0 2 qmfagent-161c7cc5-1236-438c-aa47-c38686ebdbec Y Y 0 10 10 0 170 170 1 4 qmfc-v2-ui-dhcp-37-228.lab.eng.brq.redhat.com.14262.1 Y Y 0 0 0 0 0 0 1 1 reply-dhcp-37-228.lab.eng.brq.redhat.com.14262.1 Y Y 0 58 58 0 22.2k 22.2k 1 2 qmfc-v2-dhcp-37-228.lab.eng.brq.redhat.com.14262.1 Y Y 0 13 13 0 10.1k 10.1k 1 2 topic-dhcp-37-228.lab.eng.brq.redhat.com.14262.1 Y Y 0 0 0 0 0 0 1 4 # for q in q1 q2; do echo $q; /usr/share/doc/python-qpid-0.10/examples/api/drain "$q;{mode: browse}"; done q1 Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:0'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:1'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:2'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'bf4ab9ae-2629-174b-b1c0-d92a0ea2d027:3'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'f7e0cf42-11ed-6245-b927-9b95583c9cd0:0'}) q2 Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'f7e0cf42-11ed-6245-b927-9b95583c9cd0:0'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'f7e0cf42-11ed-6245-b927-9b95583c9cd0:1'}) --- An extra message is present when c++ sender is used: ... <snip> ... # /opt/rh-qpid/clients/qpid-send --messages 100 --address amq.topic/key 2011-05-20 16:35:03 warning Exchange amq.topic cannot deliver to queue q1: resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87) 2011-05-20 16:35:03 error Execution exception: resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87) 2011-05-20 16:35:03 warning Exception received from broker: resource-limit-exceeded: resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87) [caused by 4 \x00:\x00] 2011-05-20 16:35:03 warning Ignoring frame while closing connection: Frame[BEbe; channel=1; {SessionDetachBody: name=3f281309-285c-4ba6-8562-bfd7fe4dbc2b; }] 2011-05-20 16:35:03 warning Exchange amq.topic cannot deliver to queue q1: resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87) 2011-05-20 16:35:03 error Execution exception: resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87) 2011-05-20 16:35:03 warning Exception received from broker: resource-limit-exceeded: resource-limit-exceeded: resource-limit-exceeded: Policy exceeded on q1, policy: size: max=104857600, current=0; count: max=5, current=5; type=reject (qpid/broker/QueuePolicy.cpp:87) [caused by 1 \x00:\x00] 2011-05-20 16:35:03 warning Ignoring frame while closing connection: Frame[BEbe; channel=1; {SessionDetachBody: name=3f281309-285c-4ba6-8562-bfd7fe4dbc2b; }] Failed: Failed to connect (reconnect disabled) # qpid-stat -q Queues queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind ====================================================================================================================================== qmfc-v2-dhcp-37-228.lab.eng.brq.redhat.com.14508.1 Y Y 0 13 13 0 10.1k 10.1k 1 2 q2 3 7 4 0 0 0 0 2 q1 5 5 0 0 0 0 0 2 topic-dhcp-37-228.lab.eng.brq.redhat.com.14508.1 Y Y 0 0 0 0 0 0 1 4 qmfc-v2-hb-dhcp-37-228.lab.eng.brq.redhat.com.14508.1 Y Y 0 0 0 0 0 0 1 2 reply-dhcp-37-228.lab.eng.brq.redhat.com.14508.1 Y Y 0 58 58 0 22.2k 22.2k 1 2 qmfc-v2-ui-dhcp-37-228.lab.eng.brq.redhat.com.14508.1 Y Y 0 0 0 0 0 0 1 1 qmfagent-161c7cc5-1236-438c-aa47-c38686ebdbec Y Y 0 10 10 0 170 170 1 4 # for q in q1 q2; do echo $q; /usr/share/doc/python-qpid-0.10/examples/api/drain "$q;{mode: browse}"; done q1 Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'6d6f56c7-8981-524c-886e-a71df1d6d56a:0'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'6d6f56c7-8981-524c-886e-a71df1d6d56a:1'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'6d6f56c7-8981-524c-886e-a71df1d6d56a:2'}) Message(subject=u'key', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'spout-id': u'6d6f56c7-8981-524c-886e-a71df1d6d56a:3'}) Message(subject=u'key', user_id='', correlation_id='', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'sn': 1L, u'ts': 1305902103041278000L}, content_type=u'') q2 Message(subject=u'key', user_id='', correlation_id='', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'sn': 1L, u'ts': 1305902103041278000L}, content_type=u'') Message(subject=u'key', user_id='', correlation_id='', properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'sn': 2L, u'ts': 1305902103041354000L}, content_type=u'') Message(subject=u'key', user_id='', correlation_id='', redelivered=True, properties={u'qpid.subject': u'key', 'x-amqp-0-10.routing-key': u'key', u'sn': 1L, u'ts': 1305902103041278000L}, content_type=u'')