Bug 666944 - at scale, the queue purge process can negatively impact message flow.
Summary: at scale, the queue purge process can negatively impact message flow.
Keywords:
Status: CLOSED DUPLICATE of bug 1093996
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: 1.3
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: ---
Assignee: messaging-bugs
QA Contact: MRG Quality Engineering
URL:
Whiteboard:
: 521292 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-01-03 19:33 UTC by Ken Giusti
Modified: 2014-10-10 11:38 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-10 11:38:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
testcase - loads up a set of queues with msgs with short TTLs. (3.19 KB, text/plain)
2011-01-03 21:45 UTC, Ken Giusti
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Apache JIRA QPID-5758 0 None None None Never

Description Ken Giusti 2011-01-03 19:33:55 UTC
Description of problem:

This problem was first encountered on the grid0 test bed.  Whenever the queue-purge process ran on the broker, QMF agents would time out due to loss of heartbeats.  On further investigation, a "stuck" console was found: this console had bound to the qmf topic exchanges, but was not retrieving messages.  This caused its queues to back up on the broker, creating a large number of expired messages for the queue-purge process to clean up.  When the queue-purge process ran, the broker would become CPU bound for over 30 seconds, which cased the erroneous agent timeouts.

Version-Release number of selected component (if applicable): qpid-cpp-server-0.7.946106-25.el5


How reproducible: 
Difficult - requires a large scale deployment, and some means for creating a large number of timed-out messages (expired ttls).


Steps to Reproduce:
1. Run a broker, and provide a constant traffic load.
2. Create a number of queues, fill them with messages with a short ttl (say 10 secs).
3. Observe impact of queue-purge process on traffic flow.
  
Actual results:


Expected results:


Additional info:

Comment 1 Justin Ross 2011-01-03 20:10:27 UTC
See bz 603896 for an earlier issue in the same vein.

Comment 2 Ken Giusti 2011-01-03 21:45:37 UTC
Created attachment 471549 [details]
testcase - loads up a set of queues with msgs with short TTLs.

Comment 3 Gordon Sim 2011-01-04 13:28:17 UTC
The queue purging and management processing happen on the same timer thread (is periodic processing in the brokers management agent connected with heartbeats at all?). Long running tasks will delay other periodic tasks.

Additionally while a specific queue is being purged, this will hold up any enqueue or dequeue attempts. This could be addressed by a finer grained scheme for locking.

Comment 4 Gordon Sim 2011-01-04 14:03:16 UTC
*** Bug 521292 has been marked as a duplicate of this bug. ***

Comment 5 Justin Ross 2014-10-10 11:38:40 UTC

*** This bug has been marked as a duplicate of bug 1093996 ***


Note You need to log in before you can comment on or make changes to this bug.