Bug 666944

Summary: at scale, the queue purge process can negatively impact message flow.
Product: Red Hat Enterprise MRG Reporter: Ken Giusti <kgiusti>
Component: qpid-cppAssignee: messaging-bugs <messaging-bugs>
Status: CLOSED DUPLICATE QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: medium Docs Contact:
Priority: low    
Version: 1.3CC: cctrieloff, gsim, jross, matt
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-10 11:38:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
testcase - loads up a set of queues with msgs with short TTLs. none

Description Ken Giusti 2011-01-03 19:33:55 UTC 
Description of problem:

This problem was first encountered on the grid0 test bed.  Whenever the queue-purge process ran on the broker, QMF agents would time out due to loss of heartbeats.  On further investigation, a "stuck" console was found: this console had bound to the qmf topic exchanges, but was not retrieving messages.  This caused its queues to back up on the broker, creating a large number of expired messages for the queue-purge process to clean up.  When the queue-purge process ran, the broker would become CPU bound for over 30 seconds, which cased the erroneous agent timeouts.

Version-Release number of selected component (if applicable): qpid-cpp-server-0.7.946106-25.el5


How reproducible: 
Difficult - requires a large scale deployment, and some means for creating a large number of timed-out messages (expired ttls).


Steps to Reproduce:
1. Run a broker, and provide a constant traffic load.
2. Create a number of queues, fill them with messages with a short ttl (say 10 secs).
3. Observe impact of queue-purge process on traffic flow.
  
Actual results:


Expected results:


Additional info:
Comment 1 Justin Ross 2011-01-03 20:10:27 UTC 
See bz 603896 for an earlier issue in the same vein.
Comment 2 Ken Giusti 2011-01-03 21:45:37 UTC 
Created attachment 471549 [details]
testcase - loads up a set of queues with msgs with short TTLs.
Comment 3 Gordon Sim 2011-01-04 13:28:17 UTC 
The queue purging and management processing happen on the same timer thread (is periodic processing in the brokers management agent connected with heartbeats at all?). Long running tasks will delay other periodic tasks.

Additionally while a specific queue is being purged, this will hold up any enqueue or dequeue attempts. This could be addressed by a finer grained scheme for locking.
Comment 4 Gordon Sim 2011-01-04 14:03:16 UTC 
*** Bug 521292 has been marked as a duplicate of this bug. ***
Comment 5 Justin Ross 2014-10-10 11:38:40 UTC 

*** This bug has been marked as a duplicate of bug 1093996 ***