Red Hat Bugzilla – Bug 971574
ceilometer appears to be leaking qpid exchanges
Last modified: 2016-01-04 09:42:36 EST
+++ This bug was initially created as a clone of Bug #960539 +++ From RDO forum user bostrick: http://openstack.redhat.com/forum/discussion/85/is-nova-conductor-leaking-qpid-exchanges Description of problem: After several hours of operation, there are a large number of qpid exchanges in existence, with a new one being created every ~10s. To reproduce: * Start OpenStack * After a while, run qpid-stat -e lists open exchanges. Actual results: Many exchanges reported. Expected results: Few/no exchanges reported Additional info: qpid-printevents is displaying transactions similar to the following every 10 secs: org.apache.qpid.broker:exchangeDeclare ... exName=d75b1d8f76ea46eead02b7480dd28cdf ... org.apache.qpid.broker:queueDeclare ... qName=d75b1d8f76ea46eead02b7480dd28cdf ... org.apache.qpid.broker:bind ... exName=d75b1d8f76ea46eead02b7480dd28cdf ... org.apache.qpid.broker:subscribe ... qName=d75b1d8f76ea46eead02b7480dd28cdf ... org.apache.qpid.broker:unsubscribe ... org.apache.qpid.broker:queueDelete ... qName=d75b1d8f76ea46eead02b7480dd28cdf notably missing is a complement to "exchangeDeclare" to clean it up. tracing qpid client ports, nova-conductor seems to be the culprit. --- Additional comment from David Ripton on 2013-06-06 16:12:52 EDT --- Related upstream bugs: https://bugs.launchpad.net/oslo/+bug/1033915 https://bugs.launchpad.net/nova/+bug/1178375
The underlying issue is the oslo RPC layer. The patch to address that is currently stalled upstream with gate failures: https://review.openstack.org/32187 The usual olso "managed copy'n'paste" process can be followed once there's traction on this review.
QE ack. We have a reproducer and a proposed patch. Although the fact it is stalled upstream could cause some issues given the RC build schedule.
Fix suggested by Russel Bryant was to config out the problematic path: http://pkgs.devel.redhat.com/cgit/rpms/openstack-ceilometer/commit/?h=rh-grizzly-rhel-6&id=ff04ad624
# qpid-stat -e |wc -l 49 # qpid-stat -e |wc -l 49
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0968.html