Bug 971574

Summary:	ceilometer appears to be leaking qpid exchanges
Product:	Red Hat OpenStack	Reporter:	Flavio Percoco <fpercoco>
Component:	openstack-ceilometer	Assignee:	Pádraig Brady <pbrady>
Status:	CLOSED ERRATA	QA Contact:	Ami Jeain <ajeain>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.0	CC:	apevec, breeler, dneary, dripton, jruzicka, jturner, ndipanov, pbrady, sclewis, yeylon, ykaul
Target Milestone:	rc
Target Release:	3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-ceilometer-2013.1.2-2.el6ost	Doc Type:	Bug Fix
Doc Text:	Previously, multiple qpid exchanges were created by Ceilometer but not removed if not required, leading to system out of memory errors. This has been fixed by appropriate configuration so that unnecessary qpid exchanges are removed.	Story Points:	---
Clone Of:	960539	Environment:
Last Closed:	2013-06-27 17:16:57 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Flavio Percoco 2013-06-06 20:26:15 UTC

+++ This bug was initially created as a clone of Bug #960539 +++

From RDO forum user bostrick: http://openstack.redhat.com/forum/discussion/85/is-nova-conductor-leaking-qpid-exchanges

Description of problem: After several hours of operation, there are a large number of qpid exchanges in existence, with a new one being created every ~10s.

To reproduce:
* Start OpenStack
* After a while, run qpid-stat -e lists open exchanges.

Actual results: Many exchanges reported.

Expected results: Few/no exchanges reported


Additional info:

qpid-printevents is displaying transactions similar to the following every 10 secs:

org.apache.qpid.broker:exchangeDeclare ... exName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:queueDeclare ... qName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:bind ... exName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:subscribe ... qName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:unsubscribe ...
org.apache.qpid.broker:queueDelete ... qName=d75b1d8f76ea46eead02b7480dd28cdf

notably missing is a complement to "exchangeDeclare" to clean it up.

tracing qpid client ports, nova-conductor seems to be the culprit.

--- Additional comment from David Ripton on 2013-06-06 16:12:52 EDT ---

Related upstream bugs:

https://bugs.launchpad.net/oslo/+bug/1033915
https://bugs.launchpad.net/nova/+bug/1178375

Comment 1 Eoghan Glynn 2013-06-10 14:15:23 UTC

The underlying issue is the oslo RPC layer.

The patch to address that is currently stalled upstream with gate failures:

  https://review.openstack.org/32187

The usual olso "managed copy'n'paste" process can be followed once there's traction on this review.

Comment 2 Jay Turner 2013-06-10 16:51:31 UTC

QE ack.  We have a reproducer and a proposed patch.  Although the fact it is stalled upstream could cause some issues given the RC build schedule.

Comment 3 Pádraig Brady 2013-06-11 16:02:04 UTC

Fix suggested by Russel Bryant was to config out the problematic path:
http://pkgs.devel.redhat.com/cgit/rpms/openstack-ceilometer/commit/?h=rh-grizzly-rhel-6&id=ff04ad624

Comment 5 Ami Jeain 2013-06-17 12:32:58 UTC

#  qpid-stat -e |wc -l
49
#  qpid-stat -e |wc -l
49

Comment 8 errata-xmlrpc 2013-06-27 17:16:57 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0968.html