Bug 960539

Summary:	nova-compute and other oslo based qpid services appears to be leaking qpid exchanges
Product:	[Community] RDO	Reporter:	Dave Neary <dneary>
Component:	openstack-nova	Assignee:	Pádraig Brady <pbrady>
Status:	CLOSED CURRENTRELEASE	QA Contact:	yeylon <yeylon>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	apevec, dripton, jkt, markmc, ndipanov, rbryant, srevivo, whenry
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	971565 971567 971571 971572 971574 (view as bug list)		Environment:
Last Closed:	2013-09-03 16:00:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dave Neary 2013-05-07 11:06:07 UTC

From RDO forum user bostrick: http://openstack.redhat.com/forum/discussion/85/is-nova-conductor-leaking-qpid-exchanges

Description of problem: After several hours of operation, there are a large number of qpid exchanges in existence, with a new one being created every ~10s.

To reproduce:
* Start OpenStack
* After a while, run qpid-stat -e lists open exchanges.

Actual results: Many exchanges reported.

Expected results: Few/no exchanges reported


Additional info:

qpid-printevents is displaying transactions similar to the following every 10 secs:

org.apache.qpid.broker:exchangeDeclare ... exName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:queueDeclare ... qName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:bind ... exName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:subscribe ... qName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:unsubscribe ...
org.apache.qpid.broker:queueDelete ... qName=d75b1d8f76ea46eead02b7480dd28cdf

notably missing is a complement to "exchangeDeclare" to clean it up.

tracing qpid client ports, nova-conductor seems to be the culprit.

Comment 1 David Ripton 2013-06-06 20:12:52 UTC

Related upstream bugs:

https://bugs.launchpad.net/oslo/+bug/1033915
https://bugs.launchpad.net/nova/+bug/1178375

Comment 2 Alan Pevec 2013-06-07 17:54:34 UTC

The issue is in the code copied from oslo-incubator (ex openstack-common):
$PROJECT/openstack/common/rpc/impl_qpid.py

Comment 3 Alan Pevec 2013-06-11 18:51:42 UTC

Workaround from upstream bug
https://bugs.launchpad.net/oslo/+bug/1178375/comments/7

Comment 4 Alan Pevec 2013-09-03 16:00:08 UTC

This was fixed by a workaround in RDO Grizzly openstack-nova-2013.1.2-3:
 http://pkgs.fedoraproject.org/cgit/openstack-nova.git/diff/0002-avoid-code-path-causing-qpid-exchange-leaks.patch?h=el6-grizzly&id=f858dc5c60b9c7aac1c26011fbdec164bdeecc82

Alternative (backward-incompatible for in-place upgrade) fix was proposed for stable/grizzly https://review.openstack.org/44893