RDO tickets are now tracked in Jira https://issues.redhat.com/projects/RDO/issues/
Bug 960539 - nova-compute and other oslo based qpid services appears to be leaking qpid exchanges
Summary: nova-compute and other oslo based qpid services appears to be leaking qpid ex...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RDO
Classification: Community
Component: openstack-nova
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Pádraig Brady
QA Contact: yeylon@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-05-07 11:06 UTC by Dave Neary
Modified: 2016-04-18 07:12 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 971565 971567 971571 971572 971574 (view as bug list)
Environment:
Last Closed: 2013-09-03 16:00:08 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1178375 0 None None None Never
OpenStack gerrit 32187 0 None None None Never

Description Dave Neary 2013-05-07 11:06:07 UTC
From RDO forum user bostrick: http://openstack.redhat.com/forum/discussion/85/is-nova-conductor-leaking-qpid-exchanges

Description of problem: After several hours of operation, there are a large number of qpid exchanges in existence, with a new one being created every ~10s.

To reproduce:
* Start OpenStack
* After a while, run qpid-stat -e lists open exchanges.

Actual results: Many exchanges reported.

Expected results: Few/no exchanges reported


Additional info:

qpid-printevents is displaying transactions similar to the following every 10 secs:

org.apache.qpid.broker:exchangeDeclare ... exName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:queueDeclare ... qName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:bind ... exName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:subscribe ... qName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:unsubscribe ...
org.apache.qpid.broker:queueDelete ... qName=d75b1d8f76ea46eead02b7480dd28cdf

notably missing is a complement to "exchangeDeclare" to clean it up.

tracing qpid client ports, nova-conductor seems to be the culprit.

Comment 2 Alan Pevec 2013-06-07 17:54:34 UTC
The issue is in the code copied from oslo-incubator (ex openstack-common):
$PROJECT/openstack/common/rpc/impl_qpid.py

Comment 3 Alan Pevec 2013-06-11 18:51:42 UTC
Workaround from upstream bug
https://bugs.launchpad.net/oslo/+bug/1178375/comments/7

Comment 4 Alan Pevec 2013-09-03 16:00:08 UTC
This was fixed by a workaround in RDO Grizzly openstack-nova-2013.1.2-3:
 http://pkgs.fedoraproject.org/cgit/openstack-nova.git/diff/0002-avoid-code-path-causing-qpid-exchange-leaks.patch?h=el6-grizzly&id=f858dc5c60b9c7aac1c26011fbdec164bdeecc82

Alternative (backward-incompatible for in-place upgrade) fix was proposed for stable/grizzly https://review.openstack.org/44893


Note You need to log in before you can comment on or make changes to this bug.