Bug 971567 - quantum appears to be leaking qpid exchanges
quantum appears to be leaking qpid exchanges
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-quantum (Show other bugs)
3.0
Unspecified Unspecified
high Severity high
: rc
: 3.0
Assigned To: Terry Wilson
Rami Vaknin
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-06 16:16 EDT by Flavio Percoco
Modified: 2016-04-26 14:57 EDT (History)
10 users (show)

See Also:
Fixed In Version: openstack-quantum-2013.1.2-3.el6ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 960539
Environment:
Last Closed: 2013-06-27 13:16:35 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Flavio Percoco 2013-06-06 16:16:17 EDT
+++ This bug was initially created as a clone of Bug #960539 +++

From RDO forum user bostrick: http://openstack.redhat.com/forum/discussion/85/is-nova-conductor-leaking-qpid-exchanges

Description of problem: After several hours of operation, there are a large number of qpid exchanges in existence, with a new one being created every ~10s.

To reproduce:
* Start OpenStack
* After a while, run qpid-stat -e lists open exchanges.

Actual results: Many exchanges reported.

Expected results: Few/no exchanges reported


Additional info:

qpid-printevents is displaying transactions similar to the following every 10 secs:

org.apache.qpid.broker:exchangeDeclare ... exName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:queueDeclare ... qName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:bind ... exName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:subscribe ... qName=d75b1d8f76ea46eead02b7480dd28cdf ...
org.apache.qpid.broker:unsubscribe ...
org.apache.qpid.broker:queueDelete ... qName=d75b1d8f76ea46eead02b7480dd28cdf

notably missing is a complement to "exchangeDeclare" to clean it up.

tracing qpid client ports, nova-conductor seems to be the culprit.

--- Additional comment from David Ripton on 2013-06-06 16:12:52 EDT ---

Related upstream bugs:

https://bugs.launchpad.net/oslo/+bug/1033915
https://bugs.launchpad.net/nova/+bug/1178375
Comment 5 Rami Vaknin 2013-06-18 03:43:14 EDT
Verified on grizzly running on rhel6.4 with openstack-quantum-2013.1.2-3.el6ost and openstack-nova-conductor-2013.1.2-2.el6ost.

I've installed qpid-tools, the environment was alive for ~10 hours.

# qpid-printevents  (output of few mins)
Tue Jun 18 07:35:54 2013 NOTIC qpid-printevents:brokerConnected broker=localhost:5672
Tue Jun 18 07:35:56 2013 INFO  org.apache.qpid.broker:bind broker=localhost:5672 rhost=[::1]:5672-[::1]:59275 user=anonymous exName=qpid.management qName=topic-puma07.scl.lab.tlv.redhat.com.10055.1 key=console.event.# args={}
Tue Jun 18 07:35:56 2013 INFO  org.apache.qpid.broker:bind broker=localhost:5672 rhost=[::1]:5672-[::1]:59275 user=anonymous exName=qmf.default.topic qName=qmfc-v2-hb-puma07.scl.lab.tlv.redhat.com.10055.1 key=agent.ind.heartbeat.# args={}
Tue Jun 18 07:35:56 2013 INFO  org.apache.qpid.broker:bind broker=localhost:5672 rhost=[::1]:5672-[::1]:59275 user=anonymous exName=qmf.default.topic qName=qmfc-v2-ui-puma07.scl.lab.tlv.redhat.com.10055.1 key=agent.ind.event.# args={}


# qpid-stat -e 
Exchanges
  exchange                                       type     dur  bind  msgIn  msgOut  msgDrop  byteIn  byteOut  byteDrop
  ======================================================================================================================
  reply_95e7a09c8a9c4e2fafd360e560b01ee9         direct   Y       1     0      0       0        0       0        0
  conductor_fanout                               fanout           1     0      0       0        0       0        0
  q-agent-notifier-port-update_fanout            fanout           2     0      0       0        0       0        0
  reply_0abbd42e206040ac8cac3efd0c720d80         direct   Y       0     0      0       0        0       0        0
  consoleauth_fanout                             fanout           1     0      0       0        0       0        0
  cinder-scheduler_fanout                        fanout           1     0      0       0        0       0        0
  nova                                           topic           10  6.58k  6.58k      1     9.87m   9.87m     642
  cinder-volume_fanout                           fanout           0     1      0       1      449       0      449
  amq.fanout                                     fanout   Y       0     0      0       0        0       0        0
  reply_79081828c0c54ace9ccd7fc65616e153         direct   Y       0     0      0       0        0       0        0
  amq.match                                      headers  Y       0     0      0       0        0       0        0
  compute_fanout                                 fanout           1     2      1       1     1.10k    551      551
  reply_910f020830254cabacda75f060a72947         direct   Y       1     0      0       0        0       0        0
  q-agent-notifier-tunnel-update_fanout          fanout           2     0      0       0        0       0        0
  reply_651aebf10b3b4e0d8b7b69ac0c92d3e7         direct   Y       1     0      0       0        0       0        0
  openstack                                      topic    Y       2     0      0       0        0       0        0
  reply_3fa6a4f635a64b8a8cf3738ba698e497         direct   Y       1     0      0       0        0       0        0
  qmf.default.direct                             direct           1     9      9       0     65.4k   65.4k       0
  reply_dd68b04c4c974876bcb9b9900a6fbd2a         direct   Y       0     0      0       0        0       0        0
  reply_62c1e9f436774d3caec85b1e0739cec3         direct   Y       1     0      0       0        0       0        0
  reply_b6bd263d6ac948d8877fd4237252fe46         direct   Y       1     0      0       0        0       0        0
  reply_6ddf5c935ef3432eb231b60ae981c976         direct   Y       1     0      0       0        0       0        0
  amq.direct                                     direct   Y       1    58     58       0     23.6k   23.6k       0
  cert_fanout                                    fanout           1     0      0       0        0       0        0
  reply_42f4e6ecfa304409a657717ec2084a01         direct   Y       0     0      0       0        0       0        0
  qpid.management                                topic            3  29.2k     0    29.2k    74.2m      0     74.2m
  qmf.default.topic                              topic            1  29.0k     0    29.0k     189m      0      189m
  reply_28675bf5d3184e518dfb7fa63f5fe007         direct   Y       1     0      0       0        0       0        0
  dhcp_agent_fanout                              fanout           1     0      0       0        0       0        0
  reply_d7ad97e0c62e45a791f139bdbafa7f3d         direct   Y       0     0      0       0        0       0        0
  reply_b4917a552905457991072032c6c67130         direct   Y       0     0      0       0        0       0        0
                                                 direct          46  76.4k  76.4k      0     18.8m   18.8m       0
  q-agent-notifier-network-delete_fanout         fanout           2     0      0       0        0       0        0
  amq.topic                                      topic    Y       0     0      0       0        0       0        0
  l3_agent_fanout                                fanout           1     0      0       0        0       0        0
  scheduler_fanout                               fanout           1   549    549       0      747k    747k       0
  q-agent-notifier-security_group-update_fanout  fanout           2     8     16       0     3.47k   6.95k       0
  quantum                                        topic    Y       5  33.0k  32.9k      3     22.9m   22.9m    2.23k
Comment 7 errata-xmlrpc 2013-06-27 13:16:35 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0968.html

Note You need to log in before you can comment on or make changes to this bug.