+++ This bug was initially created as a clone of Bug #960539 +++ From RDO forum user bostrick: http://openstack.redhat.com/forum/discussion/85/is-nova-conductor-leaking-qpid-exchanges Description of problem: After several hours of operation, there are a large number of qpid exchanges in existence, with a new one being created every ~10s. To reproduce: * Start OpenStack * After a while, run qpid-stat -e lists open exchanges. Actual results: Many exchanges reported. Expected results: Few/no exchanges reported Additional info: qpid-printevents is displaying transactions similar to the following every 10 secs: org.apache.qpid.broker:exchangeDeclare ... exName=d75b1d8f76ea46eead02b7480dd28cdf ... org.apache.qpid.broker:queueDeclare ... qName=d75b1d8f76ea46eead02b7480dd28cdf ... org.apache.qpid.broker:bind ... exName=d75b1d8f76ea46eead02b7480dd28cdf ... org.apache.qpid.broker:subscribe ... qName=d75b1d8f76ea46eead02b7480dd28cdf ... org.apache.qpid.broker:unsubscribe ... org.apache.qpid.broker:queueDelete ... qName=d75b1d8f76ea46eead02b7480dd28cdf notably missing is a complement to "exchangeDeclare" to clean it up. tracing qpid client ports, nova-conductor seems to be the culprit. --- Additional comment from David Ripton on 2013-06-06 16:12:52 EDT --- Related upstream bugs: https://bugs.launchpad.net/oslo/+bug/1033915 https://bugs.launchpad.net/nova/+bug/1178375
Verified on grizzly running on rhel6.4 with openstack-quantum-2013.1.2-3.el6ost and openstack-nova-conductor-2013.1.2-2.el6ost. I've installed qpid-tools, the environment was alive for ~10 hours. # qpid-printevents (output of few mins) Tue Jun 18 07:35:54 2013 NOTIC qpid-printevents:brokerConnected broker=localhost:5672 Tue Jun 18 07:35:56 2013 INFO org.apache.qpid.broker:bind broker=localhost:5672 rhost=[::1]:5672-[::1]:59275 user=anonymous exName=qpid.management qName=topic-puma07.scl.lab.tlv.redhat.com.10055.1 key=console.event.# args={} Tue Jun 18 07:35:56 2013 INFO org.apache.qpid.broker:bind broker=localhost:5672 rhost=[::1]:5672-[::1]:59275 user=anonymous exName=qmf.default.topic qName=qmfc-v2-hb-puma07.scl.lab.tlv.redhat.com.10055.1 key=agent.ind.heartbeat.# args={} Tue Jun 18 07:35:56 2013 INFO org.apache.qpid.broker:bind broker=localhost:5672 rhost=[::1]:5672-[::1]:59275 user=anonymous exName=qmf.default.topic qName=qmfc-v2-ui-puma07.scl.lab.tlv.redhat.com.10055.1 key=agent.ind.event.# args={} # qpid-stat -e Exchanges exchange type dur bind msgIn msgOut msgDrop byteIn byteOut byteDrop ====================================================================================================================== reply_95e7a09c8a9c4e2fafd360e560b01ee9 direct Y 1 0 0 0 0 0 0 conductor_fanout fanout 1 0 0 0 0 0 0 q-agent-notifier-port-update_fanout fanout 2 0 0 0 0 0 0 reply_0abbd42e206040ac8cac3efd0c720d80 direct Y 0 0 0 0 0 0 0 consoleauth_fanout fanout 1 0 0 0 0 0 0 cinder-scheduler_fanout fanout 1 0 0 0 0 0 0 nova topic 10 6.58k 6.58k 1 9.87m 9.87m 642 cinder-volume_fanout fanout 0 1 0 1 449 0 449 amq.fanout fanout Y 0 0 0 0 0 0 0 reply_79081828c0c54ace9ccd7fc65616e153 direct Y 0 0 0 0 0 0 0 amq.match headers Y 0 0 0 0 0 0 0 compute_fanout fanout 1 2 1 1 1.10k 551 551 reply_910f020830254cabacda75f060a72947 direct Y 1 0 0 0 0 0 0 q-agent-notifier-tunnel-update_fanout fanout 2 0 0 0 0 0 0 reply_651aebf10b3b4e0d8b7b69ac0c92d3e7 direct Y 1 0 0 0 0 0 0 openstack topic Y 2 0 0 0 0 0 0 reply_3fa6a4f635a64b8a8cf3738ba698e497 direct Y 1 0 0 0 0 0 0 qmf.default.direct direct 1 9 9 0 65.4k 65.4k 0 reply_dd68b04c4c974876bcb9b9900a6fbd2a direct Y 0 0 0 0 0 0 0 reply_62c1e9f436774d3caec85b1e0739cec3 direct Y 1 0 0 0 0 0 0 reply_b6bd263d6ac948d8877fd4237252fe46 direct Y 1 0 0 0 0 0 0 reply_6ddf5c935ef3432eb231b60ae981c976 direct Y 1 0 0 0 0 0 0 amq.direct direct Y 1 58 58 0 23.6k 23.6k 0 cert_fanout fanout 1 0 0 0 0 0 0 reply_42f4e6ecfa304409a657717ec2084a01 direct Y 0 0 0 0 0 0 0 qpid.management topic 3 29.2k 0 29.2k 74.2m 0 74.2m qmf.default.topic topic 1 29.0k 0 29.0k 189m 0 189m reply_28675bf5d3184e518dfb7fa63f5fe007 direct Y 1 0 0 0 0 0 0 dhcp_agent_fanout fanout 1 0 0 0 0 0 0 reply_d7ad97e0c62e45a791f139bdbafa7f3d direct Y 0 0 0 0 0 0 0 reply_b4917a552905457991072032c6c67130 direct Y 0 0 0 0 0 0 0 direct 46 76.4k 76.4k 0 18.8m 18.8m 0 q-agent-notifier-network-delete_fanout fanout 2 0 0 0 0 0 0 amq.topic topic Y 0 0 0 0 0 0 0 l3_agent_fanout fanout 1 0 0 0 0 0 0 scheduler_fanout fanout 1 549 549 0 747k 747k 0 q-agent-notifier-security_group-update_fanout fanout 2 8 16 0 3.47k 6.95k 0 quantum topic Y 5 33.0k 32.9k 3 22.9m 22.9m 2.23k
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0968.html