Bug 1323834

Summary: backport "Use expiring queues instead of auto-delete ones in RabbitMQ driver" to oslo.messaging in OSP8
Product: Red Hat OpenStack Reporter: Zhen Qin <zhenqin>
Component: python-oslo-messagingAssignee: Victor Stinner <vstinner>
Status: CLOSED DUPLICATE QA Contact: Udi Shkalim <ushkalim>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: apevec, dcadzow, jeckersb, lhh, oblaut, yeylon
Target Milestone: ---Keywords: FeatureBackport
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-06 14:41:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Zhen Qin 2016-04-04 21:46:11 UTC
Description of problem:
If reconnection occurs between openstack components and rabbitmq, for example, triggered by the server reboot, there are some queues in rabbitmq become orphaned (a.k.a., no consumer consume them), but they still accumulate RPC or fanout messages, which cause possible memory pressure.

There are upstream fix to address this issue by setting TTL for direct and fanout queues instead of relying on auto_delete flag:
https://review.openstack.org/#/c/243845/
(it's has been backported to liberty: https://review.openstack.org/#/c/279561/)
https://review.openstack.org/#/c/292313/

The last patch is the latest, which changes the default value of rabbit_transient_queues_ttl to 30mins.

Version-Release number of selected component (if applicable):
python-oslo-messaging-2.5.0-4.el7ost.src.rpm

How reproducible:
Give rabbitmq node A has most master queues running, power off or restart the server where rabbitmq node A runs. Or following the reproduce steps in following bug reports:
https://bugs.launchpad.net/oslo.messaging/+bug/1515278
or
https://bugs.launchpad.net/oslo.messaging/+bug/1495568

Steps to Reproduce:
1.
2.
3.

Actual results:
Orphaned queues generated in rabbitmq, which can be verified by running 'rabbitmqctl list_queues name messages consumers' command.

Expected results:
no orphaned queues after a short period.

Additional info:
Related bug report in upstream:
https://bugs.launchpad.net/oslo.messaging/+bug/1515278
https://bugs.launchpad.net/oslo.messaging/+bug/1495568
https://bugs.launchpad.net/oslo.messaging/+bug/1545002

Comment 2 John Eckersberg 2016-04-06 14:41:17 UTC

*** This bug has been marked as a duplicate of bug 1310157 ***