Bug 1450520

Summary: cinder-scheduler_fanout AMQP queues keeps increasing
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: python-oslo-messagingAssignee: Victor Stinner <vstinner>
Status: CLOSED WONTFIX QA Contact: Udi Shkalim <ushkalim>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0 (Kilo)CC: apevec, chjones, dhill, eharney, fpercoco, jeckersb, lhh, pgrist, srevivo, ssigwald, tshefi
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 7.0 (Kilo)Flags: tshefi: automate_bug-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-01 15:00:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Hill 2017-05-12 21:41:28 UTC 
Description of problem:
cinder-scheduler queues keeps increasing

[root@hostname01 cinder]# rabbitmqctl list_queues| grep cinder-scheduler
cinder-scheduler        0
cinder-scheduler.hostgroup      0
cinder-scheduler_fanout_0c87c0f9e6744c86b163d9ca199625aa        6701
cinder-scheduler_fanout_178f6369f7c84849bc1bcd8ce19367e6        0
cinder-scheduler_fanout_76ec3e3d0f6447cda0e005894581faef        0
cinder-scheduler_fanout_e97eac5de79848dd9ddfc3f2104e4c6a        0


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Restart cinder-scheduler on one of the controllers.
2. Kill one of the rabbitmq slave
3. cinder-scheduler_fanout starts accumulating

Actual results:
messages are no longer consumed in one of the fanout queues

Expected results:
Shouldn't happen

Additional info:
Comment 7 David Hill 2017-12-18 15:51:02 UTC 
When can we expect a backport for python-oslo-messaging or a confirmation of the feasibility of backporting those patches ?
Comment 8 John Eckersberg 2017-12-18 17:24:56 UTC 
I looked into the backport originally but I clearly forgot to update the bug when I did so.  I think I complained to Eric in the hall and then got distracted by something else.

From what I remember, the backport was going to be quite difficult, as oslo.messaging was heavily modified between Kilo and Juno.  I will check again to make sure I remember correctly, and/or Victor can weigh in with his opinion as well.
Comment 12 Victor Stinner 2018-03-01 15:00:35 UTC 
Backporting the change requires to modify a lot of code which is a high risk of regression. It was decided to not fix this issue in OSP 7.