Bug 1538058 - [RFE] RabbitMQ durable queues lead to fail of event handler for OpenStack AMQP
Summary: [RFE] RabbitMQ durable queues lead to fail of event handler for OpenStack AMQP
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.9.0
Hardware: All
OS: All
unspecified
high
Target Milestone: GA
: 5.10.0
Assignee: Scott Seago
QA Contact: Danylo Kholodov
URL:
Whiteboard:
Keywords: FutureFeature
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-24 11:56 UTC by Neha Chugh
Modified: 2019-02-07 23:01 UTC (History)
11 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2019-02-07 23:01:05 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:0212 None None None 2019-02-07 23:01 UTC

Description Neha Chugh 2018-01-24 11:56:54 UTC
1. Proposed title of this feature request.
"RabbitMQ durable queues lead to fail of event handler for OpenStack AMQP"

2. Who is the customer behind the request?
Account name: (Acct. # 645275) 
Strategic Customer: yes 

3. What is the nature and description of the request? 

OpenStack HA guide have recommendation to set durable queues/exchanges for rabbitmq:

https://docs.openstack.org/ha-guide/shared-messaging.html

But currently CFME cannot use these queues/exchanges because there is no options to configure such connection:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[----] E, [2018-01-13T00:38:19.931742 #16419:a6516c] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::EventCatcher::Runner#start_event_monitor) EMS [miq] as [openstackadmin] Event Monitor Thread aborted because [PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'false' but current is 'true']
[----] E, [2018-01-13T00:38:19.931950 #16419:a6516c] ERROR -- : [Bunny::PreconditionFailed]: PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'false' but current is 'true'  Method:[rescue in block in start_event_monitor]
[----] E, [2018-01-13T00:38:19.932041 #16419:a6516c] ERROR -- : /opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/channel.rb:1946:in `raise_if_continuation_resulted_in_a_channel_error!'
/opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/channel.rb:1179:in `exchange_declare'
/opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/exchange.rb:245:in `declare!'
/opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/exchange.rb:83:in `initialize'
/opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/channel.rb:344:in `new'
/opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/channel.rb:344:in `topic'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-dd41a2772297/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb:111:in `block in initialize_queues'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-dd41a2772297/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb:110:in `each'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-dd41a2772297/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb:110:in `initialize_queues'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-dd41a2772297/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb:77:in `start'
/var/www/miq/vmdb/app/models/manageiq/providers/openstack/event_catcher_mixin.rb:38:in `monitor_events'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/event_catcher/runner.rb:164:in `block in start_event_monitor'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From the source code It has been found that queues initialization does not use any settings for durable exchange initialization:

https://github.com/ManageIQ/manageiq-gems-pending/blob/fine/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb#L111

But Ruby Bunny have such ability:

http://rubybunny.info/articles/durability.html

Customer has  tested the following and it's work as expected with durable exchanges for nova/neutron etc..:

~~~~~~~~~~~~~~~~~~~~~
  def initialize_queues(channel)
    remove_legacy_queues
    @queues = {}
    if @options[:topics]
      @options[:topics].each do |exchange, topic|
        #init durable exchange !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
        amqp_exchange = channel.topic(exchange, :durable => true)
        queue_name = "miq-#{@client_ip}-#{exchange}"
        @queues[exchange] = channel.queue(queue_name, :auto_delete => true, :exclusive => true)
                                   .bind(amqp_exchange, :routing_key => topic)
      end
    end
  end
~~~~~~~~~~~~~~~~~~~~~

4. Why does the customer need this? (List the business requirements here) 
RabbitMQ durable queues lead to fail of event handler for OpenStack AMQP.

5. How would the customer like to achieve this? (List the functional requirements here) 
In CFME Advanced settings should be parameter to enable this.

6. For each functional requirement listed in question 5, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented. 

RedHat itself and on customer's test infrastructure

7. Is there already an existing RFE upstream or in Red Hat bugzilla?
No

8. Does the customer have any specific timeline dependencies?
As soon as possible.

9. List any affected packages or components.
manageiq-gems-pending


11. Would the customer be able to assist in testing this functionality if implemented? 
Yes.

Comment 2 Bronagh Sorota 2018-02-26 20:57:57 UTC
Sending to PM for prioritization.

Comment 4 CFME Bot 2018-09-11 19:13:25 UTC
New commit detected on ManageIQ/manageiq-providers-openstack/master:

https://github.com/ManageIQ/manageiq-providers-openstack/commit/fd6cc79deeda0007af1275f4cd9d1464e0c33cee
commit fd6cc79deeda0007af1275f4cd9d1464e0c33cee
Author:     Scott Seago <sseago@redhat.com>
AuthorDate: Mon Sep 10 11:23:51 2018 -0400
Commit:     Scott Seago <sseago@redhat.com>
CommitDate: Mon Sep 10 11:23:51 2018 -0400

    bug 1538058: Retry bunny queue initialization for durable queues

    https://bugzilla.redhat.com/show_bug.cgi?id=1538058

    If rabbitmq is using durable queue/exchanges, they must be opened
    on the ManageIQ end with durable => true. If exchange creation
    fails with Bunny::PreconditionFailed, retry with durable=>true.

 lib/manageiq/providers/openstack/legacy/events/openstack_rabbit_event_monitor.rb | 30 +-
 1 file changed, 24 insertions(+), 6 deletions(-)

Comment 8 Scott Seago 2018-12-10 14:58:02 UTC
Danylo,

Regarding your first question, there's actually nothing that needs to be done to enable this. If connecting to the queue fails because it is durable, CFME tries again setting the durable flag to 'true'.

Regarding the second, I'm not quite sure I understand what you're asking. "use CFME in some way" -- if the durable queue already exists, without this fix, trying to use AMQP messaging with the provider will fail since the attempt to create/connect to a non-durable queue will fail because the queue already exists as a durable one. With this fix, using AMQP will succeed. Perhaps, before testing this fix, make sure that you're setting up the durable queue properly by confirming that using AMQP fails for you if you are using a build without this fix. Then, if that fails, it should succeed with the fix in place.

Comment 9 Danylo Kholodov 2018-12-18 12:26:31 UTC
Verified at the CFME 5.10.0.29

Comment 10 errata-xmlrpc 2019-02-07 23:01:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:0212


Note You need to log in before you can comment on or make changes to this bug.