1. Proposed title of this feature request. "RabbitMQ durable queues lead to fail of event handler for OpenStack AMQP" 2. Who is the customer behind the request? Account name: (Acct. # 645275) Strategic Customer: yes 3. What is the nature and description of the request? OpenStack HA guide have recommendation to set durable queues/exchanges for rabbitmq: https://docs.openstack.org/ha-guide/shared-messaging.html But currently CFME cannot use these queues/exchanges because there is no options to configure such connection: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [----] E, [2018-01-13T00:38:19.931742 #16419:a6516c] ERROR -- : MIQ(ManageIQ::Providers::Openstack::CloudManager::EventCatcher::Runner#start_event_monitor) EMS [miq] as [openstackadmin] Event Monitor Thread aborted because [PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'false' but current is 'true'] [----] E, [2018-01-13T00:38:19.931950 #16419:a6516c] ERROR -- : [Bunny::PreconditionFailed]: PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost '/': received 'false' but current is 'true' Method:[rescue in block in start_event_monitor] [----] E, [2018-01-13T00:38:19.932041 #16419:a6516c] ERROR -- : /opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/channel.rb:1946:in `raise_if_continuation_resulted_in_a_channel_error!' /opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/channel.rb:1179:in `exchange_declare' /opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/exchange.rb:245:in `declare!' /opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/exchange.rb:83:in `initialize' /opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/channel.rb:344:in `new' /opt/rh/cfme-gemset/gems/bunny-2.1.0/lib/bunny/channel.rb:344:in `topic' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-dd41a2772297/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb:111:in `block in initialize_queues' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-dd41a2772297/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb:110:in `each' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-dd41a2772297/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb:110:in `initialize_queues' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-dd41a2772297/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb:77:in `start' /var/www/miq/vmdb/app/models/manageiq/providers/openstack/event_catcher_mixin.rb:38:in `monitor_events' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/event_catcher/runner.rb:164:in `block in start_event_monitor' ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From the source code It has been found that queues initialization does not use any settings for durable exchange initialization: https://github.com/ManageIQ/manageiq-gems-pending/blob/fine/lib/gems/pending/openstack/events/openstack_rabbit_event_monitor.rb#L111 But Ruby Bunny have such ability: http://rubybunny.info/articles/durability.html Customer has tested the following and it's work as expected with durable exchanges for nova/neutron etc..: ~~~~~~~~~~~~~~~~~~~~~ def initialize_queues(channel) remove_legacy_queues @queues = {} if @options[:topics] @options[:topics].each do |exchange, topic| #init durable exchange !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! amqp_exchange = channel.topic(exchange, :durable => true) queue_name = "miq-#{@client_ip}-#{exchange}" @queues[exchange] = channel.queue(queue_name, :auto_delete => true, :exclusive => true) .bind(amqp_exchange, :routing_key => topic) end end end ~~~~~~~~~~~~~~~~~~~~~ 4. Why does the customer need this? (List the business requirements here) RabbitMQ durable queues lead to fail of event handler for OpenStack AMQP. 5. How would the customer like to achieve this? (List the functional requirements here) In CFME Advanced settings should be parameter to enable this. 6. For each functional requirement listed in question 5, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented. RedHat itself and on customer's test infrastructure 7. Is there already an existing RFE upstream or in Red Hat bugzilla? No 8. Does the customer have any specific timeline dependencies? As soon as possible. 9. List any affected packages or components. manageiq-gems-pending 11. Would the customer be able to assist in testing this functionality if implemented? Yes.
Sending to PM for prioritization.
https://github.com/ManageIQ/manageiq-providers-openstack/pull/350
New commit detected on ManageIQ/manageiq-providers-openstack/master: https://github.com/ManageIQ/manageiq-providers-openstack/commit/fd6cc79deeda0007af1275f4cd9d1464e0c33cee commit fd6cc79deeda0007af1275f4cd9d1464e0c33cee Author: Scott Seago <sseago> AuthorDate: Mon Sep 10 11:23:51 2018 -0400 Commit: Scott Seago <sseago> CommitDate: Mon Sep 10 11:23:51 2018 -0400 bug 1538058: Retry bunny queue initialization for durable queues https://bugzilla.redhat.com/show_bug.cgi?id=1538058 If rabbitmq is using durable queue/exchanges, they must be opened on the ManageIQ end with durable => true. If exchange creation fails with Bunny::PreconditionFailed, retry with durable=>true. lib/manageiq/providers/openstack/legacy/events/openstack_rabbit_event_monitor.rb | 30 +- 1 file changed, 24 insertions(+), 6 deletions(-)
Danylo, Regarding your first question, there's actually nothing that needs to be done to enable this. If connecting to the queue fails because it is durable, CFME tries again setting the durable flag to 'true'. Regarding the second, I'm not quite sure I understand what you're asking. "use CFME in some way" -- if the durable queue already exists, without this fix, trying to use AMQP messaging with the provider will fail since the attempt to create/connect to a non-durable queue will fail because the queue already exists as a durable one. With this fix, using AMQP will succeed. Perhaps, before testing this fix, make sure that you're setting up the durable queue properly by confirming that using AMQP fails for you if you are using a build without this fix. Then, if that fails, it should succeed with the fix in place.
Verified at the CFME 5.10.0.29
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:0212