Description of problem:

After restarting the RabbitMQ cluster using pacemaker on the OOO controllers, Swift will block and log errors about failing connections to RabbitMQ. This doesn't happen on a single-node RabbitMQ instance; it seems to be related to the fact that the ceilometermiddleware is configured to use only the local RabbitMQ node.

Related upstream bugreport: https://bugs.launchpad.net/tripleo/+bug/1637443

Proposed upstream fix: https://review.openstack.org/#/c/391890/

Version-Release number of selected component (if applicable):


How reproducible:

Always on a HA OOO deployment using three controllers and pacemaker.

Steps to Reproduce:
1. pcs resource disable rabbitmq
2. <wait a minute or so>
3. <try to use Swift>
4. pcs resource enable rabbitmq
5. <try to use Swift again, for example "swift stat">

Actual results:

Blocking Swift operations and log entries like this:

Oct 31 10:04:00 host-192-0-2-15 proxy-server: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[275d772c-b7d4-4010-b07e-01dd10c3b1a4] AMQP server on 172.17.0.16:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: 43564 (txn: txa0bd3ae6c911476aa0698-0058174f25) (client_ip: 172.18.0.18)

Expected results:

Swift being able to continue operations after RabbitMQ recovers.

Additional info: