Bug 1390284

Summary:	Swift proxy blocks when RabbitMQ restarted in a three-node pacemaker HA setup
Product:	Red Hat OpenStack	Reporter:	Christian Schwede (cschwede) <cschwede>
Component:	puppet-tripleo	Assignee:	Christian Schwede (cschwede) <cschwede>
Status:	CLOSED DUPLICATE	QA Contact:	nlevinki <nlevinki>
Severity:	high	Docs Contact:
Priority:	high
Version:	10.0 (Newton)	CC:	egafford, emacchi, jjoyce, jschluet, pgrist, slinaber, thiago, tvignaud, zaitcev
Target Milestone:	---
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-10-31 16:00:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Christian Schwede (cschwede) 2016-10-31 15:46:33 UTC

Description of problem:

After restarting the RabbitMQ cluster using pacemaker on the OOO controllers, Swift will block and log errors about failing connections to RabbitMQ. This doesn't happen on a single-node RabbitMQ instance; it seems to be related to the fact that the ceilometermiddleware is configured to use only the local RabbitMQ node.

Related upstream bugreport: https://bugs.launchpad.net/tripleo/+bug/1637443

Proposed upstream fix: https://review.openstack.org/#/c/391890/

Version-Release number of selected component (if applicable):


How reproducible:

Always on a HA OOO deployment using three controllers and pacemaker.

Steps to Reproduce:
1. pcs resource disable rabbitmq
2. <wait a minute or so>
3. <try to use Swift>
4. pcs resource enable rabbitmq
5. <try to use Swift again, for example "swift stat">

Actual results:

Blocking Swift operations and log entries like this:

Oct 31 10:04:00 host-192-0-2-15 proxy-server: STDERR: ERROR:oslo.messaging._drivers.impl_rabbit:[275d772c-b7d4-4010-b07e-01dd10c3b1a4] AMQP server on 172.17.0.16:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: 43564 (txn: txa0bd3ae6c911476aa0698-0058174f25) (client_ip: 172.18.0.18)

Expected results:

Swift being able to continue operations after RabbitMQ recovers.

Additional info:

Comment 1 Emilien Macchi 2016-10-31 16:00:35 UTC


*** This bug has been marked as a duplicate of bug 1389999 ***