Bug 1123318

Summary:	HA: All services are missing max_retries=-1 settings for accessing the DB.
Product:	Red Hat OpenStack	Reporter:	Leonid Natapov <lnatapov>
Component:	openstack-foreman-installer	Assignee:	Crag Wolfe <cwolfe>
Status:	CLOSED ERRATA	QA Contact:	Leonid Natapov <lnatapov>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.0 (RHEL 7)	CC:	cwolfe, dnavale, jguiditt, mburns, morazi, rhos-maint, yeylon
Target Milestone:	ga
Target Release:	Installer
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-foreman-installer-2.0.18-1.el6ost	Doc Type:	Bug Fix
Doc Text:	Previously, the max_retries parameter in various configuration files was not set to -1. As a result, an OpenStack service would 'give up' rather than retry if the initial database connection failed. With this update, the value for max_retries parameter is set to -1 in the OpenStack configuration files, so services will retry connecting to the database service if the initial attempt fails, allowing for a more robust setup.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-08-21 18:06:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Leonid Natapov 2014-07-25 10:03:17 UTC

Rubygem-staypuft: HA: All services are missing max_retries=-1 settings for accessing the DB.

This means that if the DB is not started before the service, the service will fail to start. This behaviour is currently mitigated by: start-failure-is-fatal: false, but it is technically incorrect.

solution1:
set max_retries=-1 on all services with access to the DB

solution2:
add a start order constrain for services on galera/mysql. This will guarantee that the services will be always started _after_ the db. In the manual how-to I use solution1 and it is known to work.
Also a maintanance to the DB will triger a chain effect of stopping all services, while that's not required with solution1.

Comment 1 Crag Wolfe 2014-08-04 21:15:17 UTC

I'm going with Solution 1 unless somebody objects.

Comment 3 Jason Guiditta 2014-08-05 18:57:42 UTC

Merged:
https://github.com/redhat-openstack/astapor/pull/339

Comment 6 Leonid Natapov 2014-08-12 13:17:24 UTC

[root@maca25400702875 ~]# grep -P '^max_retries' /etc/*/*conf
/etc/cinder/cinder.conf:max_retries=-1
/etc/glance/glance-api.conf:max_retries=-1
/etc/glance/glance-registry.conf:max_retries=-1
/etc/heat/heat.conf:max_retries=-1
grep: /etc/httpd/conf: Is a directory
/etc/keystone/keystone.conf:max_retries=-1
/etc/nova/nova.conf:max_retries=-1


rhel-osp-installer-0.1.9-1.el6ost.noarch

Comment 7 errata-xmlrpc 2014-08-21 18:06:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1090.html