Bug 1188704

Summary:	Overloaded machine hits the timeout in Puppet+Foreman's node.rb script therefore puppet won't start running.
Product:	Red Hat OpenStack	Reporter:	Leonid Natapov <lnatapov>
Component:	rhel-osp-installer	Assignee:	Jiri Stransky <jstransk>
Status:	CLOSED ERRATA	QA Contact:	Leonid Natapov <lnatapov>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.0 (Juno)	CC:	aberezin, ajeain, dmacpher, mburns, rhos-maint, sasha, yeylon
Target Milestone:	z1	Keywords:	ZStream
Target Release:	Installer
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	foreman-installer-1.6.0-0.3.RC1.el7ost	Doc Type:	Bug Fix
Doc Text:	Some machines would hit the timeout limit when attempting to run Puppet. As a result, Puppet would not trigger. This fix increases the timeout and Puppet triggers successfully.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-03-05 18:19:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Leonid Natapov 2015-02-03 14:51:50 UTC

Overloaded machine hits the timeout in Puppet+Foreman's node.rb script therefore puppet won't start running.

Execution of '/etc/puppet/node.rb returnes 1 because of time out.
Therefore puppet won't start run automatically on the controller which will lead HA deployment to fail.

There's a default timeout of 10 seconds, which was quite probably hit on overloaded machine. Jiri was able to reproduce this bug by manually setting timeout to 1 second locally.


Workaround:

Start puppet manually.

Comment 3 Jiri Stransky 2015-02-03 14:59:46 UTC

A workaround which can prevent this from happening is to edit /etc/puppet/foreman.yaml and change the timeout from 10 to a higher number, e.g. 45 (the unit is seconds).

Fix will need to happen upstream in puppet-foreman to make the timeout value configurable from the installer, and then in rhel-osp-installer we'll set a higher value.

Comment 4 Jiri Stransky 2015-02-10 09:58:39 UTC

3 pull requests submitted, addressing reviews now:

https://github.com/theforeman/foreman-installer-staypuft/pull/134

https://github.com/theforeman/puppet-puppet/pull/236

https://github.com/theforeman/puppet-foreman/pull/289

Comment 5 Jiri Stransky 2015-02-11 09:59:25 UTC

Merged upstream. Foreman has adopted an increased default timeout of 60 seconds, so the first patch is not needed (we'll go with the new defaults), only the second and third patches are necessary.

Comment 7 Mike Burns 2015-02-12 21:47:23 UTC

The actual issue is intermittent, but we can validate that the fix is in by checking /etc/puppet/foreman.yaml and seeing timeout set to 60.

Comment 9 Leonid Natapov 2015-02-18 11:01:57 UTC

tested with foreman-installer-1.6.0-0.3.RC1.el7ost.noarch.
doesn't reproduce.

Comment 11 errata-xmlrpc 2015-03-05 18:19:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0641.html