Bug 1194269

Summary: Deployment will stop on systems which boots/shutdowns quickly and foreman-proxy is terminated before buffer is flushed.
Product: Red Hat OpenStack Reporter: Leonid Natapov <lnatapov>
Component: foreman-discovery-imageAssignee: Mike Burns <mburns>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: aberezin, adahms, lnatapov, mburns, nyechiel, oblaut, rhos-maint, sasha, sputhenp, yeylon
Target Milestone: z2Keywords: ZStream
Target Release: Installer   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/8320
Whiteboard:
Fixed In Version: foreman-proxy-1.6.0.30-6.el7ost foreman-discovery-image-7.0-20150227.0.el7ost Doc Type: Known Issue
Doc Text:
In some rare cases, a host may shut down foreman-proxy before completing the reboot process. This causes hosts to fail to reboot when starting a Red Hat Enterprise Linux OpenStack Platform installer deployment. As a workaround, you can manually trigger the host to reboot and, if necessary, click 'Resume' to resume the deployment, allowing the deployment to proceed without issue.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-04-07 15:08:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Leonid Natapov 2015-02-19 13:00:16 UTC
Description of problem:
On systems which boots/shutdowns quickly, foreman-proxy is terminated before buffer is flushed. Therefore the deployment will stop.

Scenario:
I have 3 controllers and 2 computes ready for deployment. After clicking on "deploy" button 3 controllers were rebooted but 2 computed nodes were not.
This caused deployment to stop with the error. After manually resuming deployment compute nodes went for reboot and deployment successfully finished. 

Workaround:

Resume deployment. 

production.log
---------------------------------------------------------------------------
Completed 200 OK in 585ms (Views: 580.7ms | ActiveRecord: 2.2ms)
Add the TFTP configuration for mac848f69fbc493.example.com
ForemanDiscovery: Rebooting mac848f69fbc493.example.com as its being discovered and assigned
Failed to reboot: undefined method `[]' for true:TrueClass
Add the TFTP configuration for macf04da2732fb1.example.com
ForemanDiscovery: Rebooting macf04da2732fb1.example.com as its being discovered and assigned
Failed to reboot: undefined method `[]' for true:TrueClass
---------------------------------------------------------------------------
Version-Release number of selected component (if applicable):


[root@staypuft foreman]# less production.log 
[root@staypuft foreman]# rpm -qa | grep foreman
foreman-postgresql-1.6.0.49-6.el7ost.noarch
foreman-installer-1.6.0-0.3.RC1.el7ost.noarch
rubygem-foreman_api-0.1.11-6.el7sat.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-foreman-installer-3.0.16-1.el7ost.noarch
ruby193-rubygem-foreman_discovery-1.3.0-2.3.el7ost.noarch
foreman-1.6.0.49-6.el7ost.noarch
foreman-selinux-1.6.0.14-1.el7sat.noarch
foreman-discovery-image-7.0-20140905.0.2.el7sat.noarch
ruby193-rubygem-foreman-tasks-0.6.9-1.1.1.el7ost.noarch
foreman-proxy-1.6.0.30-5.el7ost.noarch

Comment 5 Mike Burns 2015-02-19 13:08:36 UTC
Note:  bug exists in foreman-proxy, but a new foreman-discovery-image will also need to be built to include the package.

Comment 6 Mike Burns 2015-02-27 17:22:45 UTC
Backported the patch in the referenced upstream issue.  Included it in a new foreman-proxy build and rebuilt the foreman-discovery-image with this new foreman-proxy.

Comment 9 Alexander Chuzhoy 2015-03-13 22:21:04 UTC
Verified:

Environment:
rhel-osp-installer-client-0.5.7-1.el7ost.noarch
ruby193-rubygem-foreman_discovery-1.3.0-2.3.el7ost.noarch
foreman-1.6.0.49-6.el7ost.noarch
foreman-postgresql-1.6.0.49-6.el7ost.noarch
foreman-selinux-1.6.0.14-1.el7sat.noarch
openstack-foreman-installer-3.0.17-1.el7ost.noarch
rhel-osp-installer-0.5.7-1.el7ost.noarch
ruby193-rubygem-foreman-tasks-0.6.9-1.1.1.el7ost.noarch
ruby193-rubygem-staypuft-0.5.21-1.el7ost.noarch
foreman-proxy-1.6.0.30-6.el7ost.noarch
rubygem-hammer_cli_foreman-0.1.1-16.el7sat.noarch
openstack-puppet-modules-2014.2.8-2.el7ost.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
foreman-installer-1.6.0-0.3.RC1.el7ost.noarch
foreman-discovery-image-7.0-20150227.0.el7ost.noarch
rubygem-hammer_cli_foreman-doc-0.1.1-16.el7sat.no

The reported issue doesn't reproduce.

Comment 13 errata-xmlrpc 2015-04-07 15:08:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0791.html