Bug 1263801

Summary: Postdeployment does not complete if overcloud deploy command in background
Product: Red Hat OpenStack Reporter: Dan Sneddon <dsneddon>
Component: python-tripleoclientAssignee: Ben Nemec <bnemec>
Status: CLOSED ERRATA QA Contact: Gurenko Alex <agurenko>
Severity: unspecified Docs Contact:
Priority: low    
Version: 7.0 (Kilo)CC: achernet, augol, bnemec, dmatthew, hbrock, jcoufal, joea, jschluet, jslagle, lbopf, mburns, rhel-osp-director-maint, rybrown
Target Milestone: rcKeywords: TestOnly, Triaged
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: python-tripleoclient-6.1.0-5.el7ost Doc Type: Bug Fix
Doc Text:
Cause: Previous versions of OSP director had issues doing post-deployment steps if the deploy command was run in the background. Consequence: Deployments would hang during post-deployment until the command was brought into the foreground of the session. Fix: Post-deployment was re-designed so that the problematic steps are no longer done. Result: Deployment in the background now works as expected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-17 15:23:56 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Dan Sneddon 2015-09-16 13:57:15 EDT
Description of problem:
If you run openstack overcloud deploy with &> logfile, the process goes into the background. This results in the deployment hanging on one of the last steps in postdeployment, and the endpoints and security settings don't get completed.

Version-Release number of selected component (if applicable):
OSP 7 GA (probably affects Director 7.1 as well)

How reproducible:

Steps to Reproduce:
1. Run the openstack overcloud deploy in the background

Actual results:
The stack gets to CREATE_COMPLETE, but in the postdeployment steps the script will hang.

Expected results:
The postdeployment should complete.

Additional info:
When the script was in this state, ps showed it as "stopped". When we foregrounded the process, the postdeployment immediately completed and the endpoints and security for the admin user was set up. This *might* have something to do with an SSH call requiring a TTY, but definitely only works when the process is in the foreground. This causes a lot of confusion, since the stack goes to CREATE_COMPLETE but the overcloud is not functional.

A simple workaround is just to make sure that the overcloud deploy process happens in the foreground. The output can be piped to tee to write it to a log file while keeping it in the foreground.
Comment 3 Ryan Brown 2015-09-18 13:10:25 EDT
Hi, I've put together a patchset to resolve this upstream - it's somewhat large so I'd want to put together a less invasive set of patches if we want this to make it into y2. 

Comment 4 Ben Nemec 2015-12-17 10:44:36 EST
I'm aware of this happening in at least a couple of instances, but I also have a report that someone was able to deploy successfully with the process in the background, so it appears this isn't 100% reproducible.  Still unclear what the cause is.
Comment 5 Mike Burns 2016-04-07 16:50:54 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 7 Ben Nemec 2016-10-06 09:50:30 EDT
Note that this may be fixed in OSP 10, since we don't do post-deployment over ssh anymore.
Comment 9 Emilien Macchi 2017-04-20 16:26:38 EDT
*** Bug 1329343 has been marked as a duplicate of this bug. ***
Comment 11 Amit Ugol 2017-04-24 02:08:58 EDT
I couldn't reproduce it for a while and personally I always deploy to the background.
Comment 14 errata-xmlrpc 2017-05-17 15:23:56 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

Comment 15 Ben Nemec 2017-06-16 17:08:31 EDT
This did not show up in my doc-text query so I missed it.  Adding it now in case it's still useful.