Bug 1391686

Summary: Undercloud Upgrade OSP 9->10 fails
Product: Red Hat OpenStack Reporter: Chris Janiszewski <cjanisze>
Component: instack-undercloudAssignee: James Slagle <jslagle>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: cjanisze, david.costakos, mburns, rhel-osp-director-maint, sathlang
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-02 07:55:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sos report from failed upgrade
none
console dump from failed upgrade none

Description Chris Janiszewski 2016-11-03 18:52:43 UTC
Created attachment 1217118 [details]
sos report from failed upgrade

Description of problem:
Undercloud Upgrade OSP 9->10 fails 

Version-Release number of selected component (if applicable):
OSP 9 to OSP 10 Beta


How reproducible:
Deploy OSP 9 and then upgrade to OSP 10 using:
openstack undercloud upgrade
 

Steps to Reproduce:
1. Deploy OSP9
2. Upgrade to OSP10 with -> openstack undercloud upgrade
3.

Actual results:

2016-11-03 09:05:56 - Notice: Finished catalog run in 949.12 seconds
+ rc=6
+ set -e
+ echo 'puppet apply exited with exit code 6'
puppet apply exited with exit code 6
+ '[' 6 '!=' 2 -a 6 '!=' 0 ']'
+ exit 6
[2016-11-03 09:06:00,373] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]

[2016-11-03 09:06:00,373] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1047, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 904, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 508, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-install-undercloud' returned non-zero exit status 1


Expected results:
Successful upgrade 


Additional info:
Please see attached logs from the system. It looks OpenStack-nova-compute and nova-scheduler had a hard time starting up, but they are being up now after the error

Comment 1 Chris Janiszewski 2016-11-03 18:54:18 UTC
Created attachment 1217119 [details]
console dump from failed upgrade

Comment 2 Chris Janiszewski 2016-11-03 19:04:42 UTC
I just re-ran the 'openstack undercloud upgrade', without making any changes and the upgrade finished successfully - it looks like there is a timing issue with how long we wait for services to come back online ?

Comment 3 Sofer Athlan-Guyot 2016-12-02 07:55:32 UTC
Hi,

this is a known behaviour.  The current way to upgrade the undercloud is to run

sudo systemctl stop 'openstack-*'
sudo systemctl stop 'neutron-*'

before the undercloud upgrade.  Services are restarted during the upgrade.

I'm marking this bug as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1372040.  Check this one for details.

*** This bug has been marked as a duplicate of bug 1372040 ***