Bug 1391730

Summary: overcloud upgrade failed OSP 9->10 - no dhcp and rebuilding
Product: Red Hat OpenStack Reporter: Chris Janiszewski <cjanisze>
Component: instack-undercloudAssignee: James Slagle <jslagle>
Status: CLOSED WORKSFORME QA Contact: Arik Chernetsky <achernet>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: aschultz, cjanisze, mburns, rhel-osp-director-maint, sathlang
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-10 09:24:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
sosreport, templates, screenshots and console dump none

Description Chris Janiszewski 2016-11-03 20:50:29 UTC 
Created attachment 1217152 [details]
sosreport, templates, screenshots and console dump

Description of problem:
overcloud upgrade failed OSP 9->10 
after updating images in glance and trying to run overcloud deploy, the overcloud seems to be rebuilding instead of upgrading but there is no dhcp responses serving pxe boot. Please see logs for details

Version-Release number of selected component (if applicable):
OSP 9 -> 10

How reproducible:
upgrade overcloud from 9 -> 10

Steps to Reproduce:
Followed steps from https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/paged/upgrading-red-hat-openstack-platform/chapter-3-director-based-environments-performing-upgrades-to-major-versions 
with obvious changes that would cover OSP 10
1. uploaded and updated images in glance for osp 10
2. updated a templates to osp 10
3. re-deployed overcloud with the same parameters 

Actual results:
2016-11-03 15:13:55 [NovaCompute]: UPDATE_FAILED Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:55 [CephStorage]: UPDATE_FAILED Error: resources.CephStorage: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:55 [overcloud-Compute-552rctjcui55-2-tperv57mqzpj]: UPDATE_FAILED Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:55 [overcloud-CephStorage-mz7xfjsphasa-2-suk44lwoq24f]: UPDATE_FAILED Error: resources.CephStorage: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:55 [CephStorage]: UPDATE_FAILED Error: resources.CephStorage: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:55 [NovaCompute]: UPDATE_FAILED Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:55 [overcloud-CephStorage-mz7xfjsphasa-0-z53tnuwmpobh]: UPDATE_FAILED Error: resources.CephStorage: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:55 [overcloud-Compute-552rctjcui55-1-joohaycslt2j]: UPDATE_FAILED Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:55 [CephStorage]: UPDATE_FAILED Error: resources.CephStorage: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:55 [overcloud-CephStorage-mz7xfjsphasa-1-jcpehymtncr7]: UPDATE_FAILED Error: resources.CephStorage: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:56 [0]: UPDATE_FAILED resources[0]: Error: resources.CephStorage: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:56 [NovaCompute]: UPDATE_FAILED Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:56 [1]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:13:56 [overcloud-Compute-552rctjcui55-0-ee737ackqzoo]: UPDATE_FAILED Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:56 [2]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:13:56 [overcloud-CephStorage-mz7xfjsphasa]: UPDATE_FAILED resources[0]: Error: resources.CephStorage: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:56 [0]: UPDATE_FAILED resources[0]: Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:56 [2]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:13:56 [1]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:13:57 [overcloud-Compute-552rctjcui55]: UPDATE_FAILED resources[0]: Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:57 [Compute]: UPDATE_FAILED resources.Compute: resources[0]: Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:58 [Controller]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:13:58 [1]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:13:58 [CephStorage]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:13:58 [overcloud]: UPDATE_FAILED resources.Compute: resources[0]: Error: resources.NovaCompute: Rebuilding server failed, status 'ERROR'
2016-11-03 15:13:58 [2]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:13:59 [0]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:13:59 [overcloud-Controller-234d35efwqoe]: UPDATE_FAILED Operation cancelled
2016-11-03 15:14:00 [Controller]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:14:00 [overcloud-Controller-234d35efwqoe-0-dtgz2hw6ccso]: UPDATE_FAILED Operation cancelled
2016-11-03 15:14:00 [Controller]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:14:00 [Controller]: UPDATE_FAILED UPDATE aborted
2016-11-03 15:14:00 [overcloud-Controller-234d35efwqoe-2-ovpd5hgpljow]: UPDATE_FAILED Operation cancelled
2016-11-03 15:14:00 [overcloud-Controller-234d35efwqoe-1-2eum5ftnt3q3]: UPDATE_FAILED Operation cancelled
Stack overcloud UPDATE_FAILED
Heat Stack update failed.
./deploy.sh: line 6: --ntp-server: command not found


Expected results:
overcloud upgraded successfully 

Additional info:
Attaching sosreport, templates, screenshots and console dump
Comment 1 Sofer Athlan-Guyot 2017-02-10 09:24:25 UTC 
Hi Chris,

I'm closing this one as old.  If it's still an issue don't hesitate to re-open the bz.

Regards,