Bug 1371316
| Summary: | Osp-director-10: Overcloud Upgrade 9 -> 10 fails during the init stage command. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Omri Hochman <ohochman> |
| Component: | openstack-heat | Assignee: | Zane Bitter <zbitter> |
| Status: | CLOSED ERRATA | QA Contact: | Amit Ugol <augol> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | dbecker, jcoufal, mandreou, mburns, mcornea, morazi, rhel-osp-director-maint, sathlang, sbaker, shardy, srevivo |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 10.0 (Newton) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-heat-7.0.0-2.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-12-14 15:54:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1337794 | ||
|
Description
Omri Hochman
2016-08-29 22:06:08 UTC
I've change the init stage command according the changes in : https://gitlab.cee.redhat.com/sathlang/ospd-9-to-10-upgrade#controller-and-block-storage-upgrade to: #!/usr/bin/bash . stackrc cat > overcloud-repos.yaml <<EOF parameter_defaults: UpgradeInitCommand: | set -e yum localinstall -y http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm rhos-release -P 10 -d # Workaround for bz-1361148 ! [ -e /usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.d ] || rm /usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.d EOF $DEPLOY -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml \ -e /home/stack/overcloud-repos.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-overcloud-compute-hostnames.yaml results : ---------- Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 424, in wrapped 2016-08-24 11:11:13 [UpdateDeployment]: UPDATE_FAILED NotFound_Remote: resources.UpdateDeployment: Software config with id 5a80c923-abaa-44a1-8c16-679acb1b8b49 not found Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 424, in wrapped return func( 2016-08-24 11:11:14 [StorageMgmtPort]: UPDATE_FAILED UPDATE aborted 2016-08-24 11:11:14 [ExternalPort]: UPDATE_FAILED UPDATE aborted 2016-08-24 11:11:15 [Controller]: UPDATE_FAILED resources.Controller: resources[0]: NotFound_Remote: resources.UpdateDeployment: Software config with id efdb9688-0279-4173-854f-c2be1c83fe3e not found Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/heat/common/context.py", l 2016-08-24 11:11:15 [TenantPort]: UPDATE_FAILED UPDATE aborted 2016-08-24 11:11:15 [Compute]: UPDATE_FAILED UPDATE aborted 2016-08-24 11:11:15 [overcloud]: UPDATE_FAILED resources.Controller: resources[0]: NotFound_Remote: resources.UpdateDeployment: Software config with id efdb9688-0279-4173-854f-c2be1c83fe3e not found Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/heat/common/context.py", l 2016-08-24 11:11:15 [0]: UPDATE_FAILED UPDATE aborted 2016-08-24 11:11:16 [overcloud-Compute-uydvzmzehkxk]: UPDATE_FAILED Operation cancelled 2016-08-24 11:11:16 [ManagementPort]: UPDATE_FAILED UPDATE aborted 2016-08-24 11:11:17 [InternalApiPort]: UPDATE_FAILED UPDATE aborted 2016-08-24 11:11:17 [overcloud-Controller-mrkekqec3nea-2-37cg6jdefkv6]: UPDATE_FAILED NotFound_Remote: resources.UpdateDeployment: Software config with id 5a80c923-abaa-44a1-8c16-679acb1b8b49 not found Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 424, in wrapped return func( Stack overcloud UPDATE_FAILED Heat Stack update failed. Hi,
looks like the overcloud lost connectivity with undercloud:
[UpdateDeployment]: UPDATE_FAILED NotFound_Remote: resources.UpdateDeployment: Software config with id 5a80c923-abaa-44a1-8c16-679acb1b8b49 not found
Let's see if we can reproduce this one, as we had the systemctl timeout issue and finished the undercloud upgrade manually.
Oki, a upstream bug made its way into the latest puddle. Here are the related upstream bug and fix. This would be caused by too much nested stacks in tripleo which bring up corner case in heat which are not guaranty to work. The full description in launchpad. Adding the first required review. Verified a temp workaround (added to the Git doc) :
curl -o software_deployment.py \
https://git.openstack.org/cgit/openstack/heat/plain/heat/engine/resources/openstack/heat/software_deployment.py?id=8fcebfae3c2a9e86bffb8a66f8bc84fbf4237d22
sudo cp software_deployment.py \
/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/software_deployment.py
sudo systemctl restart openstack-heat-engine.service
moving this to POST as the related changes linked above have merged upstream for a while now - as omri posted with comment #6 that fix worked to overcome the issue reported here Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html |