Bug 1443663 – rhosp-director: Overcloud upgrade osp10->osp11 fails on a setup upgraded from OSP9: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'oslo::messaging::default'

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1443663 - rhosp-director: Overcloud upgrade osp10->osp11 fails on a setup upgraded from OSP9: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'oslo::messaging::default'

Summary:	rhosp-director: Overcloud upgrade osp10->osp11 fails on a setup upgraded fro...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-heat-templates (Show other bugs)
Sub Component:
Version:	11.0 (Ocata)
Hardware:	Unspecified Unspecified

Priority	urgent Severity unspecified
Target Milestone:	rc
Target Release:	11.0 (Ocata)
Assigned To:	Sofer Athlan-Guyot
QA Contact:	Marius Cornea
Docs Contact:

URL:
Whiteboard:
Keywords:	Triaged

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2017-04-19 12:24 EDT by Alexander Chuzhoy
Modified:	2017-05-17 16:23 EDT (History)
CC List:	9 users (show)

See Also:
Fixed In Version:	openstack-tripleo-heat-templates-6.0.0-7.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2017-05-17 16:23:01 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Launchpad	1684587	None	None	None	2017-04-20 06:43 EDT
OpenStack gerrit	458530	None	None	None	2017-04-20 12:44 EDT
Red Hat Product Errata	RHEA-2017:1245	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory	2017-05-17 19:01:50 EDT

Groups: None (edit)

Description Alexander Chuzhoy 2017-04-19 12:24:04 EDT

rhosp-director:  Overcloud upgrade osp10->osp11 fails on a setup upgraded from OSP9: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'oslo::messaging::default'


Environment:
openstack-tripleo-heat-templates-6.0.0-5.el7ost.noarch
instack-undercloud-6.0.0-6.el7ost.noarch
openstack-tripleo-heat-templates-compat-5.2.0-7.el7ost.noarch
openstack-puppet-modules-10.0.0-1.el7ost.noarch
python-oslo-messaging-5.17.1-2.el7ost.noarch

Steps to reproduce:
1) Deploy OSP9 with:
 Deployment command: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1   --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server 10.5.26.10 --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml


2) Upgrade the setup to OSP10

3) Attempt to upgrade to OSP11

Result:
The upgrade fails during overcloud upgrade, in step with " -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml -e init-repo.yaml"

[stack@undercloud72 ~]$ openstack stack failures list overcloud                                                                                                                               overcloud.AllNodesDeploySteps.ControllerUpgrade_Step5.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: d9f82961-e758-4f7e-903f-afa23e59c0ac
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
    TASK [Create puppet manifest to set transport_url in nova.conf] ****************
    changed: [localhost]

    TASK [Run puppet apply to set tranport_url in nova.conf] ***********************
    fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["puppet", "apply", "--detailed-exitcodes", "/root/nova-api_upgrade_manifest.pp"], "delta": "0:00:08.325284", "end": "2017-04-19 16:12:47.475946", "failed": true, "failed_when_result": true, "rc": 1, "start": "2017-04-19 16:12:39.150662", "stderr": "exception: connect failed\n\u001b[1;33mWarning: Undefined variable '::deploy_config_name'; \n   (file & line not available)\u001b[0m\n\u001b[1;31mError: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'oslo::messaging::default' at /root/nova-api_upgrade_manifest.pp:8:4 on node overcloud-controller-0.localdomain\u001b[0m", "stdout": "", "stdout_lines": [], "warnings": []}
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/0a1bf20b-93eb-41fc-87c5-57414479d07b_playbook.retry

    PLAY RECAP *********************************************************************
    localhost                  : ok=19   changed=16   unreachable=0    failed=1

    (truncated, view all with --long)
  deploy_stderr: |

(reverse-i-search)`re': cat init-^Cpo.yaml    





[stack@undercloud72 ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------------+
| resource_name                               | physical_resource_id                                                                                                     | resource_type                                                                                                         | resource_status | updated_time         | stack_name                                                                                                                   |
+---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------------+
| AllNodesDeploySteps                         | 8d14e21e-6c96-49dd-ae70-98b4da4afff4                                                                                     | OS::TripleO::PostDeploySteps                                                                                          | CREATE_FAILED   | 2017-04-19T15:36:40Z | overcloud                                                                                                                    |
| ControllerUpgrade_Step5                     | 0dbec2c0-43a8-4a36-996a-9cd588c72549                                                                                     | OS::Heat::SoftwareDeploymentGroup                                                                                     | CREATE_FAILED   | 2017-04-19T15:36:42Z | overcloud-AllNodesDeploySteps-j5zdjt3ghxuu                                                                                   |
| 0                                           | d9f82961-e758-4f7e-903f-afa23e59c0ac                                                                                     | OS::Heat::SoftwareDeployment                                                                                          | CREATE_FAILED   | 2017-04-19T16:11:57Z | overcloud-AllNodesDeploySteps-j5zdjt3ghxuu-ControllerUpgrade_Step5-sw275xztnlvh                                              |
+---------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------------+

Comment 1 Marius Cornea 2017-04-19 12:33:04 EDT

Looks like there's no /etc/puppet/modules/oslo/ directory on the node where it failed:

[root@overcloud-controller-0 ~]# ls -l /etc/puppet/modules/oslo/
ls: cannot access /etc/puppet/modules/oslo/: No such file or directory

Comment 3 Sofer Athlan-Guyot 2017-04-20 12:44:20 EDT

In review in stable/ocata.

Comment 4 Sofer Athlan-Guyot 2017-04-21 10:18:37 EDT

Merged in stable/ocata.

Comment 9 errata-xmlrpc 2017-05-17 16:23:01 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245

Note You need to log in before you can comment on or make changes to this bug.