Bug 1316968

Summary:	Upgrade overcloud 7.3 to stable/liberty fails because of " Unknown resource Type : OS::TripleO::CephStorage::Ports::TenantPort"
Product:	Red Hat OpenStack	Reporter:	Marios Andreou <mandreou>
Component:	openstack-tripleo-heat-templates	Assignee:	Marios Andreou <mandreou>
Status:	CLOSED NOTABUG	QA Contact:	Arik Chernetsky <achernet>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.0 (Liberty)	CC:	mburns, rhel-osp-director-maint
Target Milestone:	---
Target Release:	8.0 (Liberty)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-03-14 11:30:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Marios Andreou 2016-03-11 14:53:55 UTC

Description of problem:
Upgrade overcloud 7.3 to stable/liberty fails because of " Unknown resource Type : OS::TripleO::CephStorage::Ports::TenantPort"

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.original deploy latest 7.3 poodle 3/1/1/1 compute/ceph/swift with net-iso:

openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml --swift-storage-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server "0.fedora.pool.ntp.org"

2. git clone stable/liberty to a local directory and and try to upgrade the overcloud by running the first step (script deliver/initital commands if any) - this is specified with the 'major-upgrade-pacemaker-init.yaml':

openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/storage-environment.yaml -e tripleo-heat-templates/environments/network-isolation.yaml -e tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml 

This first update 'hangs' (for me at least) on the 'known' unknown heat race; I restarted heat-engine to induce the stack to fail so I don't wait 4 hours. I then re-ran the update, as above:

openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/storage-environment.yaml -e tripleo-heat-templates/environments/network-isolation.yaml -e tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml 

it fails and poking at the failed resources:

[stack@instack ~]$ heat resource-show overcloud CephStorage
| resource_status_reason | resources.CephStorage: resources[0]: Unknown resource Type : OS::TripleO::CephStorage::Ports::TenantPort                                          |

[stack@instack ~]$ heat resource-show overcloud ObjectStorage
| resource_status_reason | resources.ObjectStorage: resources[0]: Unknown resource Type : OS::TripleO::SwiftStorage::Ports::TenantPort                                         |


Actual results:


Expected results:


Additional info:
It looks like it may be because of https://review.openstack.org/#/c/268833/ "Add all isolated networks to all nodes" which very recently landed into stable/liberty. In my testing, after I reverted this patch in my local stable/liberty checkout, those failed resources ^^^ were update_complete (but still failing to complete the update because the controller stubbornly fails with resources.Controller: Exception: resources[0]: Resource update already requested - but I think that is unrelated to the 'unkown resource type' issue being reported here).

FWIW, I reverted the patch https://review.openstack.org/#/c/268833 in my local stable/liberty using

    cd tripleo-heat-templates/ 
    curl https://gist.githubusercontent.com/anonymous/19a0867945e46d020b0c/raw | git am  
    git log #confirm the patch was applied

Comment 2 Marios Andreou 2016-03-14 11:30:10 UTC

I managed to re-run the pacemaker-init step of the upgrade today:

openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/storage-environment.yaml -e tripleo-heat-templates/environments/network-isolation.yaml -e tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml 

and the upgrade scripts were delivered successfully to the non-controllers. This time my environment had 2 vcpu/8GB undercloud and I enabled 2 workers as per https://bugzilla.redhat.com/show_bug.cgi?id=1290949#c15  

so the initial fail didn't hang (and completed OK). I guess the root cause of my seeing the error above and the difference from here was that the initial update failed on the other environment (as in the description above) due to underpowered undercloud and then when you try to re-run with stable/liberty heat reports failure due to those resources being unkown as above. I am closing for now we can re-visit if needed.

For clarity, I was able to run the major-upgrade-pacemaker-init.yaml today on a bigger undercloud vm so closing this for now, we don't need to revert anything in stable/liberty.