Bug 1316968 - Upgrade overcloud 7.3 to stable/liberty fails because of " Unknown resource Type : OS::TripleO::CephStorage::Ports::TenantPort"
Upgrade overcloud 7.3 to stable/liberty fails because of " Unknown resource T...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 8.0 (Liberty)
Assigned To: Marios Andreou
Arik Chernetsky
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-11 09:53 EST by Marios Andreou
Modified: 2016-03-14 07:30 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-14 07:30:10 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Marios Andreou 2016-03-11 09:53:55 EST
Description of problem:
Upgrade overcloud 7.3 to stable/liberty fails because of " Unknown resource Type : OS::TripleO::CephStorage::Ports::TenantPort"

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.original deploy latest 7.3 poodle 3/1/1/1 compute/ceph/swift with net-iso:

openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml --swift-storage-scale 1 --libvirt-type qemu -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml --ntp-server "0.fedora.pool.ntp.org"

2. git clone stable/liberty to a local directory and and try to upgrade the overcloud by running the first step (script deliver/initital commands if any) - this is specified with the 'major-upgrade-pacemaker-init.yaml':

openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/storage-environment.yaml -e tripleo-heat-templates/environments/network-isolation.yaml -e tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml 

This first update 'hangs' (for me at least) on the 'known' unknown heat race; I restarted heat-engine to induce the stack to fail so I don't wait 4 hours. I then re-ran the update, as above:

openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/storage-environment.yaml -e tripleo-heat-templates/environments/network-isolation.yaml -e tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml 

it fails and poking at the failed resources:

[stack@instack ~]$ heat resource-show overcloud CephStorage
| resource_status_reason | resources.CephStorage: resources[0]: Unknown resource Type : OS::TripleO::CephStorage::Ports::TenantPort                                          |

[stack@instack ~]$ heat resource-show overcloud ObjectStorage
| resource_status_reason | resources.ObjectStorage: resources[0]: Unknown resource Type : OS::TripleO::SwiftStorage::Ports::TenantPort                                         |


Actual results:


Expected results:


Additional info:
It looks like it may be because of https://review.openstack.org/#/c/268833/ "Add all isolated networks to all nodes" which very recently landed into stable/liberty. In my testing, after I reverted this patch in my local stable/liberty checkout, those failed resources ^^^ were update_complete (but still failing to complete the update because the controller stubbornly fails with resources.Controller: Exception: resources[0]: Resource update already requested - but I think that is unrelated to the 'unkown resource type' issue being reported here).

FWIW, I reverted the patch https://review.openstack.org/#/c/268833 in my local stable/liberty using

    cd tripleo-heat-templates/ 
    curl https://gist.githubusercontent.com/anonymous/19a0867945e46d020b0c/raw | git am  
    git log #confirm the patch was applied
Comment 2 Marios Andreou 2016-03-14 07:30:10 EDT
I managed to re-run the pacemaker-init step of the upgrade today:

openstack overcloud deploy --templates tripleo-heat-templates -e tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e tripleo-heat-templates/environments/puppet-pacemaker.yaml -e tripleo-heat-templates/environments/storage-environment.yaml -e tripleo-heat-templates/environments/network-isolation.yaml -e tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e network_env.yaml -e tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml 

and the upgrade scripts were delivered successfully to the non-controllers. This time my environment had 2 vcpu/8GB undercloud and I enabled 2 workers as per https://bugzilla.redhat.com/show_bug.cgi?id=1290949#c15  

so the initial fail didn't hang (and completed OK). I guess the root cause of my seeing the error above and the difference from here was that the initial update failed on the other environment (as in the description above) due to underpowered undercloud and then when you try to re-run with stable/liberty heat reports failure due to those resources being unkown as above. I am closing for now we can re-visit if needed.

For clarity, I was able to run the major-upgrade-pacemaker-init.yaml today on a bigger undercloud vm so closing this for now, we don't need to revert anything in stable/liberty.

Note You need to log in before you can comment on or make changes to this bug.