Bug 1267318 - [Docs] [Director] Overcloud Updates failed due new resource type
[Docs] [Director] Overcloud Updates failed due new resource type
Status: CLOSED DUPLICATE of bug 1286798
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
urgent Severity urgent
: async
: 7.0 (Kilo)
Assigned To: Dan Macpherson
: Automation, Documentation
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-29 11:45 EDT by mathieu bultel
Modified: 2015-12-17 20:28 EST (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-17 20:28:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 240792 None None None Never

  None (edit)
Description mathieu bultel 2015-09-29 11:45:33 EDT
Description of problem:

100% reproducible, when trying to upgrade GA to the latest puddle, after the upgrade of the undercloud, the update of the overcloud failed with :

openstack overcloud update stack -i overcloud --template /usr/share/openstack-tripleo-heat-templates
starting package update on stack overcloud
ERROR: openstack ERROR: Unknown resource Type : OS::TripleO::Network::Ports::StorageVipPort
Comment 2 Zane Bitter 2015-09-29 11:58:49 EDT
The problem is that by upgrading the undercloud, we have also updated the tripleo-heat-templates. This requires changes to the environment files (e.g. to support new resource types, as we see here). The "openstack overcloud update" command, however, is designed to not pass any environment files but just retain the existing environment in Heat. This does not work if the templates have been modified.

A solution may be to simply require the user to do an update explicitly passing all of the environment files again (including explicitly specifying the default environment files) after an undercloud update. We may be able to improve on this by adding a command-line option to include the default env files so that the user doesn't have to figure out the correct paths to them. (Even better would be to have a confirmation step when this option is specified, to make sure that users remember to include all of their extra environment files too.)

Assigning to the rhel-osp-director component for now. This may end up as a docs-only bug, or it may require changes to python-rdomanager-oscplugin.
Comment 3 Steven Hardy 2015-09-30 03:57:35 EDT
So I chatted to Jan about this, wrt how we might validate this before attempting the stack update.

Unfortunately, despite us having a preview_update_stack interface which could probably validate the PATCH update, I missed updating that interface recently when I implemented PATCH updates for update_stack proper.

I raised this upstream bug to track it, they should behave consistently, and we may be able to backport the fix to preview_update_stack as AFAICT it's only a refactoring change inside service.py.

https://bugs.launchpad.net/heat/+bug/1501207
Comment 4 Jan Provaznik 2015-09-30 07:46:39 EDT
Because the same situation (old environment with new template) may happen for all CLI commands (scaling down, pkg updates, re-deploy) it seems to me that it's best to instruct users to update the existing overcloud with new environment files right after undercloud machine upgrade.

A doc patch with this instruction is here:
https://review.openstack.org/229373

An alternative to adding an extra parameter for adding default env file might be pre-update validation mentioned by Steven. If backporting of patch for preview_update_stack might happen anytime soon, I would lean to add a check to CLI commands which calls preview_update_stack before running stack-update and if this preview would fail, we could warn user.

Upstream bug for this issue:
https://bugs.launchpad.net/tripleo/+bug/1501296
Comment 5 Zane Bitter 2015-09-30 11:44:42 EDT
(In reply to Jan Provaznik from comment #4)
> Because the same situation (old environment with new template) may happen
> for all CLI commands (scaling down, pkg updates, re-deploy) it seems to me
> that it's best to instruct users to update the existing overcloud with new
> environment files right after undercloud machine upgrade.

I'm inclined to agree. My only reservation is that this could result in the user ending up with new templates but old packages until they then go and run a package update. Off the top of my head I can't think of any circumstances where that would cause a problem though.

> A doc patch with this instruction is here:
> https://review.openstack.org/229373

LGTM
Comment 6 chris alfonso 2015-09-30 12:11:18 EDT
Dan, please check the docs patch and let us know if you need more info for the product doc change.
Comment 7 Andrew Dahms 2015-10-07 21:00:15 EDT
Assigning to Dan for review.
Comment 9 Dan Macpherson 2015-10-14 22:00:50 EDT
Added Overcloud stack upgrade procedure:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Updating_Overcloud_Stack.html

Zane, Steve, Jan - How does this look? Any further changes required?
Comment 10 Jan Provaznik 2015-10-15 07:25:05 EDT
Thanks Dan. Unfortunately it seems that this fix is not sufficient because Zane's concern from comment 5 is probably already happening. Step https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Updating_Overcloud_Stack.html

failed for Gael Lambert because node resources failed with error:
Error: Could not find class tripleo::packages for strg00-prv.localdomain on node strg00-prv.localdomain

tripleo::packages puppet class is defined in openstack-puppet-modules-2015.1.8-19.el7ost.noarch but I guess it was not yet in 7.0 rpms.


So a potential fix would run directly package update and pass it explicitly all the env files again (comment 2).
Comment 11 Mike Burns 2015-10-15 20:14:39 EDT
I think the solution here is pretty simple, actually.  This error (tripleo::packages not defined) is because of an old openstack-puppet-modules on the deployed machine.  This can be solved in 2 ways:

1.  user manually runs yum update openstack-puppet-modules on every machine
2.  we change update stack to do update of openstack-puppet-modules first, then proceed with the rest.  

#1 can be a good short term solution

#2 is likely the right long term solution.  OPM is a safe update in general, as far as openstack services go.  It's also going to hit us that THT requires something that is not in the old OPM at some point in the future, especially on major version upgrades.
Comment 12 Jan Provaznik 2015-10-16 03:05:23 EDT
We are testing if running:
openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml

Basically skipping https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Updating_Overcloud_Stack.html and doing directly https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Updating_Overcloud_Packages.html with explicitly setting default environment file.

If this works it should be just matter of updating doc.
Comment 13 Cyril Lopez 2015-10-16 03:51:14 EDT
(In reply to Mike Burns from comment #11)
> I think the solution here is pretty simple, actually.  This error
> (tripleo::packages not defined) is because of an old
> openstack-puppet-modules on the deployed machine.  This can be solved in 2
> ways:
> 
> 1.  user manually runs yum update openstack-puppet-modules on every machine
> 2.  we change update stack to do update of openstack-puppet-modules first,
> then proceed with the rest.  
> 
> #1 can be a good short term solution
> 
> #2 is likely the right long term solution.  OPM is a safe update in general,
> as far as openstack services go.  It's also going to hit us that THT
> requires something that is not in the old OPM at some point in the future,
> especially on major version upgrades.

That's what we did and it's solve to do a openstack deploy but we have a other issue cf https://bugzilla.redhat.com/show_bug.cgi?id=1272347
Comment 14 Jan Provaznik 2015-10-19 09:28:40 EDT
Unfortunately a fix for this can't be tested properly until https://bugzilla.redhat.com/show_bug.cgi?id=1272357 is fixed (this one makes update fail always).
Comment 15 Jan Provaznik 2015-10-21 03:31:54 EDT
So far it seems that running directly "openstack overcloud update stack" (comment 12) solves the issue with missing tripleo::packages class on OC nodes - openstack-puppet-modules is updated by yum update before puppet runs (at least for 7.0->7.1 upgrade). So no pre-patching of OC nodes should be required for this BZ (only doc update).

Update process is still failing but in much later phase (probably related to 1272347), I'll send a doc patch for this BZ once update finishes successfully.
Comment 16 Jan Provaznik 2015-11-02 04:03:05 EST
I can confirm that running package update directly solves this particular issue for 7.0->7.1 upgrades.

An upstream doc patch:
https://review.openstack.org/240792
Comment 17 Dan Macpherson 2015-12-15 23:50:42 EST
I think this BZ is obsolete due to this BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1286798

Can anyone confirm this?
Comment 18 Giulio Fidente 2015-12-17 05:37:17 EST
hi Dan, yes the update process tracked by #1286798 covers this particular issue too.
Comment 19 Dan Macpherson 2015-12-17 20:28:58 EST
In that case, I'll close this BZ since the other bug is more relevant. However, if we need this BZ open for whatever reason, please feel free to reopen.

*** This bug has been marked as a duplicate of bug 1286798 ***

Note You need to log in before you can comment on or make changes to this bug.