Bug 1314429 - New cases of not ignoring updates to OS::Nova::Server causing redeployment
New cases of not ignoring updates to OS::Nova::Server causing redeployment
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
high Severity unspecified
: ga
: 8.0 (Liberty)
Assigned To: Steve Baker
Alexander Chuzhoy
: TestOnly
Depends On: 1303094
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-03 10:06 EST by Jiri Stransky
Modified: 2016-10-13 14:23 EDT (History)
14 users (show)

See Also:
Fixed In Version: openstack-tripleo-common-0.3.0-3.el7ost
Doc Type: Bug Fix
Doc Text:
The behavior of the OS::Nova::Server resource is to replace servers whenever certain documented properties change during a stack update. This could cause possible unintentional replacement of Overcloud nodes if properties change during a major upgrade. This fix makes the Undercloud's Heat service never replace server resources when properties change.
Story Points: ---
Clone Of: 1303094
Environment:
Last Closed: 2016-04-18 12:37:44 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1539541 None None None 2016-03-04 10:23 EST
OpenStack gerrit 288273 None None None 2016-03-04 02:27 EST
OpenStack gerrit 291771 None None None 2016-03-21 12:09 EDT

  None (edit)
Description Jiri Stransky 2016-03-03 10:06:24 EST
+++ This bug was initially created as a clone of Bug #1303094 +++

(cut out the long clone text, please look at the original bug if needed)

There are new cases of property changes causing unwanted node replacement on upgrade Liberty.

https://review.openstack.org/#/c/286739/ -- this patch could cause node replacement but probably doesn't unless the HostnameMap parameter is utilized.

https://review.openstack.org/#/c/266930/ -- this patch is probably the cause for node replacement happening.


Solution suggestion:

There is already a fix for bug #1303094 to prevent node replacement because of user_data. Expanding that fix to prevent node replacement when changing any OS::Nova::Server properties would probably be desirable for TripleO. Node replacement because of properties changing could potentially result in data loss, so it would be good to have it completely prevented by default, and only enable it subsequently if we discover that we need it for some new feature, and when we're ready to deal with node replacement correctly.
Comment 3 Steve Baker 2016-03-03 15:19:53 EST
(In reply to Jiri Stransky from comment #0)
> +++ This bug was initially created as a clone of Bug #1303094 +++
> 
> (cut out the long clone text, please look at the original bug if needed)
> 
> There are new cases of property changes causing unwanted node replacement on
> upgrade Liberty.
> 
> https://review.openstack.org/#/c/286739/ -- this patch could cause node
> replacement but probably doesn't unless the HostnameMap parameter is
> utilized.

Changing the name property does not cause replacement [1] but there is a nova call made to perform the rename. 

> https://review.openstack.org/#/c/266930/ -- this patch is probably the cause
> for node replacement happening.

I would argue that upstream heat should ignore changes to scheduler_hints - replacing a server for changing a boot time hint seems extreme.

> Solution suggestion:
> 
> There is already a fix for bug #1303094 to prevent node replacement because
> of user_data. Expanding that fix to prevent node replacement when changing
> any OS::Nova::Server properties would probably be desirable for TripleO.
> Node replacement because of properties changing could potentially result in
> data loss, so it would be good to have it completely prevented by default,
> and only enable it subsequently if we discover that we need it for some new
> feature, and when we're ready to deal with node replacement correctly.

Yes, we could continue to play whack-a-mole each time upgrade testing shows another property causing server replacement, but just preventing replacements altogether in our custom server resource seems like a good approach - and this would be a temporary measure until Mitaka when heat update restrictions can be used.

[1] http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::Server-prop-name
Comment 4 Zane Bitter 2016-03-03 17:40:15 EST
(In reply to Steve Baker from comment #3)
> I would argue that upstream heat should ignore changes to scheduler_hints -
> replacing a server for changing a boot time hint seems extreme.

That thought occurred to me too, but I'm not sure I agree. Say you have a bunch of servers deployed in random places and you want to start ensuring that e.g. some of them are co-located, how would you force that to happen? (Possible answer: rebuild rather than replace the servers if possible.) And if you didn't want that to happen, why would you have changed the scheduler hint?
Comment 5 Marios Andreou 2016-03-04 02:27:16 EST
sbaker fixup @ https://review.openstack.org/#/c/288273/ "Prevent any property change from replacing OS::Nova::Server"
Comment 6 Steve Baker 2016-03-14 16:46:01 EDT
Upstream stable/liberty change has landed, so this will make it into 8.0 on the next tripleo-common rebase (or backport, I don't know what process is being followed here)

https://review.openstack.org/#/c/291771/
Comment 13 Alexander Chuzhoy 2016-04-14 11:55:38 EDT
Verified:

Environment:
openstack-tripleo-common-0.3.1-1.el7ost.noarch


Re-ran the deployment command with 
Add a yaml with " NovaComputeSchedulerHints: {"some": "json"} ".
Got:
Stack overcloud UPDATE_COMPLETE

The nova uuid are the same as before.

Note You need to log in before you can comment on or make changes to this bug.