Description of problem: Currently heat blacklists the node indices of the overcloud nodes which fail to become part of the overcloud stck and are later removed manually by the user. These node indices permanently remain in blacklisted state : MariaDB [heat]> select resource_data.value,resource.name from resource,resource_data where resource_data.`key`="name_blacklist" and resource_data.resource_id=resource.id; +-------+---------+ | value | name | +-------+---------+ | 5 | Compute | +-------+---------+ 1 row in set (0.00 sec) There should be a way to allow heat to remove these indices from blacklist and allow it to assign it to same or different overcloud nodes during the next scale-out operation. This is frustrating at times when there are large no.of.failed scaleout nodes and the indices are sparsely assigned to the overcloud nodes seen in 'nova list': 1. it makes a novice user confusing as to why is the numbering not in a serial order. 2. When one compares the output of nova list against ironic node-list, one cant co-relate the node hostnames that are assigned - due to the blacklisted indices. 3. There are some customers who want to maintain strict operational standards based on (2). Version-Release number of selected component (if applicable): # OSP-10 : openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch : openstack-heat-engine-7.0.6-1.el7ost.noarch
The way that blacklists work in Heat - where they're sticky, and nodes remain blacklisted even if they're removed from the blacklist - was designed by and for TripleO. The reason is that blacklisting a node is not generally something you want to do in your templates, it's something you decide on the fly. So if Heat didn't maintain the state of the blacklist outside of the property value then you'd need to maintain that state externally. (If you don't remember the blacklist then you end up deleting healthy members at higher indices to fill in the gaps at lower indices.) In any event, it's impossible to change this behaviour without breaking existing templates anyway. This isn't a great design, and in fact ResourceGroups and predictable indices aren't great ideas in general IMHO. For that reason heat introduced the "resource mark unhealthy" command, which aims to achieve the same purpose out of band of the templates, which more closely matches how it's actually used. Unfortunately because of TripleO's heavy reliance on predictable indices being matched up with predefined data such as hostnames and IP addresses, it is unable to make use of mark-unhealthy because Heat creates the replacement resource before removing the unhealthy one, and conflicts ensue. One thing that could be improved in Heat is that if the ResourceGroup is scaled down below the level of a blacklisted index, that index could become available again on the next scale up. So if e.g. you had members 0, 1 & 3 with 2 blacklisted and you scaled down to size 2 (deleting 3), then when you scaled up again the next member would have index 2 instead of 3 unless 2 was _currently_ blacklisted in the template. This would improve usability slightly and probably shouldn't break most existing users, although technically it would be a change in behaviour. I'm not sure how big a difference it would make on the TripleO use case, though... OpenStack clouds don't tend to get smaller very often.
Another possibility is that we could add a whitelist property to remove stuff from the blacklist. Obviously support for using that would then have to be added to TripleO. It sounds like the problem is being exacerbated by the fact that TripleO no longer has a scale-down command but always uses node-delete (which permanently blacklists an index) for scaling down.
The feature that merged upstream was a new 'removal_policies_mode' property on OS::Heat::ResourceGroup. The default value of this property is 'append', which maintains the previous behaviour: once a resource is blacklisted, it is blacklisted permanently so adding additional members to the removal_policies adds them to the blacklist, but removing entries in the removal_policies has no effect. If the 'removal_policies_mode' is changed to 'update', the blacklist is set to the *current* contents of the removal policies, so if the user has removed any entries from there they will no longer be blacklisted and those members will get created (if their indices are smaller than the size of the group). To test this out you could do something like: * Create a ResourceGroup * update with removal_policies, check that resource(s) are deleted * update with no/fewer removal_policies, check that nothing changes * update removal_policies_mode to 'update', check that new resources are created
tested manually according the test case, verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086