Description of problem: I kindly ask for a second look for interesting problem that was reported by customer, which looks like a possible bug in THT. There was an outage in customer's production environment triggered by scale-out operation: os-net-config on existing nodes was called to re-provision networking configuration (sosreports from overcloud nodes are provided, problem occurred at ~ Feb 08 09:18) My first clue was that at some point customer had "u'NetworkDeploymentActions': [u'CREATE', u'UPDATE']" defined in his templates, removed it before scale-out operation and got caught by THT feature that prevents overcloud destruction when scale-out command is called without arguments. I asked customer to provide overcloud deployment plan (attached to case), but couldn't find any "old" definitions of NetworkDeploymentActions parameter. I asked customer to provide sosreports from director node to double-check everything and from mistral-api logs it looks like that at "2021-01-19 09:33:59.951" customer deployed overcloud with "u'NetworkDeploymentActions': [u'CREATE', u'UPDATE']". From mistral-engine logs it looks like scale out operation was executed using the templates without NetworkDeploymentActions definition. It looks like I miss some small details about how THT work and would like to ask for a second look from people with better understanding.
Once you run a deployment with "NetworkDeploymentActions: ['CREATE', 'UPDATE']", you've to reset it back with "NetworkDeploymentActions: ['CREATE']" for it to not run network configs during an scale-out(stack update). Just removing the NetworkDeploymentActions parameter would not change it as we use patch update (i.e use existing parameters in the stack, unless provided). This is the current behaviour. For instance during ffwd-upgrade we reset it back with converge[2], after changing it during prepare[1]. [1] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/lifecycle/ffwd-upgrade-prepare.yaml#L21 [2] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/lifecycle/ffwd-upgrade-converge.yaml#L14 However, I think we can change this behaviour as this is disruptive in some instances. I'll propose a fix.
Thank you very much!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 13.0 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2385