Bug 1928055

Summary: [RHOSP13][RCA] os-net-config was triggered for existing nodes during scale-out operation
Product: Red Hat OpenStack Reporter: Alex Stupnikov <astupnik>
Component: openstack-tripleo-heat-templatesAssignee: Rabi Mishra <ramishra>
Status: CLOSED ERRATA QA Contact: Joe H. Rahme <jhakimra>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: cjeanner, jelle.hoylaerts.ext, mburns, ramishra
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.4.1-79.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-16 10:58:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Stupnikov 2021-02-12 09:40:06 UTC 
Description of problem:

I kindly ask for a second look for interesting problem that was reported by customer, which looks like a possible bug in THT.

There was an outage in customer's production environment triggered by scale-out operation: os-net-config on existing nodes was called to re-provision networking configuration (sosreports from overcloud nodes are provided, problem occurred at ~ Feb 08 09:18)

My first clue was that at some point customer had "u'NetworkDeploymentActions': [u'CREATE', u'UPDATE']" defined in his templates, removed it before scale-out operation and got caught by THT feature that prevents overcloud destruction when scale-out command is called without arguments.

I asked customer to provide overcloud deployment plan (attached to case), but couldn't find any "old" definitions of NetworkDeploymentActions parameter.

I asked customer to provide sosreports from director node to double-check everything and from mistral-api logs it looks like that at "2021-01-19 09:33:59.951" customer deployed overcloud with "u'NetworkDeploymentActions': [u'CREATE', u'UPDATE']". From mistral-engine logs it looks like scale out operation was executed using the templates without NetworkDeploymentActions definition.

It looks like I miss some small details about how THT work and would like to ask for a second look from people with better understanding.
Comment 1 Rabi Mishra 2021-02-13 06:12:20 UTC 
Once you run a deployment with "NetworkDeploymentActions: ['CREATE', 'UPDATE']", you've to reset it back with "NetworkDeploymentActions: ['CREATE']" for it to not run network configs during an scale-out(stack update). Just removing the NetworkDeploymentActions parameter would not change it as we use patch update (i.e use existing parameters in the stack, unless provided).

This is the current behaviour.

For instance during ffwd-upgrade we reset it back with converge[2], after changing it during prepare[1].


[1] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/lifecycle/ffwd-upgrade-prepare.yaml#L21
[2] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/environments/lifecycle/ffwd-upgrade-converge.yaml#L14


However, I think we can change this behaviour as this is disruptive in some instances. I'll propose a fix.
Comment 2 Alex Stupnikov 2021-02-15 08:22:43 UTC 
Thank you very much!
Comment 13 errata-xmlrpc 2021-06-16 10:58:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 13.0 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2385