Bug 1516867 - OSP11 -> OSP12 upgrade: floating IP connectivity gets disrupted during major-upgrade-composable-steps-docker
Summary: OSP11 -> OSP12 upgrade: floating IP connectivity gets disrupted during major-...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 12.0 (Pike)
Assignee: Sofer Athlan-Guyot
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-23 13:06 UTC by Marius Cornea
Modified: 2018-02-05 19:18 UTC (History)
7 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.3-15.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-13 22:22:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1734135 0 None None None 2017-11-23 14:43:54 UTC
OpenStack gerrit 522833 0 None stable/pike: MERGED tripleo-heat-templates: Fix wrong logic for triggering os-net-config special handling. (I6da13ad8bc9675ea2a80006615f4dc9... 2017-11-28 15:35:14 UTC
Red Hat Bugzilla 1516890 0 urgent CLOSED OSP11 -> OSP12 upgrade: upgrade-non-controller.sh script fails while running the upgrade os-net-config tasks 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHEA-2017:3462 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Internal Links: 1516890

Description Marius Cornea 2017-11-23 13:06:31 UTC
Description of problem:
OSP11 -> OSP12 upgrade: floating IP connectivity gets disrupted during major-upgrade-composable-steps-docker.

After the major upgrade composable step completes we can see the following ping results:

3991 packets transmitted, 2491 received, +1459 errors, 37% packet loss, time 3993054ms
rtt min/avg/max/mdev = 0.516/1.087/6.887/0.387 ms, pipe 4
Ping loss higher than 1% detected


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.3-12.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11
2. Launch an instance and assign it a floating IP via a Neutron router
3. Start ping of the floating IP attached to the instance
4. Run major upgrade composable step docker 

Actual results:
The ping results show a high amount of packet loss.

Expected results:
A close to 0 percentage of packet loss should happen.

Additional info:
This is a regression introduced in the latest build.

Comment 1 Marius Cornea 2017-11-23 13:27:31 UTC
Snippet from os-collect-config journal on the networker node:

Nov 23 12:51:31 networker-0 os-collect-config[2953]: PLAY [localhost] ***************************************************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Gathering Facts] *********************************************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: ok: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Check if neutron_dhcp_agent is deployed] *********************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: changed: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Check if neutron_l3_agent is deployed] ***********************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: changed: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Check if neutron_metadata_agent is deployed] *****************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: changed: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Check if neutron_ovs_agent is deployed] **********************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: changed: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Install docker packages on upgrade if missing] ***************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: ok: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Check for os-net-config upgrade] *****************************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: changed: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Check that os-net-config has configuration] ******************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: changed: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Upgrade os-net-config] ***************************************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: skipping: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [take new os-net-config parameters into account now] **********************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: skipping: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [Update all packages] *****************************************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: changed: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: TASK [blank ipv6 rule before activating ipv6 firewall.] ************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: changed: [localhost]
Nov 23 12:51:31 networker-0 os-collect-config[2953]: PLAY RECAP *********************************************************************
Nov 23 12:51:31 networker-0 os-collect-config[2953]: localhost                  : ok=10   changed=8    unreachable=0    failed=0

We can see that the block for special case of os-net-config upgrade got skipped:

https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/tripleo-packages.yaml#L66-L75

Comment 6 errata-xmlrpc 2017-12-13 22:22:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462


Note You need to log in before you can comment on or make changes to this bug.