1496468 – OSP11 -> OSP12 upgrade: connectivity to floating IP gets is disrupted during major-upgrade-composable-steps-docker.yaml

Bug 1496468 - OSP11 -> OSP12 upgrade: connectivity to floating IP gets is disrupted during major-upgrade-composable-steps-docker.yaml

Summary: OSP11 -> OSP12 upgrade: connectivity to floating IP gets is disrupted during ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-heat-templates
Sub Component:
Version:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	urgent
Target Milestone:	beta
Target Release:	12.0 (Pike)
Assignee:	Sofer Athlan-Guyot
QA Contact:	Marius Cornea
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-09-27 13:29 UTC by Marius Cornea
Modified:	2018-02-05 19:15 UTC (History)
CC List:	9 users (show)
Fixed In Version:	openstack-tripleo-heat-templates-7.0.3-0.20171023134947.8da5e1f.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-12-13 22:11:04 UTC
Target Upstream Version:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
OpenStack gerrit	510577	None	None	None	2017-10-09 13:59:59 UTC
Red Hat Bugzilla	1491628	high	CLOSED	OSP11 -> OSP12 upgrade: Unable to spawn instance post upgrade: Failed to allocate the network(s), not rescheduling.", "c...	2022-08-02 18:02:03 UTC
Red Hat Product Errata	RHEA-2017:3462	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 12.0 Enhancement Advisory	2018-02-16 01:43:25 UTC

Internal Links: 1491628

Description Marius Cornea 2017-09-27 13:29:41 UTC

Description of problem:
OSP11 -> OSP12 upgrade: connectivity to floating IP gets is disrupted during major-upgrade-composable-steps-docker.yaml

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.1-0.20170922213833.8f864ed.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11 with 3 controller, 2 computes, 3 ceph nodes
2. Create router, floating IP on overcloud
3. Spawn instance(allowing ICMP traffic) and attach floating IP to it
4. Before major-upgrade-composable-steps-docker.yaml start ping $floating_ip

Actual results:
Connectivity is disrupted during upgrade.

Expected results:
Connectivity is not affected by upgrade.

Additional info:

Time of disconnect:

Wed Sep 27 13:18:19 UTC 2017


Which matches the time when os-net-config runs ifdown on interfaces:

Sep 27 13:18:19 controller-0 os-collect-config[3147]: + os-net-config -c /etc/os-net-config/config.json -v --detailed-exit-codes
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] Using config file at: /etc/os-net-config/config.json
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] Ifcfg net config provider created.
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] nic3 mapped to: eth2
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] nic2 mapped to: eth1
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] nic1 mapped to: eth0
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding interface: eth0
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding custom route for interface: eth0
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding bridge: br-isolated
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding interface: eth1
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding vlan: vlan20
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding vlan: vlan30
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding vlan: vlan40
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding vlan: vlan50
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding bridge: br-ex
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding custom route for interface: br-ex
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding interface: eth2
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] applying network configs...
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for interface: eth2
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for interface: eth1
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for interface: eth0
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for vlan interface: vlan20
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for vlan interface: vlan30
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for vlan interface: vlan40
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for vlan interface: vlan50
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] running ifdown on interface: vlan50
Sep 27 13:18:21 controller-0 ovs-vsctl[403999]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan50
Sep 27 13:18:21 controller-0 os-collect-config[3147]: [2017/09/27 01:18:21 PM] [INFO] running ifdown on interface: vlan20
Sep 27 13:18:21 controller-0 ovs-vsctl[404060]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan20
Sep 27 13:18:21 controller-0 os-collect-config[3147]: [2017/09/27 01:18:21 PM] [INFO] running ifdown on interface: vlan30
Sep 27 13:18:21 controller-0 ovs-vsctl[404119]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan30
Sep 27 13:18:21 controller-0 os-collect-config[3147]: [2017/09/27 01:18:21 PM] [INFO] running ifdown on interface: vlan40
Sep 27 13:18:22 controller-0 ovs-vsctl[404178]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan40
Sep 27 13:18:22 controller-0 os-collect-config[3147]: [2017/09/27 01:18:22 PM] [INFO] running ifdown on interface: eth1
Sep 27 13:18:22 controller-0 ovs-vsctl[404237]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated eth1
Sep 27 13:18:22 controller-0 os-collect-config[3147]: [2017/09/27 01:18:22 PM] [INFO] running ifdown on interface: eth2
Sep 27 13:18:23 controller-0 ovs-vsctl[404296]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex eth2
Sep 27 13:18:23 controller-0 os-collect-config[3147]: [2017/09/27 01:18:23 PM] [INFO] running ifdown on bridge: br-isolated
Sep 27 13:18:23 controller-0 ovs-vsctl[404355]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-isolated
Sep 27 13:18:23 controller-0 os-collect-config[3147]: [2017/09/27 01:18:23 PM] [INFO] running ifdown on bridge: br-ex
Sep 27 13:18:24 controller-0 ovs-vsctl[404432]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-ex
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-ex
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-isolated
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-isolated
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-ex
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-isolated
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] running ifup on bridge: br-isolated
Sep 27 13:18:24 controller-0 ovs-vsctl[404463]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-isolated -- set bridge br-isolated other-config:hwaddr=52:54:00:24:01:11 -- set bridge br-isolated fail_mode=standalone
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] running ifup on bridge: br-ex
Sep 27 13:18:24 controller-0 ovs-vsctl[404545]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex -- set bridge br-ex other-config:hwaddr=52:54:00:23:df:98 -- set bridge br-ex fail_mode=standalone -- del-controller
Sep 27 13:18:29 controller-0 os-collect-config[3147]: [2017/09/27 01:18:29 PM] [INFO] running ifup on interface: vlan50
Sep 27 13:18:29 controller-0 ovs-vsctl[404648]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan50 -- add-port br-isolated vlan50 tag=50 -- set Interface vlan50 type=internal
Sep 27 13:18:34 controller-0 os-collect-config[3147]: [2017/09/27 01:18:34 PM] [INFO] running ifup on interface: vlan20
Sep 27 13:18:34 controller-0 ovs-vsctl[404744]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan20 -- add-port br-isolated vlan20 tag=20 -- set Interface vlan20 type=internal
Sep 27 13:18:39 controller-0 os-collect-config[3147]: [2017/09/27 01:18:39 PM] [INFO] running ifup on interface: vlan30
Sep 27 13:18:39 controller-0 ovs-vsctl[404842]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan30 -- add-port br-isolated vlan30 tag=30 -- set Interface vlan30 type=internal
Sep 27 13:18:43 controller-0 os-collect-config[3147]: [2017/09/27 01:18:43 PM] [INFO] running ifup on interface: vlan40
Sep 27 13:18:43 controller-0 ovs-vsctl[404952]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan40 -- add-port br-isolated vlan40 tag=40 -- set Interface vlan40 type=internal
Sep 27 13:18:48 controller-0 os-collect-config[3147]: [2017/09/27 01:18:48 PM] [INFO] running ifup on interface: eth1
Sep 27 13:18:49 controller-0 ovs-vsctl[405118]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated eth1 -- add-port br-isolated eth1
Sep 27 13:18:49 controller-0 os-collect-config[3147]: [2017/09/27 01:18:49 PM] [INFO] running ifup on interface: eth2
Sep 27 13:18:49 controller-0 ovs-vsctl[405206]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex eth2 -- add-port br-ex eth2

Comment 1 Sofer Athlan-Guyot 2017-10-04 10:31:36 UTC

It turns out that this should be the same solution as https://bugzilla.redhat.com/show_bug.cgi?id=1491628. 

No doing duplicate, as they are two very different side effect of the same issue.

Comment 2 Sofer Athlan-Guyot 2017-10-09 14:00:00 UTC

stable/pike backport.

Comment 8 errata-xmlrpc 2017-12-13 22:11:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.