Bug 1496468

Summary: OSP11 -> OSP12 upgrade: connectivity to floating IP gets is disrupted during major-upgrade-composable-steps-docker.yaml
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Sofer Athlan-Guyot <sathlang>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: dbecker, jschluet, mbracho, mbultel, mburns, morazi, rhel-osp-director-maint, sathlang, tvignaud
Target Milestone: betaKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-7.0.3-0.20171023134947.8da5e1f.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 22:11:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2017-09-27 13:29:41 UTC 
Description of problem:
OSP11 -> OSP12 upgrade: connectivity to floating IP gets is disrupted during major-upgrade-composable-steps-docker.yaml

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.1-0.20170922213833.8f864ed.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11 with 3 controller, 2 computes, 3 ceph nodes
2. Create router, floating IP on overcloud
3. Spawn instance(allowing ICMP traffic) and attach floating IP to it
4. Before major-upgrade-composable-steps-docker.yaml start ping $floating_ip

Actual results:
Connectivity is disrupted during upgrade.

Expected results:
Connectivity is not affected by upgrade.

Additional info:

Time of disconnect:

Wed Sep 27 13:18:19 UTC 2017


Which matches the time when os-net-config runs ifdown on interfaces:

Sep 27 13:18:19 controller-0 os-collect-config[3147]: + os-net-config -c /etc/os-net-config/config.json -v --detailed-exit-codes
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] Using config file at: /etc/os-net-config/config.json
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] Ifcfg net config provider created.
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] nic3 mapped to: eth2
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] nic2 mapped to: eth1
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] nic1 mapped to: eth0
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding interface: eth0
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding custom route for interface: eth0
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding bridge: br-isolated
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding interface: eth1
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding vlan: vlan20
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding vlan: vlan30
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding vlan: vlan40
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding vlan: vlan50
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding bridge: br-ex
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding custom route for interface: br-ex
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] adding interface: eth2
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] applying network configs...
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for interface: eth2
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for interface: eth1
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for interface: eth0
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for vlan interface: vlan20
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for vlan interface: vlan30
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for vlan interface: vlan40
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] No changes required for vlan interface: vlan50
Sep 27 13:18:20 controller-0 os-collect-config[3147]: [2017/09/27 01:18:20 PM] [INFO] running ifdown on interface: vlan50
Sep 27 13:18:21 controller-0 ovs-vsctl[403999]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan50
Sep 27 13:18:21 controller-0 os-collect-config[3147]: [2017/09/27 01:18:21 PM] [INFO] running ifdown on interface: vlan20
Sep 27 13:18:21 controller-0 ovs-vsctl[404060]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan20
Sep 27 13:18:21 controller-0 os-collect-config[3147]: [2017/09/27 01:18:21 PM] [INFO] running ifdown on interface: vlan30
Sep 27 13:18:21 controller-0 ovs-vsctl[404119]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan30
Sep 27 13:18:21 controller-0 os-collect-config[3147]: [2017/09/27 01:18:21 PM] [INFO] running ifdown on interface: vlan40
Sep 27 13:18:22 controller-0 ovs-vsctl[404178]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan40
Sep 27 13:18:22 controller-0 os-collect-config[3147]: [2017/09/27 01:18:22 PM] [INFO] running ifdown on interface: eth1
Sep 27 13:18:22 controller-0 ovs-vsctl[404237]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated eth1
Sep 27 13:18:22 controller-0 os-collect-config[3147]: [2017/09/27 01:18:22 PM] [INFO] running ifdown on interface: eth2
Sep 27 13:18:23 controller-0 ovs-vsctl[404296]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex eth2
Sep 27 13:18:23 controller-0 os-collect-config[3147]: [2017/09/27 01:18:23 PM] [INFO] running ifdown on bridge: br-isolated
Sep 27 13:18:23 controller-0 ovs-vsctl[404355]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-isolated
Sep 27 13:18:23 controller-0 os-collect-config[3147]: [2017/09/27 01:18:23 PM] [INFO] running ifdown on bridge: br-ex
Sep 27 13:18:24 controller-0 ovs-vsctl[404432]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-ex
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-ex
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route6-br-isolated
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/route-br-isolated
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-ex
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-br-isolated
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] running ifup on bridge: br-isolated
Sep 27 13:18:24 controller-0 ovs-vsctl[404463]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-isolated -- set bridge br-isolated other-config:hwaddr=52:54:00:24:01:11 -- set bridge br-isolated fail_mode=standalone
Sep 27 13:18:24 controller-0 os-collect-config[3147]: [2017/09/27 01:18:24 PM] [INFO] running ifup on bridge: br-ex
Sep 27 13:18:24 controller-0 ovs-vsctl[404545]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex -- set bridge br-ex other-config:hwaddr=52:54:00:23:df:98 -- set bridge br-ex fail_mode=standalone -- del-controller
Sep 27 13:18:29 controller-0 os-collect-config[3147]: [2017/09/27 01:18:29 PM] [INFO] running ifup on interface: vlan50
Sep 27 13:18:29 controller-0 ovs-vsctl[404648]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan50 -- add-port br-isolated vlan50 tag=50 -- set Interface vlan50 type=internal
Sep 27 13:18:34 controller-0 os-collect-config[3147]: [2017/09/27 01:18:34 PM] [INFO] running ifup on interface: vlan20
Sep 27 13:18:34 controller-0 ovs-vsctl[404744]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan20 -- add-port br-isolated vlan20 tag=20 -- set Interface vlan20 type=internal
Sep 27 13:18:39 controller-0 os-collect-config[3147]: [2017/09/27 01:18:39 PM] [INFO] running ifup on interface: vlan30
Sep 27 13:18:39 controller-0 ovs-vsctl[404842]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan30 -- add-port br-isolated vlan30 tag=30 -- set Interface vlan30 type=internal
Sep 27 13:18:43 controller-0 os-collect-config[3147]: [2017/09/27 01:18:43 PM] [INFO] running ifup on interface: vlan40
Sep 27 13:18:43 controller-0 ovs-vsctl[404952]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated vlan40 -- add-port br-isolated vlan40 tag=40 -- set Interface vlan40 type=internal
Sep 27 13:18:48 controller-0 os-collect-config[3147]: [2017/09/27 01:18:48 PM] [INFO] running ifup on interface: eth1
Sep 27 13:18:49 controller-0 ovs-vsctl[405118]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-isolated eth1 -- add-port br-isolated eth1
Sep 27 13:18:49 controller-0 os-collect-config[3147]: [2017/09/27 01:18:49 PM] [INFO] running ifup on interface: eth2
Sep 27 13:18:49 controller-0 ovs-vsctl[405206]: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex eth2 -- add-port br-ex eth2
Comment 1 Sofer Athlan-Guyot 2017-10-04 10:31:36 UTC 
It turns out that this should be the same solution as https://bugzilla.redhat.com/show_bug.cgi?id=1491628. 

No doing duplicate, as they are two very different side effect of the same issue.
Comment 2 Sofer Athlan-Guyot 2017-10-09 14:00:00 UTC 
stable/pike backport.
Comment 8 errata-xmlrpc 2017-12-13 22:11:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462