Bug 1419159

Summary: RHEL 7 upgrading openvswitch version 2.4 -> 2.5 causes OVS interfaces to lose IP addresses
Product: Red Hat Enterprise Linux 7 Reporter: Matt Flusche <mflusche>
Component: openvswitchAssignee: Open vSwitch development team <ovs-team>
Status: CLOSED DUPLICATE QA Contact: ovs-qe
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: atragler, fleitner, mflusche
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-07 21:43:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Matt Flusche 2017-02-03 18:05:56 UTC
Description of problem:

A yum upgrade of the openvswitch package from 2.4 -> 2.4 causes OVS interfaces to lose IP addresses.

Ran into this issue while upgrading all packages from 7.2 -> 7.3.  Narrowed the issue down to just the openvswitch package.

Version-Release number of selected component (if applicable):

openvswitch-2.5.0-14.git20160727.el7fdp.x86_64

How reproducible:
100% so far


Steps to Reproduce:
upgrade openvswitch package.

In this environment I have two OVS interfaces with addresses assigned.

[root@overcloud-controller-0 ~]# ip a ls br-ex
16: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 52:54:00:b7:9f:2d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.132/24 brd 192.168.0.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feb7:9f2d/64 scope link 
       valid_lft forever preferred_lft forever


[root@overcloud-controller-0 ~]# ip a ls vlan5
18: vlan5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 2a:de:c0:f3:e6:56 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.81/24 brd 192.168.2.255 scope global vlan5
       valid_lft forever preferred_lft forever
    inet 192.168.2.80/32 brd 192.168.2.255 scope global vlan5
       valid_lft forever preferred_lft forever
    inet6 fe80::28de:c0ff:fef3:e656/64 scope link 
       valid_lft forever preferred_lft forever


Version of OVS before:

[root@overcloud-controller-0 ~]# rpm -q openvswitch
openvswitch-2.4.0-2.el7_2.x86_64

Update:

[root@overcloud-controller-0 ~]# yum -y update openvswitch

...
Updated:
  openvswitch.x86_64 0:2.5.0-14.git20160727.el7fdp                                                                                    

Complete!

OVS interfaces are now missing addresses:

[root@overcloud-controller-0 ~]# ip a ls br-ex
23: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 52:54:00:b7:9f:2d brd ff:ff:ff:ff:ff:ff

[root@overcloud-controller-0 ~]# ip a ls vlan5
24: vlan5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 16:e8:16:88:26:ac brd ff:ff:ff:ff:ff:ff


ifup will bring them back online:

[root@overcloud-controller-0 ~]# ifup br-ex
[root@overcloud-controller-0 ~]# ifup vlan5

[root@overcloud-controller-0 ~]# ip a ls br-ex
23: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 52:54:00:b7:9f:2d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.132/24 brd 192.168.0.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feb7:9f2d/64 scope link 
       valid_lft forever preferred_lft forever

[root@overcloud-controller-0 ~]# ip a ls vlan5
24: vlan5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 16:e8:16:88:26:ac brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.81/24 brd 192.168.2.255 scope global vlan5
       valid_lft forever preferred_lft forever
    inet6 fe80::14e8:16ff:fe88:26ac/64 scope link 
       valid_lft forever preferred_lft forever



Actual results:
IP addresses missing after patch.

Expected results:
OVS networking to function the same as before the update

Additional info:

Initially we thought this was related to other package update BZs:
NetworkManager
https://bugzilla.redhat.com/show_bug.cgi?id=1367580
initscripts
https://bugzilla.redhat.com/show_bug.cgi?id=1366348

I tested those updates and they didn't break this environment.

Comment 3 Flavio Leitner 2017-02-07 21:18:34 UTC
The installed OVS 2.4 package restarts the service during the upgrade (%postun) and while doing it, it will destroy internal ports like br-ex and add a fresh one.  Unfortunately, there is no notification system to restore the networking configuration for those devices that OVS could make use of it.

We have fixed OVS 2.5 package (openvswitch-2.5.0-23.git20160727.el7fdp) to not restart the service during the upgrade or to make sure the service is not broken after the upgrade, but the recommended way is to restart the whole networking after upgrading OVS package.

See bz#1403958 and bz#1397045.

Comment 4 Matt Flusche 2017-02-07 21:43:30 UTC
Hi Flavio,

Thanks for the follow up.  I wasn't aware that a work-around was introduced into a new version of the OSP Director heat templates to deal with this.  https://bugzilla.redhat.com/show_bug.cgi?id=1388546.  I was testing with an old version.  I will close this as a duplicate.

Thanks,

Matt

*** This bug has been marked as a duplicate of bug 1397045 ***