Bug 1115151

A bit more context: Most OpenStack admins running OpenStack on a Fedora based OS will want to persist IP information on the "external" OVS bridge. br-ex
will be connected to eth0, for example, and will exist on the Neutron network node. The only way that I know of to persist IP info on br-ex is via ifcfg-br-ex, with OVSBridge as the device type so that ifup/down-ovs will take care of that device, resulting in successful reboots (IE: br-ex will retain its IP address post-reboot).

The issue is that ifup/down-ovs destroy/create the bridge, meaning that 'service network restart' and its systemd equivalent destroy the bridge. Neutron virtual routers connected to br-ex will be disconnected and won't be reconnected, unless the admin restarts the agent responsible for those routers. That's a very expensive operation and causes (Possibly very long / 30+ minutes) network outage, depending on scale.

It should be noted that this has been the behavior since forever, it's not a regression. It's only come up recently because the Red Hat OpenStack deployment tools have begun to create br-ex and hook it up to eth* by default, which was previously not the case. However, users would still have to create br-ex, connect it to eth* and create ifcfg-br-ex manually. 'service network restart' still destroys the bridges, and I'm not aware of any reasonable workaround we could recommend until this behavior is fixed at the OVS network scripts level.

Info from fleitner:

"I've proposed a RFC in upstream mailing list[1] but got 0 traction until now.
[1] http://openvswitch.org/pipermail/discuss/2014-July/014507.html
 
and after some more thought I think the current behavior is correct."

Flavio, based on this, what do you suggest we do with this bugzilla?

Maybe we could discuss why the current behavior is correct or incorrect?

Here's my case:

With the supplied ifcfg-br-lb and ifcfg-br-ovs, I ran ifup br-ovs, ifup br-lb.
Both the OVS and the Linux bridge were created. I then added dummy0 and dummy1 to the OVS bridge and dummy2 and dummy3 to the Linux bridge via ovs-vsctl and brctl.

ifdown br-ovs && ifup br-ovs caused the detachment of the dummy NICs from the bridge, because the bridge was destroyed and created again.

ifdown br-lb && ifup br-lb merely set the bridge as down and then up, retaining its connected dummy NICs.

I think that the Linux bridge behavior makes more sense.

Created attachment 919333 [details]
Linux bridge example ifcfg

Hi,

The OVS bridge keeps a database with all the OVS configuration like controller IP address, attached ports, ports types, QoS and much more.  So ifdown destroys the bridge and that cleans up the database as well. After an ifup, the bridge's configuration can be safely restored.

Another important difference is that if you put the Linux bridge down, no traffic pass to any port because you turned off the datapath.  For OVS bridges that's not the case because the bridge is just another port attached to the datapath. The result is that a single port named after the bridge's name is down, not the whole datapath.

Having said the above, I would like to know what exactly are you trying to archive by running 'ifdown <ovsbr>'.

Thanks!

The issue is customers running 'service network restart'. On an OpenStack Neutron network node you're gonna have an OVS bridge called br-ex connected to a physical NIC that's connected to a public network.

To persist IP information on br-ex the customer will normally create ifcfg-br-ex. service network restart will then trigger ifdown on br-ex which will destroy the bridge, ifup will recreate it, but any ports previously connected to br-ex will no longer be connected to it. Neutron's virtual routers are connected to br-ex. Having them no longer connected means that any VM traffic destined to the public network will drop. This can be fixed by restarting the Neutron l3 agent as well, which will reconnect the virtual routers to br-ex.

I reported the bug because the behavior of destroying and recreating bridges surprised me with my exposure Linux bridge's behavior.

Are we pleased with 'service network restart' destroying connectivity? This isn't the case when running with the Linux bridge agent to the best of my knowledge, for example.

(In reply to Assaf Muller from comment #7)
[...]
> but any ports previously connected to br-ex will no longer be
> connected to it. Neutron's virtual routers are connected to br-ex.

It seems you are configuring the bridge to persist but not the port. Is there a way to configure the port to persist as well?  That would trigger ifup in the port and attach it back to the bridge resolving the issue.

It's easy enough to persist the physical NIC that's connected to br-ex, but not all of the Neutron ports connected to it.

Ok, because the opposite also can happen.  If you run 'service network restart' and for some reason a port is not available anymore, it will remain on ovs DB (assuming that ovs bridge isn't destroyed) which might break scripts trying to add the same port again later.

This emulates what I am trying to say:
// creating the bridge
# ovs-vsctl add-br ovsbr0
# ip link set ovsbr0 up

// creating a virtual port 
# ip link add name int0 type veth peer name host0 
# ip link set int0 up

// adding to the bridge
# ovs-vsctl add-port ovsbr0 int0 
# ovs-vsctl show
95f955a4-1e68-4121-94a8-8cf9dde29eea
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "int0"
            Interface "int0"
    ovs_version: "2.1.2"

// delete the port
# ip link del int0
# ip link show dev int0
Device "int0" does not exist.
# ovs-vsctl show
95f955a4-1e68-4121-94a8-8cf9dde29eea
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "int0"
            Interface "int0"
    ovs_version: "2.1.2"

// add the same port
# ip link add name int0 type veth peer name host0 
# ovs-vsctl add-port ovsbr0 int0 
ovs-vsctl: cannot create a port named int0 because a port named int0 already exists on bridge ovsbr0

Also, during boot time the openvswitch service needs to run before network service.  Once it starts, the DB info is used and all the interfaces attached to the bridge before shutdown will be present in the DB because ifdown won't clean it up anymore.  Then, all ifup on ports later will fail because they are already attached.

To top it off, I am sure there are scripts out there relying on the current behavior, so it seems to me that we will have more troubles changing the current behavior.

One possible workaround is to use a custom /sbin/ifup-local script.  It receives the device (in this case the ovs bridge) as an argument so you can fix up your ports there. e.g.:
if [ "$1" == "br-ex" ]; then
   if device_exists neutron_port;
      ovs-vsctl add-port br-ex neutron-port
   fi
fi

Hi Assaf,

What do you think about the workaround provided in comment#10?
Thanks!

If ifcfg files can take care of OVS patch ports then I might have a workaround. Failing that I can look into ifup-local extracting the br-int name and the external network OVS bridge name, and hook them up with OVS patch ports. That should solve the issue. We'll have to maintain this type of workaround script in all of our deployment tools.

Keeping the needinfo on me until I investigate a possible workaround.

Hi Assaf, 

Yes, RHEL-7 OVS scripts starting at version 2.0.1-2 supports patch ports.
Here is one pair of ifcfgs as an example:

----8<----
DEVICE=patch0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSPatchPort
OVS_BRIDGE=ovsbr0
OVS_PATCH_PEER=patch1
----8<----

and:

----8<----
DEVICE=patch1
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSPatchPort
OVS_BRIDGE=ovsbr1
OVS_PATCH_PEER=patch0
----8<----

*** Bug 1117252 has been marked as a duplicate of this bug. ***

Hi Assaf,

Any news on the work around?
Let me know if you need my help.
Thanks
fbl

Hi Assaf,

Just wondering how things are going here.
Do you need any help from me?
Thanks,
fbl

Hi Assaf, 
Any updates
Can you please give a time frame when you will be able to have a look 
THanks