Bug 1663932

> When using legacy init-scripts, it is common to add an OVS option
> that deletes the OVS controller when an OVS bridge is restarted.

What do you mean with restarting an OVS bridge? I suppose this is
'ifup <bridge>', right?

I tried this using initscripts (NM is disabled), but the controller
is not deleted after a restart:

  # (create the bridge and activate it)

 [root@rhel8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br-ex
 DEVICE=br-ex
 ONBOOT=yes
 HOTPLUG=no
 NM_CONTROLLED=no
 PEERDNS=no
 DEVICETYPE=ovs
 TYPE=OVSBridge
 OVS_EXTRA="set bridge br-ex fail_mode=standalone -- del-controller br-ex"

 [root@rhel8 ~]# ifup br-ex
 WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
 WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.
 WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.

 [root@rhel8 ~]# ovs-vsctl show
 7a06ef3b-0b36-43d4-8bac-cbb45fca6925
     Bridge br-ex
         fail_mode: standalone
         Port br-ex
             Interface br-ex
                 type: internal
     ovs_version: "2.9.0"

  # (manually set a controller)

 [root@rhel8 ~]# ovs-vsctl set-controller br-ex tcp:127.0.0.1

 [root@rhel8 ~]# ovs-vsctl show
 7a06ef3b-0b36-43d4-8bac-cbb45fca6925
     Bridge br-ex
         Controller "tcp:127.0.0.1"
         fail_mode: standalone
         Port br-ex
             Interface br-ex
                 type: internal
     ovs_version: "2.9.0"

  # (now restart the bridge)

 [root@rhel8 ~]# ifup br-ex
 WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
 WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.
 WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.

 [root@rhel8 ~]# ovs-vsctl show
 7a06ef3b-0b36-43d4-8bac-cbb45fca6925
     Bridge br-ex
         Controller "tcp:127.0.0.1"
         fail_mode: standalone
         Port br-ex
             Interface br-ex
                 type: internal
     ovs_version: "2.9.0"

  # (the controller it's still there)


Talking instead about NetworkManager, it always creates a new bridge
from scratch in the ovsdb when the bridge connections gets activated,
so I guess the 'del-controller' parameter is not needed, because at
the time the 'del-controller' would be called there is no controller.

Dan, what do you think?

(In reply to Beniamino Galvani from comment #1)
> > When using legacy init-scripts, it is common to add an OVS option
> > that deletes the OVS controller when an OVS bridge is restarted.
> 
> What do you mean with restarting an OVS bridge? I suppose this is
> 'ifup <bridge>', right?

Actually, the problem is only apparent when using OpenStack Neutron with a remote OVS controller, and happens when the OVS system is restarted. I'm actually not sure if a simple "ifdown/ifup" sequence will trigger the problem.

What happens is that after the bridge is created on the host, the Neutron OVS agent running on the host will connect OVS to a remote OVS controller running on the Neutron server and the fail mode is set to "secure". If the OVS process is restarted (as happens during upgrades), traffic will cease to flow on the bridge due to the secure fail mode. In an architecture where the remote controller is accessed over the bridge, this means that the host cannot connect to the OVS controller since traffic isn't flowing on the bridge.

In order to address this situation, we add the "set bridge br-ex fail_mode=standalone -- del-controller br-ex" to the bridge ifcfg file. This has the effect of bringing up the bridge in standalone mode, and deleting the connection to the remote server. The Neutron OVS agent will then reconnect OVS to the remote controller and set the fail mode to "secure" again.

> I tried this using initscripts (NM is disabled), but the controller
> is not deleted after a restart:
> [...]

I am not sure why you saw the behavior where the 127.0.0.1 controller is not deleted. When the controller is remote, and the Neutron OVS agent is the one that sets the controller, the remote controller is deleted by the "del-controller <bridge>" command.

> Talking instead about NetworkManager, it always creates a new bridge
> from scratch in the ovsdb when the bridge connections gets activated,
> so I guess the 'del-controller' parameter is not needed, because at
> the time the 'del-controller' would be called there is no controller.
> 
> Dan, what do you think?

The fact that NetworkManager will always create a new bridge from scratch when the bridge is activated might alleviate the condition where the bridge stops forwarding traffic. It will also have a side effect of deleting all existing flows from ovsdb when the bridge is recreated. This probably won't create additional issues, since the remote controller should recreate those flows I believe. We will need to perform testing with OpenStack since the workflow is different (deleting the bridge from ovsdb rather than simply modifying the state of an existing bridge).

I am meeting with the Neutron developers this week and will ask them what (if any) differences they expect in behavior if the bridge is always recreated.

> I am meeting with the Neutron developers this week and will ask them what (if any) differences they expect in behavior if the bridge is always recreated.

Hi Dan,

do you have any update?

(In reply to Beniamino Galvani from comment #5)
> > I am meeting with the Neutron developers this week and will ask them what (if any) differences they expect in behavior if the bridge is always recreated.
> 
> Hi Dan,
> 
> do you have any update?

I am sorry, the last time I updated this bug was over a year ago, and I don't recall what the status was that long ago.

There are still features missing from NetworkManager and nmstate that prevent us from using NetworkManager with OpenStack, so this isn't something we can test. 
I suspect that if the bridge is recreated every time NetworkManager brings up the bridge using "ifup" that there will not be any issues. In perhaps 6 months we hope that NetworkManager will support enough features that we can use it with OpenStack in RHEL 8, and then we should have some more data to tell us if there are still any problems.

Feel free to close this bug in the mean time if you would like.

(In reply to Dan Sneddon from comment #6)
> Feel free to close this bug in the mean time if you would like.

Okay, thank you. Please reopen if needed.