RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1663932 - [RFE] NetworkManager should support deleting OVS controllers on restart
Summary: [RFE] NetworkManager should support deleting OVS controllers on restart
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: NetworkManager
Version: 9.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: beta
: 9.0 Beta
Assignee: NetworkManager Development Team
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks: nmstate-nm
TreeView+ depends on / blocked
 
Reported: 2019-01-07 10:23 UTC by Dan Sneddon
Modified: 2021-08-17 17:16 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-08-17 17:16:26 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-92707 0 None None None 2021-08-06 16:40:06 UTC

Internal Links: 1645689

Description Dan Sneddon 2019-01-07 10:23:34 UTC 
Description of problem:
When using legacy init-scripts, it is common to add an OVS option that deletes the OVS controller when an OVS bridge is restarted. This is commonly done to force a higher-level application to reconnect to an external OVS controller. There needs to be an equivalent function for NetworkManager.

Actual results:
There is no way to configure NetworkManager to delete the bridge controller.

Expected results:
There should be a way to cause this action to be sent to ovs-vsctl when bridge "br-test" is restarted:

del-controller br-test

Additional info:
With ifcfg files, you can simply add OVS commands to the OVS_EXTRA configuration parameter, with each command separated by "--" double dashes. Here is a common pair of commands:

set bridge br-test fail_mode=standalone -- del-controller br-test
Comment 1 Beniamino Galvani 2019-01-08 13:10:57 UTC 
> When using legacy init-scripts, it is common to add an OVS option
> that deletes the OVS controller when an OVS bridge is restarted.

What do you mean with restarting an OVS bridge? I suppose this is
'ifup <bridge>', right?

I tried this using initscripts (NM is disabled), but the controller
is not deleted after a restart:

  # (create the bridge and activate it)

 [root@rhel8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br-ex
 DEVICE=br-ex
 ONBOOT=yes
 HOTPLUG=no
 NM_CONTROLLED=no
 PEERDNS=no
 DEVICETYPE=ovs
 TYPE=OVSBridge
 OVS_EXTRA="set bridge br-ex fail_mode=standalone -- del-controller br-ex"

 [root@rhel8 ~]# ifup br-ex
 WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
 WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.
 WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.

 [root@rhel8 ~]# ovs-vsctl show
 7a06ef3b-0b36-43d4-8bac-cbb45fca6925
     Bridge br-ex
         fail_mode: standalone
         Port br-ex
             Interface br-ex
                 type: internal
     ovs_version: "2.9.0"

  # (manually set a controller)

 [root@rhel8 ~]# ovs-vsctl set-controller br-ex tcp:127.0.0.1

 [root@rhel8 ~]# ovs-vsctl show
 7a06ef3b-0b36-43d4-8bac-cbb45fca6925
     Bridge br-ex
         Controller "tcp:127.0.0.1"
         fail_mode: standalone
         Port br-ex
             Interface br-ex
                 type: internal
     ovs_version: "2.9.0"

  # (now restart the bridge)

 [root@rhel8 ~]# ifup br-ex
 WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
 WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.
 WARN      : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.

 [root@rhel8 ~]# ovs-vsctl show
 7a06ef3b-0b36-43d4-8bac-cbb45fca6925
     Bridge br-ex
         Controller "tcp:127.0.0.1"
         fail_mode: standalone
         Port br-ex
             Interface br-ex
                 type: internal
     ovs_version: "2.9.0"

  # (the controller it's still there)


Talking instead about NetworkManager, it always creates a new bridge
from scratch in the ovsdb when the bridge connections gets activated,
so I guess the 'del-controller' parameter is not needed, because at
the time the 'del-controller' would be called there is no controller.

Dan, what do you think?
Comment 2 Dan Sneddon 2019-01-08 21:34:55 UTC 
(In reply to Beniamino Galvani from comment #1)
> > When using legacy init-scripts, it is common to add an OVS option
> > that deletes the OVS controller when an OVS bridge is restarted.
> 
> What do you mean with restarting an OVS bridge? I suppose this is
> 'ifup <bridge>', right?

Actually, the problem is only apparent when using OpenStack Neutron with a remote OVS controller, and happens when the OVS system is restarted. I'm actually not sure if a simple "ifdown/ifup" sequence will trigger the problem.

What happens is that after the bridge is created on the host, the Neutron OVS agent running on the host will connect OVS to a remote OVS controller running on the Neutron server and the fail mode is set to "secure". If the OVS process is restarted (as happens during upgrades), traffic will cease to flow on the bridge due to the secure fail mode. In an architecture where the remote controller is accessed over the bridge, this means that the host cannot connect to the OVS controller since traffic isn't flowing on the bridge.

In order to address this situation, we add the "set bridge br-ex fail_mode=standalone -- del-controller br-ex" to the bridge ifcfg file. This has the effect of bringing up the bridge in standalone mode, and deleting the connection to the remote server. The Neutron OVS agent will then reconnect OVS to the remote controller and set the fail mode to "secure" again.

> I tried this using initscripts (NM is disabled), but the controller
> is not deleted after a restart:
> [...]

I am not sure why you saw the behavior where the 127.0.0.1 controller is not deleted. When the controller is remote, and the Neutron OVS agent is the one that sets the controller, the remote controller is deleted by the "del-controller <bridge>" command.

> Talking instead about NetworkManager, it always creates a new bridge
> from scratch in the ovsdb when the bridge connections gets activated,
> so I guess the 'del-controller' parameter is not needed, because at
> the time the 'del-controller' would be called there is no controller.
> 
> Dan, what do you think?

The fact that NetworkManager will always create a new bridge from scratch when the bridge is activated might alleviate the condition where the bridge stops forwarding traffic. It will also have a side effect of deleting all existing flows from ovsdb when the bridge is recreated. This probably won't create additional issues, since the remote controller should recreate those flows I believe. We will need to perform testing with OpenStack since the workflow is different (deleting the bridge from ovsdb rather than simply modifying the state of an existing bridge).

I am meeting with the Neutron developers this week and will ask them what (if any) differences they expect in behavior if the bridge is always recreated.
Comment 5 Beniamino Galvani 2020-03-24 13:46:24 UTC 
> I am meeting with the Neutron developers this week and will ask them what (if any) differences they expect in behavior if the bridge is always recreated.

Hi Dan,

do you have any update?
Comment 6 Dan Sneddon 2020-03-25 21:09:19 UTC 
(In reply to Beniamino Galvani from comment #5)
> > I am meeting with the Neutron developers this week and will ask them what (if any) differences they expect in behavior if the bridge is always recreated.
> 
> Hi Dan,
> 
> do you have any update?

I am sorry, the last time I updated this bug was over a year ago, and I don't recall what the status was that long ago.

There are still features missing from NetworkManager and nmstate that prevent us from using NetworkManager with OpenStack, so this isn't something we can test. 
I suspect that if the bridge is recreated every time NetworkManager brings up the bridge using "ifup" that there will not be any issues. In perhaps 6 months we hope that NetworkManager will support enough features that we can use it with OpenStack in RHEL 8, and then we should have some more data to tell us if there are still any problems.

Feel free to close this bug in the mean time if you would like.
Comment 7 Beniamino Galvani 2020-03-26 08:24:49 UTC 
(In reply to Dan Sneddon from comment #6)
> Feel free to close this bug in the mean time if you would like.

Okay, thank you. Please reopen if needed.
Note You need to log in before you can comment on or make changes to this bug.