1799059 – NIC wont come back online after removing ovs-bridge

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1799059 - NIC wont come back online after removing ovs-bridge

Summary: NIC wont come back online after removing ovs-bridge

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	nmstate
Sub Component:
Version:	8.2
Hardware:	x86_64
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	8.2
Assignee:	Gris Ge
QA Contact:	Mingyu Shi
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-02-06 14:45 UTC by omergi
Modified:	2022-10-11 15:42 UTC (History)
CC List:	5 users (show)
Fixed In Version:	nmstate-0.2.5-1.el8
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-04-28 16:00:37 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	nmstate nmstate issues 779	None	closed	Cannot set interface up when certain interface used in OVS bridge which about to be deleted.	2020-03-26 08:30:34 UTC
Red Hat Issue Tracker	RHELPLAN-37489	None	None	None	2022-10-11 15:42:29 UTC
Red Hat Product Errata	RHBA-2020:1696	None	None	None	2020-04-28 16:00:51 UTC

Description omergi 2020-02-06 14:45:35 UTC

Description of problem:
When removing ovs-bridge that one of its ports is the primary interface,
the interface wont come online.

when creating a different state to bring the interface online it is works.

Version-Release number of selected component (if applicable):

nmstate-0.2.2, 0.2.3
NetworkManager-1.22
NetworkManager-ovs-1.22
openvswitch-2.12.0-1.fc31.x86_64

How reproducible:
Create ovs-bridge on primary interface.
Remove the bridge and connect the interface in the same state.

Notice that if we apply two states, one for removing the bridge and the second for bringing the interface back online, because the primary interface is connected to the ovs-bridge, and we remove it, we wont be able to send the second state.
Eventually we end up with no connection to the VM/host/node.

Steps to Reproduce:
cat << EOF>state.yamk
---
interfaces:
- name: br
  type: ovs-bridge
  state: up
  bridge:
   port:
   - name: br0
   - name: enp7s0
- name: br0
  type: ovs-interface
  state: up
  mac-address: 52:54:00:XX:XX:XX 
  ipv4:
   enabled: true
   dhcp: true


EOF

nmstatectl set state.yaml

cat << EOF>state.yaml
---
interfaces:
- name: br
  type: ovs-bridge
  state: absent
- name: enp7s0
  type: ethernet
  state: up
  ipv4: 
   enabled: true
   dhcp: true


EOF

nmstatectl set revert-state.yaml


Actual results:
The ovs-bridge wont delete.
'ip a' output:
.
.
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:f9:f1:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.53/24 brd 192.168.122.255 scope global dynamic noprefixroute enp1s0
       valid_lft 2733sec preferred_lft 2733sec
    inet6 fe80::8d13:f913:f871:e5ed/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:2d:62:33 brd ff:ff:ff:ff:ff:ff
.
.
30: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 22:36:18:c5:c9:2c brd ff:ff:ff:ff:ff:ff
31: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:2d:62:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.213/24 brd 192.168.122.255 scope global dynamic noprefixroute br0
       valid_lft 3359sec preferred_lft 3359sec

'nmcli c s' output:
NAME             UUID                                  TYPE           DEVICE          
enp1s0           1076a848-d279-44d4-870c-8ade9a1f6729  ethernet       enp1s0          
br0              7b8f39d3-0518-4460-af14-a0d1c53b2754  ovs-interface  br0             
docker0          9aed7a06-9c4a-4fcc-b1d8-05e5841365db  bridge         docker0         
br               3dd38c01-21d0-4408-a0aa-91bcd6ad5618  ovs-bridge     br              
enp7s0           c18b347a-2397-4cf5-8f4c-ac58c1c8165e  ethernet       enp7s0          
ovs-port-br0     06926715-2977-40b0-9424-126a150cee5a  ovs-port       uKw0Hjz8ZRezfBs 
ovs-port-enp7s0  8c4fc6b9-9cef-422a-8a29-ef6a71c7cf12  ovs-port       ghQ2p08nanr2Koz 

Expected results:
ovs-bridge is deleted, enp7s0 in online.
'ip a' output:
.
.
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:f9:f1:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.53/24 brd 192.168.122.255 scope global dynamic noprefixroute enp1s0
       valid_lft 2459sec preferred_lft 2459sec
    inet6 fe80::8d13:f913:f871:e5ed/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:2d:62:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.213/24 brd 192.168.122.255 scope global dynamic noprefixroute enp7s0
       valid_lft 3594sec preferred_lft 3594sec
    inet6 fe80::5054:ff:fe2d:6233/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
.
.

'nmcli c s' output:
NAME     UUID                                  TYPE      DEVICE  
enp1s0   1076a848-d279-44d4-870c-8ade9a1f6729  ethernet  enp1s0  
enp7s0   460e1a09-04f3-4f1a-b2e2-a9db2c4e2461  ethernet  enp7s0  
docker0  9aed7a06-9c4a-4fcc-b1d8-05e5841365db  bridge    docker0  

Additional info:

'nmstatectl set revert-state.yaml' output:

2020-02-06 16:18:31,112 root         DEBUG    Checkpoint /org/freedesktop/NetworkManager/Checkpoint/10 created for all devices: 60
2020-02-06 16:18:31,112 root         DEBUG    Adding new interfaces: []
2020-02-06 16:18:31,113 root         DEBUG    Editing interfaces: ['enp7s0', 'br']
2020-02-06 16:18:31,117 root         DEBUG    Executing NM action: func=commit_changes_async
2020-02-06 16:18:31,121 root         DEBUG    Connection update succeeded: dev=enp7s0
2020-02-06 16:18:31,121 root         DEBUG    Executing NM action: func=_safe_modify_async
2020-02-06 16:18:31,125 root         DEBUG    Device reapply succeeded: dev=enp7s0
2020-02-06 16:18:31,126 root         DEBUG    Executing NM action: func=_safe_deactivate_async
2020-02-06 16:18:31,138 root         DEBUG    Connection deactivation succeeded on br
2020-02-06 16:18:31,138 root         DEBUG    Executing NM action: func=_safe_delete_async
2020-02-06 16:18:31,142 root         DEBUG    Connection deletion succeeded: dev=br
2020-02-06 16:18:31,142 root         DEBUG    Executing NM action: func=_safe_delete_device_async
2020-02-06 16:18:31,163 root         DEBUG    Interface is not real anymore: iface=br
2020-02-06 16:18:31,163 root         DEBUG    Executing NM action: func=_safe_deactivate_async
2020-02-06 16:18:31,185 root         DEBUG    Connection deactivation succeeded on ghQ2p08nanr2Koz
2020-02-06 16:18:31,185 root         DEBUG    Executing NM action: func=_safe_delete_async
2020-02-06 16:18:31,235 root         DEBUG    Connection deletion succeeded: dev=ghQ2p08nanr2Koz
2020-02-06 16:18:31,235 root         DEBUG    Executing NM action: func=_safe_deactivate_async
2020-02-06 16:18:31,235 root         DEBUG    Executing NM action: func=_safe_delete_async
2020-02-06 16:18:31,289 root         DEBUG    Connection deletion succeeded: dev=uKw0Hjz8ZRezfBs
2020-02-06 16:18:31,290 root         DEBUG    Executing NM action: func=_safe_deactivate_async
2020-02-06 16:18:31,290 root         DEBUG    Executing NM action: func=_safe_delete_async
2020-02-06 16:18:31,326 root         DEBUG    Connection deletion succeeded: dev=br0
2020-02-06 16:18:31,827 root         DEBUG    NM action queue exhausted, quiting mainloop
2020-02-06 16:18:31,861 root         DEBUG    Checkpoint /org/freedesktop/NetworkManager/Checkpoint/10 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/28'): dbus.UInt32(3), dbus.String('/org/freedesktop/NetworkManager/Devices/29'): dbus.UInt32(3), dbus.String('/org/freedesktop/NetworkManager/Devices/30'): dbus.UInt32(3), dbus.String('/org/freedesktop/NetworkManager/Devices/31'): dbus.UInt32(3), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0)}, signature=dbus.Signature('su'))
Traceback (most recent call last):
  File "/usr/bin/nmstatectl", line 11, in <module>
    load_entry_point('nmstate==0.2.3', 'console_scripts', 'nmstatectl')()
  File "/usr/lib/python3.7/site-packages/nmstatectl/nmstatectl.py", line 59, in main
    return args.func(args)
  File "/usr/lib/python3.7/site-packages/nmstatectl/nmstatectl.py", line 217, in apply
    statedata, args.verify, args.commit, args.timeout
  File "/usr/lib/python3.7/site-packages/nmstatectl/nmstatectl.py", line 241, in apply_state
    rollback_timeout=timeout,
  File "/usr/lib/python3.7/site-packages/libnmstate/nm/nmclient.py", line 98, in wrapped
    ret = func(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/libnmstate/netapplier.py", line 67, in apply
    state.State(desired_state), verify_change, commit, rollback_timeout
  File "/usr/lib/python3.7/site-packages/libnmstate/netapplier.py", line 152, in _apply_ifaces_state
    _verify_change(desired_state)
  File "/usr/lib/python3.7/site-packages/libnmstate/netapplier.py", line 196, in _verify_change
    desired_state.verify_interfaces(current_state)
  File "/usr/lib/python3.7/site-packages/libnmstate/state.py", line 329, in verify_interfaces
    self._assert_interfaces_equal(other_state)
  File "/usr/lib/python3.7/site-packages/libnmstate/state.py", line 651, in _assert_interfaces_equal
    current_state.interfaces[ifname],
libnmstate.error.NmstateVerificationError: 
desired
=======
---
name: enp7s0
type: ethernet
state: up
ipv4:
  address: []
  dhcp: true
  enabled: true
ipv6:
  enabled: false
mac-address: 52:54:00:2d:62:33
mtu: 1500

current
=======
---
name: enp7s0
type: ethernet
state: down
ethernet:
  auto-negotiation: true
  duplex: full
  speed: 1000
ipv4:
  enabled: false
ipv6:
  enabled: false
mac-address: 52:54:00:2d:62:33
mtu: 1500

difference
==========
--- desired
+++ current
@@ -1,11 +1,13 @@
 ---
 name: enp7s0
 type: ethernet
-state: up
+state: down
+ethernet:
+  auto-negotiation: true
+  duplex: full
+  speed: 1000
 ipv4:
-  address: []
-  dhcp: true
-  enabled: true
+  enabled: false
 ipv6:
   enabled: false
 mac-address: 52:54:00:2d:62:33

Comment 1 Gris Ge 2020-02-12 12:18:35 UTC

Upstream issue:

https://github.com/nmstate/nmstate/issues/779

Comment 7 errata-xmlrpc 2020-04-28 16:00:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1696

Note You need to log in before you can comment on or make changes to this bug.