Bug 1799059

Summary: NIC wont come back online after removing ovs-bridge
Product: Red Hat Enterprise Linux 8 Reporter: omergi
Component: nmstateAssignee: Gris Ge <fge>
Status: CLOSED ERRATA QA Contact: Mingyu Shi <mshi>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.2CC: ferferna, jiji, jishi, network-qe, till
Target Milestone: rc   
Target Release: 8.2   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: nmstate-0.2.5-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-28 16:00:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description omergi 2020-02-06 14:45:35 UTC
Description of problem:
When removing ovs-bridge that one of its ports is the primary interface,
the interface wont come online.

when creating a different state to bring the interface online it is works.

Version-Release number of selected component (if applicable):

nmstate-0.2.2, 0.2.3
NetworkManager-1.22
NetworkManager-ovs-1.22
openvswitch-2.12.0-1.fc31.x86_64

How reproducible:
Create ovs-bridge on primary interface.
Remove the bridge and connect the interface in the same state.

Notice that if we apply two states, one for removing the bridge and the second for bringing the interface back online, because the primary interface is connected to the ovs-bridge, and we remove it, we wont be able to send the second state.
Eventually we end up with no connection to the VM/host/node.

Steps to Reproduce:
cat << EOF>state.yamk
---
interfaces:
- name: br
  type: ovs-bridge
  state: up
  bridge:
   port:
   - name: br0
   - name: enp7s0
- name: br0
  type: ovs-interface
  state: up
  mac-address: 52:54:00:XX:XX:XX 
  ipv4:
   enabled: true
   dhcp: true


EOF

nmstatectl set state.yaml

cat << EOF>state.yaml
---
interfaces:
- name: br
  type: ovs-bridge
  state: absent
- name: enp7s0
  type: ethernet
  state: up
  ipv4: 
   enabled: true
   dhcp: true


EOF

nmstatectl set revert-state.yaml


Actual results:
The ovs-bridge wont delete.
'ip a' output:
.
.
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:f9:f1:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.53/24 brd 192.168.122.255 scope global dynamic noprefixroute enp1s0
       valid_lft 2733sec preferred_lft 2733sec
    inet6 fe80::8d13:f913:f871:e5ed/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:2d:62:33 brd ff:ff:ff:ff:ff:ff
.
.
30: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 22:36:18:c5:c9:2c brd ff:ff:ff:ff:ff:ff
31: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:2d:62:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.213/24 brd 192.168.122.255 scope global dynamic noprefixroute br0
       valid_lft 3359sec preferred_lft 3359sec

'nmcli c s' output:
NAME             UUID                                  TYPE           DEVICE          
enp1s0           1076a848-d279-44d4-870c-8ade9a1f6729  ethernet       enp1s0          
br0              7b8f39d3-0518-4460-af14-a0d1c53b2754  ovs-interface  br0             
docker0          9aed7a06-9c4a-4fcc-b1d8-05e5841365db  bridge         docker0         
br               3dd38c01-21d0-4408-a0aa-91bcd6ad5618  ovs-bridge     br              
enp7s0           c18b347a-2397-4cf5-8f4c-ac58c1c8165e  ethernet       enp7s0          
ovs-port-br0     06926715-2977-40b0-9424-126a150cee5a  ovs-port       uKw0Hjz8ZRezfBs 
ovs-port-enp7s0  8c4fc6b9-9cef-422a-8a29-ef6a71c7cf12  ovs-port       ghQ2p08nanr2Koz 

Expected results:
ovs-bridge is deleted, enp7s0 in online.
'ip a' output:
.
.
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:f9:f1:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.53/24 brd 192.168.122.255 scope global dynamic noprefixroute enp1s0
       valid_lft 2459sec preferred_lft 2459sec
    inet6 fe80::8d13:f913:f871:e5ed/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:2d:62:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.213/24 brd 192.168.122.255 scope global dynamic noprefixroute enp7s0
       valid_lft 3594sec preferred_lft 3594sec
    inet6 fe80::5054:ff:fe2d:6233/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
.
.

'nmcli c s' output:
NAME     UUID                                  TYPE      DEVICE  
enp1s0   1076a848-d279-44d4-870c-8ade9a1f6729  ethernet  enp1s0  
enp7s0   460e1a09-04f3-4f1a-b2e2-a9db2c4e2461  ethernet  enp7s0  
docker0  9aed7a06-9c4a-4fcc-b1d8-05e5841365db  bridge    docker0  

Additional info:

'nmstatectl set revert-state.yaml' output:

2020-02-06 16:18:31,112 root         DEBUG    Checkpoint /org/freedesktop/NetworkManager/Checkpoint/10 created for all devices: 60
2020-02-06 16:18:31,112 root         DEBUG    Adding new interfaces: []
2020-02-06 16:18:31,113 root         DEBUG    Editing interfaces: ['enp7s0', 'br']
2020-02-06 16:18:31,117 root         DEBUG    Executing NM action: func=commit_changes_async
2020-02-06 16:18:31,121 root         DEBUG    Connection update succeeded: dev=enp7s0
2020-02-06 16:18:31,121 root         DEBUG    Executing NM action: func=_safe_modify_async
2020-02-06 16:18:31,125 root         DEBUG    Device reapply succeeded: dev=enp7s0
2020-02-06 16:18:31,126 root         DEBUG    Executing NM action: func=_safe_deactivate_async
2020-02-06 16:18:31,138 root         DEBUG    Connection deactivation succeeded on br
2020-02-06 16:18:31,138 root         DEBUG    Executing NM action: func=_safe_delete_async
2020-02-06 16:18:31,142 root         DEBUG    Connection deletion succeeded: dev=br
2020-02-06 16:18:31,142 root         DEBUG    Executing NM action: func=_safe_delete_device_async
2020-02-06 16:18:31,163 root         DEBUG    Interface is not real anymore: iface=br
2020-02-06 16:18:31,163 root         DEBUG    Executing NM action: func=_safe_deactivate_async
2020-02-06 16:18:31,185 root         DEBUG    Connection deactivation succeeded on ghQ2p08nanr2Koz
2020-02-06 16:18:31,185 root         DEBUG    Executing NM action: func=_safe_delete_async
2020-02-06 16:18:31,235 root         DEBUG    Connection deletion succeeded: dev=ghQ2p08nanr2Koz
2020-02-06 16:18:31,235 root         DEBUG    Executing NM action: func=_safe_deactivate_async
2020-02-06 16:18:31,235 root         DEBUG    Executing NM action: func=_safe_delete_async
2020-02-06 16:18:31,289 root         DEBUG    Connection deletion succeeded: dev=uKw0Hjz8ZRezfBs
2020-02-06 16:18:31,290 root         DEBUG    Executing NM action: func=_safe_deactivate_async
2020-02-06 16:18:31,290 root         DEBUG    Executing NM action: func=_safe_delete_async
2020-02-06 16:18:31,326 root         DEBUG    Connection deletion succeeded: dev=br0
2020-02-06 16:18:31,827 root         DEBUG    NM action queue exhausted, quiting mainloop
2020-02-06 16:18:31,861 root         DEBUG    Checkpoint /org/freedesktop/NetworkManager/Checkpoint/10 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/28'): dbus.UInt32(3), dbus.String('/org/freedesktop/NetworkManager/Devices/29'): dbus.UInt32(3), dbus.String('/org/freedesktop/NetworkManager/Devices/30'): dbus.UInt32(3), dbus.String('/org/freedesktop/NetworkManager/Devices/31'): dbus.UInt32(3), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0)}, signature=dbus.Signature('su'))
Traceback (most recent call last):
  File "/usr/bin/nmstatectl", line 11, in <module>
    load_entry_point('nmstate==0.2.3', 'console_scripts', 'nmstatectl')()
  File "/usr/lib/python3.7/site-packages/nmstatectl/nmstatectl.py", line 59, in main
    return args.func(args)
  File "/usr/lib/python3.7/site-packages/nmstatectl/nmstatectl.py", line 217, in apply
    statedata, args.verify, args.commit, args.timeout
  File "/usr/lib/python3.7/site-packages/nmstatectl/nmstatectl.py", line 241, in apply_state
    rollback_timeout=timeout,
  File "/usr/lib/python3.7/site-packages/libnmstate/nm/nmclient.py", line 98, in wrapped
    ret = func(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/libnmstate/netapplier.py", line 67, in apply
    state.State(desired_state), verify_change, commit, rollback_timeout
  File "/usr/lib/python3.7/site-packages/libnmstate/netapplier.py", line 152, in _apply_ifaces_state
    _verify_change(desired_state)
  File "/usr/lib/python3.7/site-packages/libnmstate/netapplier.py", line 196, in _verify_change
    desired_state.verify_interfaces(current_state)
  File "/usr/lib/python3.7/site-packages/libnmstate/state.py", line 329, in verify_interfaces
    self._assert_interfaces_equal(other_state)
  File "/usr/lib/python3.7/site-packages/libnmstate/state.py", line 651, in _assert_interfaces_equal
    current_state.interfaces[ifname],
libnmstate.error.NmstateVerificationError: 
desired
=======
---
name: enp7s0
type: ethernet
state: up
ipv4:
  address: []
  dhcp: true
  enabled: true
ipv6:
  enabled: false
mac-address: 52:54:00:2d:62:33
mtu: 1500

current
=======
---
name: enp7s0
type: ethernet
state: down
ethernet:
  auto-negotiation: true
  duplex: full
  speed: 1000
ipv4:
  enabled: false
ipv6:
  enabled: false
mac-address: 52:54:00:2d:62:33
mtu: 1500

difference
==========
--- desired
+++ current
@@ -1,11 +1,13 @@
 ---
 name: enp7s0
 type: ethernet
-state: up
+state: down
+ethernet:
+  auto-negotiation: true
+  duplex: full
+  speed: 1000
 ipv4:
-  address: []
-  dhcp: true
-  enabled: true
+  enabled: false
 ipv6:
   enabled: false
 mac-address: 52:54:00:2d:62:33

Comment 1 Gris Ge 2020-02-12 12:18:35 UTC
Upstream issue:

https://github.com/nmstate/nmstate/issues/779

Comment 7 errata-xmlrpc 2020-04-28 16:00:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1696