Bug 1903712

Description of problem:
When creating a linux-bridge, with one of the node's physical NICs as a port, the port of the OVS bridge (in our case - vxlan_sys_4789 of the ovs-system bridge) is deleted.


Version-Release number of selected component (if applicable):
nmstate-0.3.4-15.el8_3


How reproducible:
Always


Steps to Reproduce:
1. On an Openshift-SDN cluster - enter one of the worker nodes:
[cnv-qe-jenkins@myakove-hhsbc-executor]$ ssh core.1.244
Red Hat Enterprise Linux CoreOS 47.83.202011252347-0
  Part of OpenShift 4.7, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.7/architecture/architecture-rhcos.html

---
Last login: Wed Dec  2 16:21:48 2020 from 192.168.3.62
[systemd]
Failed Units: 1
  NetworkManager-wait-online.service
[core@myakove-hhsbc-worker-0-99vpj ~]$

2. Make sure you can view the vxlan_sys_4789 interface:
[core@myakove-hhsbc-worker-0-99vpj ~]$ ip link show dev vxlan_sys_4789
9: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 02:e7:b3:88:04:6c brd ff:ff:ff:ff:ff:ff

3. Back in the cluster - apply this NNCP:
$ cat << EOF | oc apply -f
apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br1test-nncp
spec:
  desiredState:
    interfaces:
    - bridge:
        options:
          stp:
            enabled: false
        port:
        - name: ens9
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        enabled: false
      name: br1test
      state: up
      type: linux-bridge
  nodeSelector:
    node-role.kubernetes.io/worker: 'myakove26-mh89k-worker-0-5tskb'
EOF

* Change port name (ens9) to whatever physical (secondary NIC) there is on your cluster node.
* Change the value of the nodeSelector to the name of the worker you logged-in in step #1.

4. Login to the node and search for the vxlan_sys_4789 interface again.


Actual results:
Interface is not found anymore.


Expected results:
Interface exists.


Additional info:
For CNV - it prevents creating VMs with multus (secondary NICs). Trying to create a VM fails with an error saying "no trace to route".

Some debug prints we added to nmstatectl show that wrongly it dectect ovs-interfaces as something to be deleted
Unable to use a TTY - input is not a terminal or the right kind of file
2020-12-02 15:13:38,628 root         DEBUG    Async action: Create checkpoint started
2020-12-02 15:13:38,637 root         DEBUG    Checkpoint None created for all devices
2020-12-02 15:13:38,638 root         DEBUG    Async action: Create checkpoint finished
2020-12-02 15:13:38,649 root         DEBUG    Async action: Delete device: br0 started
2020-12-02 15:13:38,650 root         DEBUG    Async action: Delete device: tun0 started
2020-12-02 15:13:38,650 root         DEBUG    Async action: Delete device: vxlan_sys_4789 started
2020-12-02 15:13:38,661 root         DEBUG    Interface is not real anymore: iface=br0
2020-12-02 15:13:38,661 root         DEBUG    Async action: Delete device: br0 finished
2020-12-02 15:13:38,662 root         DEBUG    Interface is not real anymore: iface=tun0
2020-12-02 15:13:38,662 root         DEBUG    Async action: Delete device: tun0 finished
2020-12-02 15:13:38,731 root         DEBUG    Interface is not real anymore: iface=vxlan_sys_4789
2020-12-02 15:13:38,731 root         DEBUG    Async action: Delete device: vxlan_sys_4789 finished
2020-12-02 15:13:38,772 root         DEBUG    Async action: Destroy checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 started
2020-12-02 15:13:38,776 root         DEBUG    Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 destroyed
2020-12-02 15:13:38,776 root         DEBUG    Async action: Destroy checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 finished
{'name': 'br0', 'type': 'ovs-interface', 'state': 'absent', 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}, 'mtu': 1400, 'mac-address': '3A:0C:AD:DA:B9:49', 'lldp': {'enabled': False}, 'bridge': {}}
{'name': 'tun0', 'type': 'ovs-interface', 'state': 'absent', 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}, 'mtu': 1400, 'mac-address': 'FE:8B:E3:38:7A:DC', 'lldp': {'enabled': False}}
{'name': 'vxlan_sys_4789', 'type': 'vxlan', 'state': 'absent', 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}, 'mtu': 65000, 'mac-address': 'B6:19:D4:BB:DD:95', 'lldp': {'enabled': False}, 'vxlan': {'id': 0, 'base-iface': '', 'remote': '', 'destination-port': 4789}}
Desired state applied: 
---
interfaces:
- name: br1test
  type: linux-bridge
  state: absent
  bridge:
    options:
      stp:
        enabled: false
    port:
    - name: ens9
  ipv4:
    enabled: false
    dhcp: false
  ipv6:
    enabled: false