Bug 1908363 - Applying NodeNetworkConfigurationPolicy for different NIC than default disables br-ex bridge and nodes lose connectivity
Summary: Applying NodeNetworkConfigurationPolicy for different NIC than default disabl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Networking
Version: 2.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 2.6.0
Assignee: Petr Horáček
QA Contact: Meni Yakove
URL:
Whiteboard:
Depends On: 1904889
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-16 14:13 UTC by Marius Cornea
Modified: 2021-03-10 11:23 UTC (History)
2 users (show)

Fixed In Version: kubernetes-nmstate-handler-container-v2.6.0-8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-10 11:22:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
NetworkManager journal (2.29 MB, text/plain)
2020-12-16 14:13 UTC, Marius Cornea
no flags Details
nmcli con output before applying NNCP (553 bytes, text/plain)
2020-12-16 14:14 UTC, Marius Cornea
no flags Details
nmcli con output after applying NNCP (830 bytes, text/plain)
2020-12-16 14:14 UTC, Marius Cornea
no flags Details
ip a output before applying NNCP (5.68 KB, text/plain)
2020-12-16 14:16 UTC, Marius Cornea
no flags Details
ip a output after applying NNCP (5.83 KB, text/plain)
2020-12-16 14:16 UTC, Marius Cornea
no flags Details
nmstate pod log from one of the affected nodes (3.45 MB, text/plain)
2020-12-16 14:17 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:0799 0 None None None 2021-03-10 11:23:19 UTC

Description Marius Cornea 2020-12-16 14:13:02 UTC
Created attachment 1739653 [details]
NetworkManager journal

Description of problem:

Applying NodeNetworkConfigurationPolicy for different NIC than default disables br-ex bridge and nodes lose connectivity.

Version-Release number of selected component (if applicable):
2.6.0
iib registry-proxy.engineering.redhat.com/rh-osbs/iib:31420

How reproducible:
100%

Steps to Reproduce:

1. Deploy OCP 4.7 via baremetal IPI flow

2. Deploy CNV 2.6

3. Create the following NodeNetworkConfigurationPolicy:

---
apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: bigip-bridges
spec:
  nodeSelector:
    node-role.kubernetes.io/worker-lb: ""
  desiredState:
    interfaces:
    - name: ens4f1.375
      type: vlan
      state: up
      vlan:
        base-iface: ens4f1
        id: 375

    - name: bigip-mgmt
      description: Linux bridge with ens4f1 vlan375 as a port!
      type: linux-bridge
      state: up
      bridge:
        options:
          stp:
            enabled: false
        port:
        - name: ens4f1.375
    - name: ens4f1.376
      type: vlan
      state: up
      vlan:
        base-iface: ens4f1
        id: 376

    - name: bigip-ha
      description: Linux bridge with ens4f1 vlan376 as a port!
      type: linux-bridge
      state: up
      bridge:
        options:
          stp:
            enabled: false
        port:
        - name: ens4f1.376


Actual results:
Configuration fails and br-ex bridge which provides external connectivity gets disabled.

Expected results:
Vlan interfaces and bridges get created and do not interrupt existing connections.

Additional info:

Attaching nmcli con, ip a output before and after applying NNCP, NetworkManager journal and nmstate-handler pod logs.

Comment 1 Marius Cornea 2020-12-16 14:14:03 UTC
Created attachment 1739654 [details]
nmcli con output before applying NNCP

Comment 2 Marius Cornea 2020-12-16 14:14:30 UTC
Created attachment 1739655 [details]
nmcli con output after applying NNCP

Comment 3 Marius Cornea 2020-12-16 14:16:24 UTC
Created attachment 1739656 [details]
ip a output before applying NNCP

Comment 4 Marius Cornea 2020-12-16 14:16:48 UTC
Created attachment 1739657 [details]
ip a output after applying NNCP

Comment 5 Marius Cornea 2020-12-16 14:17:18 UTC
Created attachment 1739658 [details]
nmstate pod log from one of the affected nodes

Comment 6 Petr Horáček 2020-12-16 14:19:12 UTC
Thanks for reporting this, Marius.

I believe this is caused by https://bugzilla.redhat.com/show_bug.cgi?id=1903712. The build of knmstate fixing this is on its way to 2.6. It will be available as soon as it finishes its run through our D/S pipeline.

Comment 7 Yossi Segev 2020-12-21 21:21:15 UTC
Verified on
OCP 4.7.0
CNV 2.6.0
nmstate-0.3.4-17.el8_3.noarch
kubernetes-nmstate-handler-rhel8@sha256:d7e6493bde74eaa72748086cf6a2abaa7abefdc1d78cb273df763a910dfb4d9

I applied the NodeNetworkConfigurationPolicy from the bug description, with changing the interface to a secondary interface available on my nodes (ens8).
Result:
1. NNCP successfully configured.
2. br-ex interface found in "ip a" output on one of the worker nodes (in the output provided in the bug original description - it was missing).
3. In the "nmcli c show" output of one of the worker nodes:
 ovs-if-br-ex appears as ovs-interface of br-ex
 ovs-port-br-ex appears as ovs-port of br-ex
(in the output provided in the bug original description they were missing).
In addition - the new VLAN interfaces also appear.

"nmcli c show" output (after applying the policy) is given below:

NAME              UUID                                  TYPE           DEVICE 
ovs-if-br-ex      caee1e5d-8045-4523-a85f-4ec1af0b8a28  ovs-interface  br-ex      
bigip-ha          2bf56523-aa17-42aa-b072-cfa2519b171f  bridge         bigip-ha   
bigip-mgmt        430eeca8-b54a-4f3e-a135-47e19a61cfea  bridge         bigip-mgmt 
br-ex             db52a338-97eb-41bc-a59a-2a022e32abba  ovs-bridge     br-ex      
ens8.375          23ad82cc-5645-4e7e-b9fe-0b10aae334d0  vlan           ens8.375   
ens8.376          3d434b47-4842-4e11-a36d-f101ae38b7b6  vlan           ens8.376   
ovs-if-phys0      8e7ebff2-4766-4a71-a630-f6aebaf1949a  ethernet       ens3       
ovs-port-br-ex    28c79aac-ae4e-4ab1-89ed-25ac1b28ad45  ovs-port       br-ex      
ovs-port-phys0    5a8d5ee6-bc63-4933-a019-8bed626046fc  ovs-port       ens3

Comment 8 Yossi Segev 2020-12-21 21:22:06 UTC
An additional note - the bug was verified on an OVN-Kubernetes cluster (the br-ex OVS bridge is not deployed on an OpenshiftSDN cluster).

Comment 11 errata-xmlrpc 2021-03-10 11:22:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 2.6.0 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0799


Note You need to log in before you can comment on or make changes to this bug.