This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
Bug 1885605 - It is not possible to reconfigure node's default interface using NodeNetworkConfigurationPolicy when OVN Kubernetes is used
Summary: It is not possible to reconfigure node's default interface using NodeNetworkC...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Networking
Version: 2.5.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.15.0
Assignee: Petr Horáček
QA Contact: Nir Rozen
URL:
Whiteboard:
Depends On: 1887456 1889309 1894139 1964307
Blocks: 1893160 1929782
TreeView+ depends on / blocked
 
Reported: 2020-10-06 13:41 UTC by Petr Horáček
Modified: 2023-12-14 16:07 UTC (History)
19 users (show)

Fixed In Version: 4.10
Doc Type: Known Issue
Doc Text:
If your OpenShift Container Platform cluster uses OVN-Kubernetes as the default Container Network Interface (CNI) provider, then you cannot attach a Linux bridge or bonding to the default interface of a host because of a change in the host network topology of OVN-Kubernetes.
Clone Of:
: 1893160 1906354 (view as bug list)
Environment:
Last Closed: 2023-12-14 16:07:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   CNV-10590 0 None None None 2023-12-14 16:07:03 UTC

Internal Links: 1887456 1901523 1929782 1964307

Description Petr Horáček 2020-10-06 13:41:32 UTC 
Description of problem:

One of the common network setups of OpenShift Virtualization is a VM connected to both the default OpenShift network and also to a secondary network providing flat L2 network. This flat L2 network is needed to allow users to do PXE booting and to maintain open connections during live migrations. This network may be also used to isolate part of the traffic or provide better performance.

This network is provided by configuring a Linux bridge on cluster nodes. This bridge is connected either directly or through a bonding to a physical NIC. Later, users can be connected to these bridges using Multus+Linux bridge CNI.

When users have only a single NIC available on their nodes, it is not uncommon to use this NIC for both the default network and for a secondary L2 network. In these cases we configure a linux bridge on top of the NIC and move the original IP of the NIC on top of the bridge. That way, this network can be still used by the default SDN while also utilized for secondary L2 connections.

With a recent change of the gateway mode in OVN Kubernetes, the default NIC of a host is now attached to an OVS bridge "br-ex". Due to that, it is not possible anymore to use this NIC under a Linux bridge.

This change in OVN Kubernetes breaks a common setup of OpenShift Virtualization and worse - it prevents customers who used this setup in previous versions from upgrading.


Version-Release number of selected component (if applicable):
OCP 4.6
CNV 2.5


How reproducible:
Always


Steps to Reproduce:
1. Deploy OCP and CNV, with OVN Kubernetes used as the network plugin
2. Create a Linux bridge on top of the default interface using:

cat <<EOF | oc apply -f -
apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br10-eth1
spec:
  desiredState:
    interfaces:
    - name: br10
      type: linux-bridge
      state: up
      ipv4:
        enabled: false
      bridge:
        port:
        - name: eth0
EOF


Actual results:
Configuration fails since the interface is already assigned to br-ex.


Expected results:
We are still able to configure bridges on top of the default network.
Comment 5 Petr Horáček 2020-11-26 12:40:44 UTC 
The OCP depency should land in 4.7 https://bugzilla.redhat.com/show_bug.cgi?id=1887456.

However, that won't be enough. We need a way to configure a veth pair between br-ex and our bridge. This will be allowed only once veth is supported on nmstate https://bugzilla.redhat.com/show_bug.cgi?id=1901523. This would be available only in 4.8.

Until then, the workaround may be to configure the veth using MCO.
Comment 6 Petr Horáček 2021-01-11 12:47:15 UTC 
This can be handled only once nmstate supports veth configuration. That will happen only with nmstate 1.0 / NetworkManager 1.30, i.e. in RHEL 8.4, CNV 2.8.
Comment 7 Quique Llorente 2021-05-24 10:41:39 UTC 
So veth is working fine at kubernetes-nmstdate for CNV 4.8 tested the following

kind: NodeNetworkConfigurationPolicy                                            
metadata:                                                                       
  name: cnv-veth
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  desiredState:
    interfaces:                                                                 
    - name: br-cnv
      type: linux-bridge                                                        
      state: up                                                                 
      ipv4:                                                                     
        dhcp: true                                                              
        enabled: true                                                           
      bridge:                                                                   
        options:                                                                
          stp:                                                                  
            enabled: false                                                      
        port:                                                                   
        - name: cnv-veth                      
    - ipv4:
        enabled: false
        dhcp: false
      ipv6:
        enabled: false
        autoconf: false
        dhcp: false
      name: cnv-veth
      state: up
      type: veth
      veth:
        peer: br-ext


I have try to startup a pair of VMs but I am not able to communicate between the two.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: br-cnv-100-l2
  annotations:
    k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br-cnv
spec:
  config: >
    {
        "cniVersion": "0.3.1",
        "name": "br-cnv-100-l2-config",
        "plugins": [
            {
                "type": "bridge",
                "bridge": "br-cnv",
                "vlan": 100,
                "ipam": {}
            },
            {
                "type": "tuning"
            }
        ]
    }
Comment 8 Quique Llorente 2021-05-24 11:26:27 UTC 
After removing the br-cnv and attach addresses to the cnv-veth looks like there is no connectivity between nodes, the arp table does not show the addresses

[core@worker-0-0 ~]$ ip a show cnv-veth
141: cnv-veth@br-ext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ca:20:3f:5e:63:09 brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.1/24 scope global cnv-veth
       valid_lft forever preferred_lft forever

[core@worker-0-1 ~]$ ip a show cnv-veth
1202: cnv-veth@br-ext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:19:a0:92:e1:93 brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.2/24 scope global cnv-veth
       valid_lft forever preferred_lft forever


core@worker-0-1
[core@worker-0-0 ~]$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.33.2                     (incomplete)                              cnv-veth

[core@worker-0-1 ~]$ arp -n |grep 33
192.168.33.1                     (incomplete)                              cnv-veth
Comment 9 Quique Llorente 2021-05-24 11:59:47 UTC 
Previous message are all wrong this is not he way you configure veths 

NMState fails configuring OVS 

libnmstate.error.NmstateDependencyError: openvswitch service is not started.

This is suppose to be fixed with https://github.com/nmstate/nmstate/commit/d67ba7f6b7c3036ceeca62545c95a1b10bdea513

apiVersion: nmstate.io/v1beta1                                                  
kind: NodeNetworkConfigurationPolicy                                            
metadata:                                                                       
  name: cnv-veth
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  desiredState:
    interfaces:                                                                 
    - name: br-cnv
      type: linux-bridge                                                        
      state: up                                                                 
      ipv4:                                                                     
        dhcp: true                                                              
        enabled: true                                                           
      bridge:                                                                   
        options:                                                                
          stp:                                                                  
            enabled: false                                                      
        port:                                                                   
        - name: cnv-veth
    - name: br-ex
      type: ovs-bridge                                                        
      bridge:      
        port:                                                                   
        - name: ovn-veth
    - ipv4:
        enabled: false
        dhcp: false
      ipv6:
        enabled: false
        autoconf: false
        dhcp: false
      name: cnv-veth
      state: up
      type: veth
      veth:
        peer: ovn-veth
Comment 10 Quique Llorente 2021-05-24 14:14:06 UTC 
Doing adding the ovs ports manually works fine and we can ping between VMs at different nodes.
Comment 11 Quique Llorente 2021-05-25 07:41:00 UTC 
Created a bz on nmstate https://bugzilla.redhat.com/show_bug.cgi?id=1964307
Comment 13 Fernando F. Mancera 2021-08-02 11:07:53 UTC 
Please use the following state instead:

interfaces:
- name: br-ex
  type: ovs-bridge
  state: up
  bridge:
    options:
      fail-mode: ''
      mcast-snooping-enable: false
      rstp: false
      stp: false
    port:
    - name: br-ex
    - name: enp5s0
    - name: ovn-veth
  lldp:
    enabled: false
- ipv4:
    enabled: false
    dhcp: false
  ipv6:
    enabled: false
    autoconf: false
    dhcp: false
  name: cnv-veth
  state: up
  type: veth
  veth:
    peer: ovn-veth
- ipv4:
    enabled: false
    dhcp: false
  ipv6:
    enabled: false
    autoconf: false
    dhcp: false
  name: ovn-veth
  state: up
  type: veth
  veth:
    peer: cnv-veth

The both veths need to be explicitly up.
Comment 14 Quique Llorente 2021-09-03 06:47:22 UTC 
> Please use the following state instead:
> 
> interfaces:
> - name: br-ex
>   type: ovs-bridge
>   state: up
>   bridge:
>     options:
>       fail-mode: ''
>       mcast-snooping-enable: false
>       rstp: false
>       stp: false
>     port:
>     - name: br-ex
>     - name: enp5s0
>     - name: ovn-veth
>   lldp:
>     enabled: false
> - ipv4:
>     enabled: false
>     dhcp: false
>   ipv6:
>     enabled: false
>     autoconf: false
>     dhcp: false
>   name: cnv-veth
>   state: up
>   type: veth
>   veth:
>     peer: ovn-veth
> - ipv4:
>     enabled: false
>     dhcp: false
>   ipv6:
>     enabled: false
>     autoconf: false
>     dhcp: false
>   name: ovn-veth
>   state: up
>   type: veth
>   veth:
>     peer: cnv-veth
> 
> The both veths need to be explicitly up.

Testing it with OCP 4.9 (nmstate RPMs 1.0.2-14) is unable to add the port to the bridge 

desired                                                                         
=======                                                                         
---                                                                             
name: br-ex                                                                     
type: ovs-bridge                                                                
state: up                                                                       
bridge:                                                                         
  options:                                                                      
    fail-mode: ''                                                               
    mcast-snooping-enable: false                                                
    rstp: false                                                                 
    stp: false                                                                  
  port:                                                                         
  - name: br-ex                                                                 
  - name: enp0s4                                                                
  - name: ovn-veth                                                              
lldp:                                                                           
  enabled: false                                                                
                                                                                
current                                                                         
=======                                                                         
---                                                                             
name: br-ex                                                                     
type: ovs-bridge                                                                
state:                                                                          
      up                                                                        
bridge:                                                                         
  options:                                                                      
    fail-mode: ''                                                               
    mcast-snooping-enable: false                                                
    rstp: false                                                                 
    stp: false                                                                  
  port:                                                                         
  - name: ovn-veth                                                              
lldp:                                                                           
  enabled:                                                                      
      false                                                                     
ovs-db:                                                                         
  external_ids: {}                                                              
                                                                                
difference                                                                      
==========                                                                      
--- desired                                                                     
+++                                                                             
      current                                                                   
@@ -9,8 +9,8 @@                                                                 
     rstp: false                                                                
     stp: false                                                                 
   port:                                                                        
-  - name: br-ex                                                                
-  - name: enp0s4                                                               
   - name: ovn-veth                                                             
 lldp:                                                                          
   enabled: false
Comment 21 Quique Llorente 2021-09-10 08:19:51 UTC 
So looks like the issue is that we are not mounting the ovs socket, having the NM dbus socket is not enough.
Comment 22 Quique Llorente 2021-09-10 09:24:24 UTC 
u/s fix https://github.com/nmstate/kubernetes-nmstate/pull/829
Comment 23 Quique Llorente 2021-09-10 10:25:48 UTC 
u/s fix CNAO https://github.com/kubevirt/cluster-network-addons-operator/pull/1008
Comment 31 Radim Hrazdil 2021-11-05 11:31:22 UTC 
Managed to verify this scenario with the desiredState described in comment 20.

A new issue has was identified: nmstate renames ovs ports by prepending 'ovs-port-' to each new and existing port of an ovs-bridge.
This causes ovnkube-node pods to crash.
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2019457


Desired stat that also creates a linux bridge connected to ovs-bridge br-ex via veth pair:
interfaces:
- name: br-ex
  type: ovs-interface
  state: up
- name: eno1
  type: ethernet
  state: up
- name: br101
  description: Linux bridge with cnv-veth as port
  type: linux-bridge
  state: up
  ipv4:
    dhcp: false
    enabled: false
  ipv6:
    enabled: false
    autoconf: false
    dhcp: false
  bridge:
    options:
      stp:
        enabled: false
    port:
    - name: cnv-veth
- name: br-ex
  type: ovs-bridge
  state: up
  bridge:
    options:
      fail-mode: ''
      mcast-snooping-enable: false
      rstp: false
      stp: false
    port:
    - name: br-ex
    - name: eno1
    - name: ovn-veth
  lldp:
    enabled: false
- name: cnv-veth
  type: veth
  state: up
  ipv4:
    enabled: false
    dhcp: false
  ipv6:
    enabled: false
    autoconf: false
    dhcp: false
  veth:
    peer: ovn-veth
- name: ovn-veth
  type: veth
  state: up
  ipv4:
    enabled: false
    dhcp: false
  ipv6:
    enabled: false
    autoconf: false
    dhcp: false
  veth:
    peer: cnv-veth



Verified connectivity between Virtual Machines connected to the linux-bridge br101 works.
Used the following NAD:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: a-bridge-network
  annotations:
    k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br101
spec:
  config: '{
    "cniVersion": "0.3.1",
    "name": "a-bridge-network", 
    "type": "cnv-bridge", 
    "bridge": "br101", 
    "vlan": 100 
  }'
Comment 32 Petr Horáček 2022-01-20 12:29:56 UTC 
https://bugzilla.redhat.com/show_bug.cgi?id=2019457 will land in RHEL only after the GA of 4.10. Therefore we are deferring this BZ to 4.11.
Comment 33 ctomasko 2022-02-07 19:58:33 UTC 
Because the fix is deferred to 4.11, we will keep this as a known issue in the CNV 4.10 release notes. I've updated the release notes field to describe the known issue. The docs team will pull the doc text field for the CNV 4.10 release notes.


For future reference, here is the draft of the release note text for the enhancement. The text has not yet been verified.

Feature: 
reuse primary interface of a node for OVNKubernetes and L2 traffic
Reason: 
Customers may have a singe interface that they can reuse for both types of traffic: management and L2.
Result: 
The cluster admin may set an NNCP, as described in the documentation, that connects a Linux bridge br0 to OVNKubernetes-managed br-ex. Admin can then define NAD with br0 and connect VMs to it.
Comment 34 ctomasko 2022-03-15 22:32:58 UTC 
Added Release note > known issue

If your OpenShift Container Platform cluster uses OVN-Kubernetes as the default Container Network Interface (CNI) provider, you cannot attach a Linux bridge or bonding to the default interface of a host because of a change in the host network topology of OVN-Kubernetes.

As a workaround, you can use a secondary network interface connected to your host, or switch to the OpenShift SDN default CNI provider. (BZ#1885605)

https://github.com/openshift/openshift-docs/pull/42530
https://deploy-preview-42530--osdocs.netlify.app/openshift-enterprise/latest/virt/virt-4-10-release-notes#virt-4-10-known-issues

Future link: After the OpenShift Virtualization 4.10 releases, you can find the release notes here: https://docs.openshift.com/container-platform/4.10/virt/virt-4-10-release-notes.html
or on the portal,
https://access.redhat.com/documentation/en-us/openshift_container_platform/4.10
Comment 35 Petr Horáček 2022-04-14 12:55:23 UTC 
Moving to 4.12 due to capacity
Comment 36 Novonil Choudhuri 2022-08-08 16:51:39 UTC 
Can we use a secondary interface for VLAN when using OVN ?
Comment 37 Petr Horáček 2022-08-09 13:34:17 UTC 
(In reply to Novonil Choudhuri from comment #36)
> Can we use a secondary interface for VLAN when using OVN ?

Not sure I fully understand your question, could you expand on it?
Comment 38 Novonil Choudhuri 2022-08-09 15:50:20 UTC 
Can we use this [1] while using OVN and a secondary network ? I understand we cannot use the default interface due to this bug. 

[1] https://docs.openshift.com/container-platform/4.10/virt/node_network/virt-updating-node-network-config.html#virt-example-vlan-nncp_virt-updating-node-network-config
Comment 39 Petr Horáček 2022-08-09 16:04:03 UTC 
Gotcha. I believe you should be able to configure a VLAN on the interface used by br-ex. And then connect a bridge to the VLAN, to be used by VMs/Pods.
Comment 40 Novonil Choudhuri 2022-08-09 16:07:55 UTC 
Do we need a second network interface to do this or we can use the default network interface ?
Comment 41 Petr Horáček 2022-08-10 10:31:11 UTC 
On the default interface, you should be able to create VLAN interfaces and use them for additional bridges.

When using a dedicated interface, it is preferred to connect the bridge directly to the interface, without VLAN interface in between.
Comment 42 Petr Horáček 2022-08-11 09:40:47 UTC 
I'm moving the target to future. Instead of working around OVN, we'd like to solve this issue by supporting secondary networks in OVN Kubernetes. This BZ will be closed once we have a satisfying solution.

Until then, the default interface can be shared with OVN's br-ex by creating a VLAN interface on it:

[eth0] --- [br-ex] --- {default pod network}
    \--- [eth0.10] --- [br10] --- {additional network}

That can be accomplished by using kubernetes-nmstate: https://docs.openshift.com/container-platform/4.10/virt/node_network/virt-updating-node-network-config.html#virt-example-vlan-nncp_virt-updating-node-network-config
Comment 43 Novonil Choudhuri 2022-08-11 17:07:34 UTC 
Thank you that answers my questions.
Comment 53 Petr Horáček 2023-06-29 12:33:53 UTC 
This should be resolved with https://issues.redhat.com/browse/CNV-24261 in 4.15.
Comment 54 Yossi Segev 2023-07-02 07:48:42 UTC 
@phoracek Hi,
Based on the epic this fix depends on (https://issues.redhat.com/browse/CNV-24261), I assume it cannot be back-ported to any CNV version earlier than 4.15.
Please keep me honest here, so we can decide how to handle it in the tests.
Thank you.
Comment 55 Petr Horáček 2023-07-14 11:23:02 UTC 
Hey. Correct, this won't be backported.

The fix will be available in OCP 4.14 as a dev preview. We will productize this https://issues.redhat.com/browse/CNV-24261 in 4.15 and will be finally able to close this BZ.
Note You need to log in before you can comment on or make changes to this bug.