Created attachment 1718964 [details] nmstate-handler-xvhf2.log Description of problem: On a bare metal IPI deployment creating a NodeNetworkConfigurationPolicy to configure tagged vlans over the external interface fails. I suspect this is related to the external interface getting automatically added to an OVS br-ex bridge in 4.6. Nevertheless the same configuration used to work in 4.5 so I'd like to clarify if there are any limitations introduced in 4.6 by the br-ex OVS bridge and if there are any ways to overcome them. Version-Release number of selected component (if applicable): 2.5.0 How reproducible: 100% Steps to Reproduce: 1. Create the following NodeNetworkConfigurationPolicy: --- apiVersion: nmstate.io/v1alpha1 kind: NodeNetworkConfigurationPolicy metadata: name: bigip-bridges spec: nodeSelector: node-role.kubernetes.io/worker-lb: "" desiredState: interfaces: - name: eno2np1.375 type: vlan state: up vlan: base-iface: eno2np1 id: 375 - name: bigip-mgmt description: Linux bridge with eno2np1 vlan375 as a port! type: linux-bridge state: up bridge: options: stp: enabled: false port: - name: eno2np1.375 - name: eno2np1.376 type: vlan state: up vlan: base-iface: eno2np1 id: 376 - name: bigip-ha description: Linux bridge with eno2np1 vlan376 as a port! type: linux-bridge state: up bridge: options: stp: enabled: false port: - name: eno2np1.376 Actual results: FailedToConfigure Expected results: Configuration succeeds. Additional info: Attaching nmstate-handler log from one of the nodes.
NIC status from one of the workers: [root@openshift-worker-7 core]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether bc:97:e1:69:a1:a0 brd ff:ff:ff:ff:ff:ff inet6 fe80::be97:e1ff:fe69:a1a0/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000 link/ether bc:97:e1:69:a1:a1 brd ff:ff:ff:ff:ff:ff 4: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 1c:34:da:75:bc:76 brd ff:ff:ff:ff:ff:ff inet6 fe80::1e34:daff:fe75:bc76/64 scope link noprefixroute valid_lft forever preferred_lft forever 5: ens4f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 1c:34:da:75:bc:77 brd ff:ff:ff:ff:ff:ff inet6 fe80::1e34:daff:fe75:bc77/64 scope link noprefixroute valid_lft forever preferred_lft forever 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 56:80:34:f7:58:db brd ff:ff:ff:ff:ff:ff 7: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 link/ether ae:e3:2f:1b:32:4c brd ff:ff:ff:ff:ff:ff inet6 fe80::ace3:2fff:fe1b:324c/64 scope link valid_lft forever preferred_lft forever 8: ovn-k8s-mp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 22:47:7d:f7:ad:dd brd ff:ff:ff:ff:ff:ff inet 10.129.4.2/23 brd 10.129.5.255 scope global ovn-k8s-mp0 valid_lft forever preferred_lft forever inet6 fe80::2047:7dff:fef7:addd/64 scope link valid_lft forever preferred_lft forever 9: br-int: <BROADCAST,MULTICAST> mtu 1400 qdisc noop state DOWN group default qlen 1000 link/ether 06:2d:ea:f5:b7:4d brd ff:ff:ff:ff:ff:ff 10: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 link/ether fe:d9:3a:35:c9:ac brd ff:ff:ff:ff:ff:ff 11: br-ext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether b6:6a:7a:b5:01:44 brd ff:ff:ff:ff:ff:ff 12: ovn-k8s-gw0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 0a:58:a9:fe:00:01 brd ff:ff:ff:ff:ff:ff inet 169.254.0.1/20 brd 169.254.15.255 scope global ovn-k8s-gw0 valid_lft forever preferred_lft forever inet6 fe80::858:a9ff:fefe:1/64 scope link valid_lft forever preferred_lft forever 13: br-local: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether e2:23:2b:67:ab:4e brd ff:ff:ff:ff:ff:ff inet6 fe80::e023:2bff:fe67:ab4e/64 scope link valid_lft forever preferred_lft forever 14: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether bc:97:e1:69:a1:a1 brd ff:ff:ff:ff:ff:ff inet 10.46.57.30/24 brd 10.46.57.255 scope global dynamic noprefixroute br-ex valid_lft 39269sec preferred_lft 39269sec inet6 2620:52:0:2e39:ea0:53e1:f96f:c2ff/64 scope global dynamic noprefixroute valid_lft 2591839sec preferred_lft 604639sec inet6 fe80::c0a5:eda6:751d:b329/64 scope link noprefixroute valid_lft forever preferred_lft forever 15: ddeeaf2d4d1c2f7@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default link/ether 66:e3:2c:76:08:fd brd ff:ff:ff:ff:ff:ff link-netns 0731db1a-5fed-4451-8198-07c185b38cf1 inet6 fe80::64e3:2cff:fe76:8fd/64 scope link valid_lft forever preferred_lft forever 16: fef9763f672c7de@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default link/ether be:3a:54:40:e3:67 brd ff:ff:ff:ff:ff:ff link-netns 6cd8069f-9fbf-438b-b120-ccaa6abdd351 inet6 fe80::bc3a:54ff:fe40:e367/64 scope link valid_lft forever preferred_lft forever 17: 00db64dbec31d02@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default link/ether b2:c9:72:95:28:c3 brd ff:ff:ff:ff:ff:ff link-netns 52478bc1-aab4-43a6-b247-23a0a434bbfd inet6 fe80::b0c9:72ff:fe95:28c3/64 scope link valid_lft forever preferred_lft forever 18: b9dbe100229f649@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default link/ether 1e:3d:ce:4e:7a:6b brd ff:ff:ff:ff:ff:ff link-netns 41f20e83-1aea-4721-870e-99bc5b9c5b8c inet6 fe80::1c3d:ceff:fe4e:7a6b/64 scope link valid_lft forever preferred_lft forever 19: 23ce0b00a3f561a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default link/ether 8e:02:dc:8d:ec:a7 brd ff:ff:ff:ff:ff:ff link-netns 48d21d86-8f00-46b8-807e-109261c7b430 inet6 fe80::8c02:dcff:fe8d:eca7/64 scope link valid_lft forever preferred_lft forever 27: 56cf1bf46d64b45@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default link/ether be:d5:76:b0:bc:1c brd ff:ff:ff:ff:ff:ff link-netns 15acad40-1b6c-446c-88f7-250b5add2188 inet6 fe80::bcd5:76ff:feb0:bc1c/64 scope link valid_lft forever preferred_lft forever 30: ae5e9fae5c07f59@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default link/ether 5e:d0:c9:84:3d:f3 brd ff:ff:ff:ff:ff:ff link-netns b9d70228-84ca-4316-a584-b55ed41f5ee8 inet6 fe80::5cd0:c9ff:fe84:3df3/64 scope link valid_lft forever preferred_lft forever 32: 98c59bc0ab6c508@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether ea:77:f5:f0:ed:17 brd ff:ff:ff:ff:ff:ff link-netns debc2dc4-ed37-4d14-800b-b82346b4189c inet6 fe80::e877:f5ff:fef0:ed17/64 scope link valid_lft forever preferred_lft forever [root@openshift-worker-7 core]# ovs-vsctl show ee497a89-be46-4436-8d60-0342905b9328 Bridge br-ex Port br-ex Interface br-ex type: internal Port patch-br-ex_openshift-worker-7-to-br-int Interface patch-br-ex_openshift-worker-7-to-br-int type: patch options: {peer=patch-br-int-to-br-ex_openshift-worker-7} Port eno2np1 Interface eno2np1 type: system Bridge br-local Port patch-lnet-node_local_switch-to-br-int Interface patch-lnet-node_local_switch-to-br-int type: patch options: {peer=patch-br-int-to-lnet-node_local_switch} Port br-local Interface br-local type: internal Port ovn-k8s-gw0 Interface ovn-k8s-gw0 type: internal Bridge br-ext fail_mode: secure Port ext-vxlan Interface ext-vxlan type: vxlan options: {dst_port="4789", key=flow, remote_ip=flow} Port br-ext Interface br-ext type: internal Port ext Interface ext type: patch options: {peer=int} Bridge br-int fail_mode: secure Port ovn-efa4ae-0 Interface ovn-efa4ae-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.22"} Port patch-br-int-to-lnet-node_local_switch Interface patch-br-int-to-lnet-node_local_switch type: patch options: {peer=patch-lnet-node_local_switch-to-br-int} Port b9dbe100229f649 Interface b9dbe100229f649 Port ddeeaf2d4d1c2f7 Interface ddeeaf2d4d1c2f7 Port patch-br-int-to-br-ex_openshift-worker-7 Interface patch-br-int-to-br-ex_openshift-worker-7 type: patch options: {peer=patch-br-ex_openshift-worker-7-to-br-int} Port ovn-6c9ec6-0 Interface ovn-6c9ec6-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.29"} Port br-int Interface br-int type: internal Port "23ce0b00a3f561a" Interface "23ce0b00a3f561a" Port "00db64dbec31d02" Interface "00db64dbec31d02" Port ovn-57903f-0 Interface ovn-57903f-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.21"} Port ovn-ed0f0f-0 Interface ovn-ed0f0f-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.25"} Port ovn-c5dbad-0 Interface ovn-c5dbad-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.26"} Port ovn-0dabfc-0 Interface ovn-0dabfc-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.20"} Port ovn-k8s-mp0 Interface ovn-k8s-mp0 type: internal Port int Interface int type: patch options: {peer=ext} Port ovn-8e4f87-0 Interface ovn-8e4f87-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.24"} Port fef9763f672c7de Interface fef9763f672c7de Port ovn-738f81-0 Interface ovn-738f81-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.23"} Port "56cf1bf46d64b45" Interface "56cf1bf46d64b45" Port ovn-9c1d11-0 Interface ovn-9c1d11-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.28"} Port ovn-af3fa5-0 Interface ovn-af3fa5-0 type: geneve options: {csum="true", key=flow, remote_ip="10.46.57.27"} Port ae5e9fae5c07f59 Interface ae5e9fae5c07f59 ovs_version: "2.13.2"
Setting as a blocker since it prevents us from using nmstate with OVN Kubernetes. @Meni, are we reproducing this on tier2?
Per my initial investigation on a local setup, the problem is that `br-ex` is **managed** by NetworkManager and nmstate has a problem reporting it [1]. Here is the `nmcli device` report: ``` DEVICE TYPE STATE CONNECTION br-ex ovs-interface connected ovs-if-br-ex enp1s0 ethernet connected Wired Connection enp2s0 ethernet connected ovs-if-phys0 br-ex ovs-bridge connected br-ex br-ex ovs-port connected ovs-port-br-ex enp2s0 ovs-port connected ovs-port-phys0 ... ``` The problem is resolved if one instructs NM not to manage these devices (it is managed intentionally?). To unmanage `br-ex`, execute several times the command `nmcli device set br-ex managed no` until the state appears as `unmanaged`. 1. OVSDB (and NetworkManager) supports having the bridge, port and interface with the same name. Nmstate on the other hand has legacy issues with interfaces that have identical names, explicitly not supporting a bridge and the internal interface having the same name. (reasoning to the wish to treat each interface as unique, like the kernel does) On the setup I tested, nmstate 0.2.10 is used. I will need to check if this is resolves (even partially) on latest versions. I will discuss this with the nmstate team anyway and update, but I would recommend to use the workaround suggested above for the meantime.
I tried the suggested workaround(running 'nmcli device set br-ex managed no' several times on the nodes) but unfortunately it didn't have the expected results. The NNCP reports SuccessfullyConfigured but the bridges/vlan interfaces weren't configured on the nodes. Moreover the nodes lose their hostnames which were received via DHCP over the br-ex interface.
This issue is being addressed in nmstate via https://bugzilla.redhat.com/show_bug.cgi?id=1886334. Once it is solved there, we need to get the hotfix into CNV.
(In reply to Petr Horáček from comment #2) > Setting as a blocker since it prevents us from using nmstate with OVN > Kubernetes. > > @Meni, are we reproducing this on tier2? Its in progress
Created attachment 1726086 [details] nmstate-handler-c5c9b.log ***NOT FIXED*** ENVIORMENT ---------- $ oc version Client Version: 4.6.0-202010271321.p0-ffd6836 Server Version: 4.6.1 Kubernetes Version: v1.19.0+d59ce34 $ oc get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v2.5.0 OpenShift Virtualization 2.5.0 kubevirt-hyperconverged-operator.v2.4.2 Succeeded $ oc describe daemonset.apps/nmstate-handler -n openshift-cnv | grep Image Image: registry.redhat.io/container-native-virtualization/kubernetes-nmstate-handler-rhel8@sha256:997feb555f9b9ac31c41d34b45ba2a200f478058419e4096c9ad9991bceb5430 image corresponds to nmstate-handler v2.5.0-18 so bug should be fixed STEPS ----- create following nncp: apiVersion: nmstate.io/v1alpha1 kind: NodeNetworkConfigurationPolicy metadata: name: bigip-bridges spec: nodeSelector: node-role.kubernetes.io/worker: "" desiredState: interfaces: - name: eno2np1.375 type: vlan state: up vlan: base-iface: eno2np1 id: 375 - name: bigip-mgmt description: Linux bridge with eno2np1 vlan375 as a port! type: linux-bridge state: up bridge: options: stp: enabled: false port: - name: eno2np1.375 - name: eno2np1.376 type: vlan state: up vlan: base-iface: eno2np1 id: 376 - name: bigip-ha description: Linux bridge with eno2np1 vlan376 as a port! type: linux-bridge state: up bridge: options: stp: enabled: false port: - name: eno2np1.376 RESAULTS --------- attached log of one of nmstate-handler pods $ oc get nncp NAME STATUS bigip-bridges FailedToConfigure [core@25-ovn-7wr7r-worker-0-2xh4j ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master ovs-system state UP group default qlen 1000 link/ether fa:16:3e:7f:56:1d brd ff:ff:ff:ff:ff:ff 3: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:3c:df:08 brd ff:ff:ff:ff:ff:ff 4: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:a7:a2:bb brd ff:ff:ff:ff:ff:ff 5: ens10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000 link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 2a:ff:2e:77:c7:d8 brd ff:ff:ff:ff:ff:ff 7: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000 link/ether 82:48:6d:e4:39:43 brd ff:ff:ff:ff:ff:ff inet6 fe80::8048:6dff:fee4:3943/64 scope link valid_lft forever preferred_lft forever 8: ovn-k8s-mp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether d2:27:65:23:37:13 brd ff:ff:ff:ff:ff:ff inet 10.131.0.2/23 brd 10.131.1.255 scope global ovn-k8s-mp0 valid_lft forever preferred_lft forever inet6 fe80::d027:65ff:fe23:3713/64 scope link valid_lft forever preferred_lft forever 9: br-int: <BROADCAST,MULTICAST> mtu 1350 qdisc noop state DOWN group default qlen 1000 link/ether ba:1f:60:c1:28:42 brd ff:ff:ff:ff:ff:ff 10: ovn-k8s-gw0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 0a:58:a9:fe:00:01 brd ff:ff:ff:ff:ff:ff inet 169.254.0.1/20 brd 169.254.15.255 scope global ovn-k8s-gw0 valid_lft forever preferred_lft forever inet6 fe80::858:a9ff:fefe:1/64 scope link valid_lft forever preferred_lft forever 11: br-local: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 82:90:a5:55:bc:4d brd ff:ff:ff:ff:ff:ff inet6 fe80::8090:a5ff:fe55:bc4d/64 scope link valid_lft forever preferred_lft forever 12: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:7f:56:1d brd ff:ff:ff:ff:ff:ff inet 192.168.3.75/18 brd 192.168.63.255 scope global dynamic noprefixroute br-ex valid_lft 72545sec preferred_lft 72545sec inet6 fe80::6496:d4b2:2858:cac0/64 scope link noprefixroute valid_lft forever preferred_lft forever 13: ea2c9649d7e529d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 86:88:30:f9:35:fa brd ff:ff:ff:ff:ff:ff link-netns d3552510-e595-41d6-ab0c-d4b4a2e1c5c0 inet6 fe80::8488:30ff:fef9:35fa/64 scope link valid_lft forever preferred_lft forever 14: 2d7116e3627bb6e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 16:93:9f:a3:42:7c brd ff:ff:ff:ff:ff:ff link-netns 14d9e311-e2b4-4935-9fd4-e6bc89893e91 inet6 fe80::1493:9fff:fea3:427c/64 scope link valid_lft forever preferred_lft forever 15: 5e6cb2053162c63@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 3a:4d:8f:57:bb:c4 brd ff:ff:ff:ff:ff:ff link-netns 77048285-bd6b-4a89-a3f0-e9f927e58998 inet6 fe80::384d:8fff:fe57:bbc4/64 scope link valid_lft forever preferred_lft forever 16: 94b77bcacc46c02@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether d2:4f:7b:95:32:76 brd ff:ff:ff:ff:ff:ff link-netns 2f284ed3-cd39-4e24-b5d1-19069d653056 inet6 fe80::d04f:7bff:fe95:3276/64 scope link valid_lft forever preferred_lft forever 17: f39596539d2aaa9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether e6:a2:8c:b2:d1:f1 brd ff:ff:ff:ff:ff:ff link-netns 75404278-ee1f-4b87-8d8d-b5c14b5a5be6 inet6 fe80::e4a2:8cff:feb2:d1f1/64 scope link valid_lft forever preferred_lft forever 18: ff3d409976e995a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether a2:8b:ce:b4:7f:a9 brd ff:ff:ff:ff:ff:ff link-netns 6682e8d0-cabe-4386-9944-e22f77d5e2e1 inet6 fe80::a08b:ceff:feb4:7fa9/64 scope link valid_lft forever preferred_lft forever 19: 872c2dfaef181b7@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 82:75:f8:c8:51:5b brd ff:ff:ff:ff:ff:ff link-netns a893a8a9-2226-4ceb-989d-28e7c3782769 inet6 fe80::8075:f8ff:fec8:515b/64 scope link valid_lft forever preferred_lft forever 20: 00550d780a91404@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 36:3c:74:70:db:16 brd ff:ff:ff:ff:ff:ff link-netns 63dfc261-c412-4cc5-9483-059780596128 inet6 fe80::343c:74ff:fe70:db16/64 scope link valid_lft forever preferred_lft forever 21: 87399a0eae641bc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 0a:bf:8f:5e:1e:04 brd ff:ff:ff:ff:ff:ff link-netns cc2b616e-f3c5-4503-a80a-d9483c28ad93 inet6 fe80::8bf:8fff:fe5e:1e04/64 scope link valid_lft forever preferred_lft forever 22: 2c37bc123c551c9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 32:53:dd:cf:3c:57 brd ff:ff:ff:ff:ff:ff link-netns 474c0471-8de7-4c77-8093-d1af9c0f4b0c inet6 fe80::3053:ddff:fecf:3c57/64 scope link valid_lft forever preferred_lft forever 23: 1a8e37d013fc61f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether d2:87:25:cd:59:8e brd ff:ff:ff:ff:ff:ff link-netns 799afe0b-391d-4598-910b-f03e7b0ece79 inet6 fe80::d087:25ff:fecd:598e/64 scope link valid_lft forever preferred_lft forever 24: a6479254129594e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 62:67:a4:83:06:5c brd ff:ff:ff:ff:ff:ff link-netns e86dfb82-7a79-4bcd-ae8e-34554d5757b8 inet6 fe80::6067:a4ff:fe83:65c/64 scope link valid_lft forever preferred_lft forever 25: b0e245f5d81f475@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 62:82:79:90:59:56 brd ff:ff:ff:ff:ff:ff link-netns 981d9b99-710d-4558-9368-a07a5efbfa51 inet6 fe80::6082:79ff:fe90:5956/64 scope link valid_lft forever preferred_lft forever 26: 61d71037e4b282a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether ca:08:9b:f4:d7:33 brd ff:ff:ff:ff:ff:ff link-netns f481fde7-1eb1-41f5-b2aa-0360380204ef inet6 fe80::c808:9bff:fef4:d733/64 scope link valid_lft forever preferred_lft forever 27: 7fda66c026fcfc6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether fa:2d:aa:e9:65:81 brd ff:ff:ff:ff:ff:ff link-netns 31d61394-64f8-4362-b7de-2b6e5cc15926 inet6 fe80::f82d:aaff:fee9:6581/64 scope link valid_lft forever preferred_lft forever 28: 7766a6bed505cce@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether a2:89:40:3b:21:33 brd ff:ff:ff:ff:ff:ff link-netns 210f4ef9-c4b2-4215-966e-9bb3281b4c91 inet6 fe80::a089:40ff:fe3b:2133/64 scope link valid_lft forever preferred_lft forever 29: 38e542cb0f8baa7@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 66:c0:f1:9e:9b:33 brd ff:ff:ff:ff:ff:ff link-netns bdfd2d3e-be64-4516-bf54-9ee4ea97bde8 inet6 fe80::64c0:f1ff:fe9e:9b33/64 scope link valid_lft forever preferred_lft forever 30: 7582e76bd00e770@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether b6:e0:33:5f:e0:07 brd ff:ff:ff:ff:ff:ff link-netns 439508b4-836e-4d9b-acfa-ae4c420c4518 inet6 fe80::b4e0:33ff:fe5f:e007/64 scope link valid_lft forever preferred_lft forever 31: 18324aefb1260c6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether ce:e0:8c:5e:08:cb brd ff:ff:ff:ff:ff:ff link-netns 3f615f2a-2fd2-4884-9aa7-e34b3cc84c05 inet6 fe80::cce0:8cff:fe5e:8cb/64 scope link valid_lft forever preferred_lft forever 32: 6f261d6f7811bf1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 4a:52:a0:5d:b4:38 brd ff:ff:ff:ff:ff:ff link-netns d5713ddf-083d-4a8a-acf7-3bb7ec28e4e0 inet6 fe80::4852:a0ff:fe5d:b438/64 scope link valid_lft forever preferred_lft forever 33: 3697fff24c5dcc2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 0a:2c:cb:a6:b5:03 brd ff:ff:ff:ff:ff:ff link-netns 3bb0874a-a52a-46a2-8972-42c4ad07ef26 inet6 fe80::82c:cbff:fea6:b503/64 scope link valid_lft forever preferred_lft forever 34: 61663a07e14834d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether ca:5d:90:70:b0:c8 brd ff:ff:ff:ff:ff:ff link-netns 6ed49ff7-5d29-4043-b279-668604212cd4 inet6 fe80::c85d:90ff:fe70:b0c8/64 scope link valid_lft forever preferred_lft forever 35: cf8cdce04f8430f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether de:d2:a9:f2:a2:b7 brd ff:ff:ff:ff:ff:ff link-netns 25a1e794-00d8-4bc4-a1ba-2c92315495ac inet6 fe80::dcd2:a9ff:fef2:a2b7/64 scope link valid_lft forever preferred_lft forever 36: 4bab8e939519a8d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether ca:f0:6f:b2:80:82 brd ff:ff:ff:ff:ff:ff link-netns 398e0002-f711-4984-842c-23c37f1c839a inet6 fe80::c8f0:6fff:feb2:8082/64 scope link valid_lft forever preferred_lft forever 37: bf1816dba0f9a8b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 5e:89:e0:26:dd:1f brd ff:ff:ff:ff:ff:ff link-netns a728b655-a9e6-4f46-9c8b-d04b2ce10ff8 inet6 fe80::5c89:e0ff:fe26:dd1f/64 scope link valid_lft forever preferred_lft forever 38: 3352261b9605743@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 36:7a:f5:f6:c4:a7 brd ff:ff:ff:ff:ff:ff link-netns 3e0ae4bf-7d67-4bbb-8d28-2c159ceb3f08 inet6 fe80::347a:f5ff:fef6:c4a7/64 scope link valid_lft forever preferred_lft forever 39: e1f91184aef5491@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether d6:24:01:cc:09:89 brd ff:ff:ff:ff:ff:ff link-netns 4e6bb4fa-fabc-4c9d-a0cb-6e44f9ca5d62 inet6 fe80::d424:1ff:fecc:989/64 scope link valid_lft forever preferred_lft forever 40: c8541055b2db5e9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether b2:cf:d1:7a:c2:ee brd ff:ff:ff:ff:ff:ff link-netns 93aa25b9-187a-4b1c-8831-41f24d73a610 inet6 fe80::b0cf:d1ff:fe7a:c2ee/64 scope link valid_lft forever preferred_lft forever 42: cf65cefe06d70a4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 66:31:aa:c6:cf:61 brd ff:ff:ff:ff:ff:ff link-netns b8304603-6683-48d4-9f4c-f122af35dbe5 inet6 fe80::6431:aaff:fec6:cf61/64 scope link valid_lft forever preferred_lft forever 43: 0d30c5ae2d6f4aa@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether b6:7d:77:63:86:d0 brd ff:ff:ff:ff:ff:ff link-netns dd301ad7-34e7-4476-919f-b39796770e37 inet6 fe80::b47d:77ff:fe63:86d0/64 scope link valid_lft forever preferred_lft forever 44: 118de30d9435362@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UP group default link/ether ee:98:07:85:44:a5 brd ff:ff:ff:ff:ff:ff link-netns eecbc6cb-16a0-4003-9d7f-a455d7cef165 inet6 fe80::ec98:7ff:fe85:44a5/64 scope link valid_lft forever preferred_lft forever 45: 56a30e16a72a0ea@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 9e:89:73:af:76:4d brd ff:ff:ff:ff:ff:ff link-netns e7b89063-b916-4dec-8273-d8f856692d34 inet6 fe80::9c89:73ff:feaf:764d/64 scope link valid_lft forever preferred_lft forever 46: 761998dc18ea934@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default link/ether 96:c4:0b:68:e7:2d brd ff:ff:ff:ff:ff:ff link-netns f56ff567-e253-41b5-8b10-901440307759 inet6 fe80::94c4:bff:fe68:e72d/64 scope link valid_lft forever preferred_lft forever
What is the version/release of the Nmstate RPM in the tested container image?
***FIXED*** ENVIORMENT ---------- $ oc version Client Version: 4.6.0-202010271321.p0-ffd6836 Server Version: 4.6.1 Kubernetes Version: v1.19.0+d59ce34 $ oc get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v2.5.0 OpenShift Virtualization 2.5.0 kubevirt-hyperconverged-operator.v2.4.2 Succeeded $ oc describe daemonset.apps/nmstate-handler -n openshift-cnv | grep Image Image: registry.redhat.io/container-native-virtualization/kubernetes-nmstate-handler-rhel8@sha256:997feb555f9b9ac31c41d34b45ba2a200f478058419e4096c9ad9991bceb5430 image corresponds to nmstate-handler v2.5.0-18 so bug should be fixed STEPS ----- 1) check cluster have managed OvS bridge 2) check following nncp is configured successfully: apiVersion: nmstate.io/v1alpha1 kind: NodeNetworkConfigurationPolicy metadata: name: bigip-bridges spec: nodeSelector: node-role.kubernetes.io/worker: "" desiredState: interfaces: - name: ens6.375 type: vlan state: up vlan: base-iface: ens6 id: 375 - name: bigip-mgmt description: Linux bridge with ens6 vlan375 as a port! type: linux-bridge state: up bridge: options: stp: enabled: false port: - name: ens6.375 - name: ens6.376 type: vlan state: up vlan: base-iface: ens6 id: 376 - name: bigip-ha description: Linux bridge with ens6 vlan376 as a port! type: linux-bridge state: up bridge: options: stp: enabled: false port: - name: ens6.376 3) check nmstate shows state of all interfaces configured in that nncp RESAULTS --------- 1) [core@25-ovn-7wr7r-worker-0-2xh4j ~]$ nmcli d DEVICE TYPE STATE CONNECTION br-ex ovs-interface connected ovs-if-br-ex bigip-ha bridge connected bigip-ha bigip-mgmt bridge connected bigip-mgmt ens3 ethernet connected ovs-if-phys0 br-ex ovs-bridge connected br-ex br-ex ovs-port connected ovs-port-br-ex ens3 ovs-port connected ovs-port-phys0 ens6.375 vlan connected ens6.375 ens6.376 vlan connected ens6.376 2) [cnv-qe-jenkins@243ovn-h276h-executor ~]$ oc get nncp NAME STATUS bigip-bridges SuccessfullyConfigured [core@25-ovn-7wr7r-worker-0-2xh4j ~]$ ip a ... 45: bigip-ha: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff 46: bigip-mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff 47: ens6.375@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master bigip-mgmt state UP group default qlen 1000 link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff 48: ens6.376@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master bigip-ha state UP group default qlen 1000 link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff 3) knmstate shows state of interfaces configured in nncp: $ oc get nns 25-ovn-7wr7r-worker-0-swlb2 -oyaml - bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: ens6.376 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 description: Linux bridge with ens6 vlan376 as a port! ipv4: dhcp: false enabled: false ipv6: autoconf: false dhcp: false enabled: false mac-address: FA:16:3E:C1:CB:FB mtu: 1450 name: bigip-ha state: up type: linux-bridge - bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: ens6.375 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 description: Linux bridge with ens6 vlan375 as a port! ipv4: dhcp: false enabled: false ipv6: autoconf: false dhcp: false enabled: false mac-address: FA:16:3E:C1:CB:FB mtu: 1450 name: bigip-mgmt state: up type: linux-bridge - ipv4: dhcp: false enabled: false ipv6: autoconf: false dhcp: false enabled: false mac-address: FA:16:3E:C1:CB:FB mtu: 1450 name: ens6.375 state: up type: vlan vlan: base-iface: ens6 id: 375 - ipv4: dhcp: false enabled: false ipv6: autoconf: false dhcp: false enabled: false mac-address: FA:16:3E:C1:CB:FB mtu: 1450 name: ens6.376 state: up type: vlan vlan: base-iface: ens6 id: 376
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:5127