Bug 1885174
| Summary: | tracker: knmstate fails to show state if node has a managed OvS bridge | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Marius Cornea <mcornea> | ||||||
| Component: | Networking | Assignee: | Edward Haas <edwardh> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Meni Yakove <myakove> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 2.5.0 | CC: | cnv-qe-bugs, danken, edwardh, myakove, oramraz, phoracek, rgelobte, till, yjoseph, yprokule, yzaindbe | ||||||
| Target Milestone: | --- | Keywords: | TestBlocker | ||||||
| Target Release: | 2.5.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | kubernetes-nmstate-handler-container-v2.5.0-16 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | |||||||||
| : | 1886334 1893639 (view as bug list) | Environment: | |||||||
| Last Closed: | 2020-11-17 13:24:56 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1886334, 1887996, 1887997, 1893639 | ||||||||
| Attachments: |
|
||||||||
NIC status from one of the workers:
[root@openshift-worker-7 core]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether bc:97:e1:69:a1:a0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::be97:e1ff:fe69:a1a0/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether bc:97:e1:69:a1:a1 brd ff:ff:ff:ff:ff:ff
4: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 1c:34:da:75:bc:76 brd ff:ff:ff:ff:ff:ff
inet6 fe80::1e34:daff:fe75:bc76/64 scope link noprefixroute
valid_lft forever preferred_lft forever
5: ens4f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 1c:34:da:75:bc:77 brd ff:ff:ff:ff:ff:ff
inet6 fe80::1e34:daff:fe75:bc77/64 scope link noprefixroute
valid_lft forever preferred_lft forever
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 56:80:34:f7:58:db brd ff:ff:ff:ff:ff:ff
7: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether ae:e3:2f:1b:32:4c brd ff:ff:ff:ff:ff:ff
inet6 fe80::ace3:2fff:fe1b:324c/64 scope link
valid_lft forever preferred_lft forever
8: ovn-k8s-mp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 22:47:7d:f7:ad:dd brd ff:ff:ff:ff:ff:ff
inet 10.129.4.2/23 brd 10.129.5.255 scope global ovn-k8s-mp0
valid_lft forever preferred_lft forever
inet6 fe80::2047:7dff:fef7:addd/64 scope link
valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,MULTICAST> mtu 1400 qdisc noop state DOWN group default qlen 1000
link/ether 06:2d:ea:f5:b7:4d brd ff:ff:ff:ff:ff:ff
10: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether fe:d9:3a:35:c9:ac brd ff:ff:ff:ff:ff:ff
11: br-ext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether b6:6a:7a:b5:01:44 brd ff:ff:ff:ff:ff:ff
12: ovn-k8s-gw0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 0a:58:a9:fe:00:01 brd ff:ff:ff:ff:ff:ff
inet 169.254.0.1/20 brd 169.254.15.255 scope global ovn-k8s-gw0
valid_lft forever preferred_lft forever
inet6 fe80::858:a9ff:fefe:1/64 scope link
valid_lft forever preferred_lft forever
13: br-local: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether e2:23:2b:67:ab:4e brd ff:ff:ff:ff:ff:ff
inet6 fe80::e023:2bff:fe67:ab4e/64 scope link
valid_lft forever preferred_lft forever
14: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether bc:97:e1:69:a1:a1 brd ff:ff:ff:ff:ff:ff
inet 10.46.57.30/24 brd 10.46.57.255 scope global dynamic noprefixroute br-ex
valid_lft 39269sec preferred_lft 39269sec
inet6 2620:52:0:2e39:ea0:53e1:f96f:c2ff/64 scope global dynamic noprefixroute
valid_lft 2591839sec preferred_lft 604639sec
inet6 fe80::c0a5:eda6:751d:b329/64 scope link noprefixroute
valid_lft forever preferred_lft forever
15: ddeeaf2d4d1c2f7@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default
link/ether 66:e3:2c:76:08:fd brd ff:ff:ff:ff:ff:ff link-netns 0731db1a-5fed-4451-8198-07c185b38cf1
inet6 fe80::64e3:2cff:fe76:8fd/64 scope link
valid_lft forever preferred_lft forever
16: fef9763f672c7de@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default
link/ether be:3a:54:40:e3:67 brd ff:ff:ff:ff:ff:ff link-netns 6cd8069f-9fbf-438b-b120-ccaa6abdd351
inet6 fe80::bc3a:54ff:fe40:e367/64 scope link
valid_lft forever preferred_lft forever
17: 00db64dbec31d02@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default
link/ether b2:c9:72:95:28:c3 brd ff:ff:ff:ff:ff:ff link-netns 52478bc1-aab4-43a6-b247-23a0a434bbfd
inet6 fe80::b0c9:72ff:fe95:28c3/64 scope link
valid_lft forever preferred_lft forever
18: b9dbe100229f649@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default
link/ether 1e:3d:ce:4e:7a:6b brd ff:ff:ff:ff:ff:ff link-netns 41f20e83-1aea-4721-870e-99bc5b9c5b8c
inet6 fe80::1c3d:ceff:fe4e:7a6b/64 scope link
valid_lft forever preferred_lft forever
19: 23ce0b00a3f561a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default
link/ether 8e:02:dc:8d:ec:a7 brd ff:ff:ff:ff:ff:ff link-netns 48d21d86-8f00-46b8-807e-109261c7b430
inet6 fe80::8c02:dcff:fe8d:eca7/64 scope link
valid_lft forever preferred_lft forever
27: 56cf1bf46d64b45@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default
link/ether be:d5:76:b0:bc:1c brd ff:ff:ff:ff:ff:ff link-netns 15acad40-1b6c-446c-88f7-250b5add2188
inet6 fe80::bcd5:76ff:feb0:bc1c/64 scope link
valid_lft forever preferred_lft forever
30: ae5e9fae5c07f59@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP group default
link/ether 5e:d0:c9:84:3d:f3 brd ff:ff:ff:ff:ff:ff link-netns b9d70228-84ca-4316-a584-b55ed41f5ee8
inet6 fe80::5cd0:c9ff:fe84:3df3/64 scope link
valid_lft forever preferred_lft forever
32: 98c59bc0ab6c508@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
link/ether ea:77:f5:f0:ed:17 brd ff:ff:ff:ff:ff:ff link-netns debc2dc4-ed37-4d14-800b-b82346b4189c
inet6 fe80::e877:f5ff:fef0:ed17/64 scope link
valid_lft forever preferred_lft forever
[root@openshift-worker-7 core]# ovs-vsctl show
ee497a89-be46-4436-8d60-0342905b9328
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port patch-br-ex_openshift-worker-7-to-br-int
Interface patch-br-ex_openshift-worker-7-to-br-int
type: patch
options: {peer=patch-br-int-to-br-ex_openshift-worker-7}
Port eno2np1
Interface eno2np1
type: system
Bridge br-local
Port patch-lnet-node_local_switch-to-br-int
Interface patch-lnet-node_local_switch-to-br-int
type: patch
options: {peer=patch-br-int-to-lnet-node_local_switch}
Port br-local
Interface br-local
type: internal
Port ovn-k8s-gw0
Interface ovn-k8s-gw0
type: internal
Bridge br-ext
fail_mode: secure
Port ext-vxlan
Interface ext-vxlan
type: vxlan
options: {dst_port="4789", key=flow, remote_ip=flow}
Port br-ext
Interface br-ext
type: internal
Port ext
Interface ext
type: patch
options: {peer=int}
Bridge br-int
fail_mode: secure
Port ovn-efa4ae-0
Interface ovn-efa4ae-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.22"}
Port patch-br-int-to-lnet-node_local_switch
Interface patch-br-int-to-lnet-node_local_switch
type: patch
options: {peer=patch-lnet-node_local_switch-to-br-int}
Port b9dbe100229f649
Interface b9dbe100229f649
Port ddeeaf2d4d1c2f7
Interface ddeeaf2d4d1c2f7
Port patch-br-int-to-br-ex_openshift-worker-7
Interface patch-br-int-to-br-ex_openshift-worker-7
type: patch
options: {peer=patch-br-ex_openshift-worker-7-to-br-int}
Port ovn-6c9ec6-0
Interface ovn-6c9ec6-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.29"}
Port br-int
Interface br-int
type: internal
Port "23ce0b00a3f561a"
Interface "23ce0b00a3f561a"
Port "00db64dbec31d02"
Interface "00db64dbec31d02"
Port ovn-57903f-0
Interface ovn-57903f-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.21"}
Port ovn-ed0f0f-0
Interface ovn-ed0f0f-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.25"}
Port ovn-c5dbad-0
Interface ovn-c5dbad-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.26"}
Port ovn-0dabfc-0
Interface ovn-0dabfc-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.20"}
Port ovn-k8s-mp0
Interface ovn-k8s-mp0
type: internal
Port int
Interface int
type: patch
options: {peer=ext}
Port ovn-8e4f87-0
Interface ovn-8e4f87-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.24"}
Port fef9763f672c7de
Interface fef9763f672c7de
Port ovn-738f81-0
Interface ovn-738f81-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.23"}
Port "56cf1bf46d64b45"
Interface "56cf1bf46d64b45"
Port ovn-9c1d11-0
Interface ovn-9c1d11-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.28"}
Port ovn-af3fa5-0
Interface ovn-af3fa5-0
type: geneve
options: {csum="true", key=flow, remote_ip="10.46.57.27"}
Port ae5e9fae5c07f59
Interface ae5e9fae5c07f59
ovs_version: "2.13.2"
Setting as a blocker since it prevents us from using nmstate with OVN Kubernetes. @Meni, are we reproducing this on tier2? Per my initial investigation on a local setup, the problem is that `br-ex` is **managed** by NetworkManager and nmstate has a problem reporting it [1]. Here is the `nmcli device` report: ``` DEVICE TYPE STATE CONNECTION br-ex ovs-interface connected ovs-if-br-ex enp1s0 ethernet connected Wired Connection enp2s0 ethernet connected ovs-if-phys0 br-ex ovs-bridge connected br-ex br-ex ovs-port connected ovs-port-br-ex enp2s0 ovs-port connected ovs-port-phys0 ... ``` The problem is resolved if one instructs NM not to manage these devices (it is managed intentionally?). To unmanage `br-ex`, execute several times the command `nmcli device set br-ex managed no` until the state appears as `unmanaged`. 1. OVSDB (and NetworkManager) supports having the bridge, port and interface with the same name. Nmstate on the other hand has legacy issues with interfaces that have identical names, explicitly not supporting a bridge and the internal interface having the same name. (reasoning to the wish to treat each interface as unique, like the kernel does) On the setup I tested, nmstate 0.2.10 is used. I will need to check if this is resolves (even partially) on latest versions. I will discuss this with the nmstate team anyway and update, but I would recommend to use the workaround suggested above for the meantime. I tried the suggested workaround(running 'nmcli device set br-ex managed no' several times on the nodes) but unfortunately it didn't have the expected results. The NNCP reports SuccessfullyConfigured but the bridges/vlan interfaces weren't configured on the nodes. Moreover the nodes lose their hostnames which were received via DHCP over the br-ex interface. This issue is being addressed in nmstate via https://bugzilla.redhat.com/show_bug.cgi?id=1886334. Once it is solved there, we need to get the hotfix into CNV. (In reply to Petr Horáček from comment #2) > Setting as a blocker since it prevents us from using nmstate with OVN > Kubernetes. > > @Meni, are we reproducing this on tier2? Its in progress Created attachment 1726086 [details]
nmstate-handler-c5c9b.log
***NOT FIXED***
ENVIORMENT
----------
$ oc version
Client Version: 4.6.0-202010271321.p0-ffd6836
Server Version: 4.6.1
Kubernetes Version: v1.19.0+d59ce34
$ oc get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
kubevirt-hyperconverged-operator.v2.5.0 OpenShift Virtualization 2.5.0 kubevirt-hyperconverged-operator.v2.4.2 Succeeded
$ oc describe daemonset.apps/nmstate-handler -n openshift-cnv | grep Image
Image: registry.redhat.io/container-native-virtualization/kubernetes-nmstate-handler-rhel8@sha256:997feb555f9b9ac31c41d34b45ba2a200f478058419e4096c9ad9991bceb5430
image corresponds to nmstate-handler v2.5.0-18 so bug should be fixed
STEPS
-----
create following nncp:
apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
name: bigip-bridges
spec:
nodeSelector:
node-role.kubernetes.io/worker: ""
desiredState:
interfaces:
- name: eno2np1.375
type: vlan
state: up
vlan:
base-iface: eno2np1
id: 375
- name: bigip-mgmt
description: Linux bridge with eno2np1 vlan375 as a port!
type: linux-bridge
state: up
bridge:
options:
stp:
enabled: false
port:
- name: eno2np1.375
- name: eno2np1.376
type: vlan
state: up
vlan:
base-iface: eno2np1
id: 376
- name: bigip-ha
description: Linux bridge with eno2np1 vlan376 as a port!
type: linux-bridge
state: up
bridge:
options:
stp:
enabled: false
port:
- name: eno2np1.376
RESAULTS
---------
attached log of one of nmstate-handler pods
$ oc get nncp
NAME STATUS
bigip-bridges FailedToConfigure
[core@25-ovn-7wr7r-worker-0-2xh4j ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master ovs-system state UP group default qlen 1000
link/ether fa:16:3e:7f:56:1d brd ff:ff:ff:ff:ff:ff
3: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
link/ether fa:16:3e:3c:df:08 brd ff:ff:ff:ff:ff:ff
4: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
link/ether fa:16:3e:a7:a2:bb brd ff:ff:ff:ff:ff:ff
5: ens10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2a:ff:2e:77:c7:d8 brd ff:ff:ff:ff:ff:ff
7: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether 82:48:6d:e4:39:43 brd ff:ff:ff:ff:ff:ff
inet6 fe80::8048:6dff:fee4:3943/64 scope link
valid_lft forever preferred_lft forever
8: ovn-k8s-mp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether d2:27:65:23:37:13 brd ff:ff:ff:ff:ff:ff
inet 10.131.0.2/23 brd 10.131.1.255 scope global ovn-k8s-mp0
valid_lft forever preferred_lft forever
inet6 fe80::d027:65ff:fe23:3713/64 scope link
valid_lft forever preferred_lft forever
9: br-int: <BROADCAST,MULTICAST> mtu 1350 qdisc noop state DOWN group default qlen 1000
link/ether ba:1f:60:c1:28:42 brd ff:ff:ff:ff:ff:ff
10: ovn-k8s-gw0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 0a:58:a9:fe:00:01 brd ff:ff:ff:ff:ff:ff
inet 169.254.0.1/20 brd 169.254.15.255 scope global ovn-k8s-gw0
valid_lft forever preferred_lft forever
inet6 fe80::858:a9ff:fefe:1/64 scope link
valid_lft forever preferred_lft forever
11: br-local: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 82:90:a5:55:bc:4d brd ff:ff:ff:ff:ff:ff
inet6 fe80::8090:a5ff:fe55:bc4d/64 scope link
valid_lft forever preferred_lft forever
12: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:7f:56:1d brd ff:ff:ff:ff:ff:ff
inet 192.168.3.75/18 brd 192.168.63.255 scope global dynamic noprefixroute br-ex
valid_lft 72545sec preferred_lft 72545sec
inet6 fe80::6496:d4b2:2858:cac0/64 scope link noprefixroute
valid_lft forever preferred_lft forever
13: ea2c9649d7e529d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 86:88:30:f9:35:fa brd ff:ff:ff:ff:ff:ff link-netns d3552510-e595-41d6-ab0c-d4b4a2e1c5c0
inet6 fe80::8488:30ff:fef9:35fa/64 scope link
valid_lft forever preferred_lft forever
14: 2d7116e3627bb6e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 16:93:9f:a3:42:7c brd ff:ff:ff:ff:ff:ff link-netns 14d9e311-e2b4-4935-9fd4-e6bc89893e91
inet6 fe80::1493:9fff:fea3:427c/64 scope link
valid_lft forever preferred_lft forever
15: 5e6cb2053162c63@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 3a:4d:8f:57:bb:c4 brd ff:ff:ff:ff:ff:ff link-netns 77048285-bd6b-4a89-a3f0-e9f927e58998
inet6 fe80::384d:8fff:fe57:bbc4/64 scope link
valid_lft forever preferred_lft forever
16: 94b77bcacc46c02@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether d2:4f:7b:95:32:76 brd ff:ff:ff:ff:ff:ff link-netns 2f284ed3-cd39-4e24-b5d1-19069d653056
inet6 fe80::d04f:7bff:fe95:3276/64 scope link
valid_lft forever preferred_lft forever
17: f39596539d2aaa9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether e6:a2:8c:b2:d1:f1 brd ff:ff:ff:ff:ff:ff link-netns 75404278-ee1f-4b87-8d8d-b5c14b5a5be6
inet6 fe80::e4a2:8cff:feb2:d1f1/64 scope link
valid_lft forever preferred_lft forever
18: ff3d409976e995a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether a2:8b:ce:b4:7f:a9 brd ff:ff:ff:ff:ff:ff link-netns 6682e8d0-cabe-4386-9944-e22f77d5e2e1
inet6 fe80::a08b:ceff:feb4:7fa9/64 scope link
valid_lft forever preferred_lft forever
19: 872c2dfaef181b7@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 82:75:f8:c8:51:5b brd ff:ff:ff:ff:ff:ff link-netns a893a8a9-2226-4ceb-989d-28e7c3782769
inet6 fe80::8075:f8ff:fec8:515b/64 scope link
valid_lft forever preferred_lft forever
20: 00550d780a91404@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 36:3c:74:70:db:16 brd ff:ff:ff:ff:ff:ff link-netns 63dfc261-c412-4cc5-9483-059780596128
inet6 fe80::343c:74ff:fe70:db16/64 scope link
valid_lft forever preferred_lft forever
21: 87399a0eae641bc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 0a:bf:8f:5e:1e:04 brd ff:ff:ff:ff:ff:ff link-netns cc2b616e-f3c5-4503-a80a-d9483c28ad93
inet6 fe80::8bf:8fff:fe5e:1e04/64 scope link
valid_lft forever preferred_lft forever
22: 2c37bc123c551c9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 32:53:dd:cf:3c:57 brd ff:ff:ff:ff:ff:ff link-netns 474c0471-8de7-4c77-8093-d1af9c0f4b0c
inet6 fe80::3053:ddff:fecf:3c57/64 scope link
valid_lft forever preferred_lft forever
23: 1a8e37d013fc61f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether d2:87:25:cd:59:8e brd ff:ff:ff:ff:ff:ff link-netns 799afe0b-391d-4598-910b-f03e7b0ece79
inet6 fe80::d087:25ff:fecd:598e/64 scope link
valid_lft forever preferred_lft forever
24: a6479254129594e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 62:67:a4:83:06:5c brd ff:ff:ff:ff:ff:ff link-netns e86dfb82-7a79-4bcd-ae8e-34554d5757b8
inet6 fe80::6067:a4ff:fe83:65c/64 scope link
valid_lft forever preferred_lft forever
25: b0e245f5d81f475@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 62:82:79:90:59:56 brd ff:ff:ff:ff:ff:ff link-netns 981d9b99-710d-4558-9368-a07a5efbfa51
inet6 fe80::6082:79ff:fe90:5956/64 scope link
valid_lft forever preferred_lft forever
26: 61d71037e4b282a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether ca:08:9b:f4:d7:33 brd ff:ff:ff:ff:ff:ff link-netns f481fde7-1eb1-41f5-b2aa-0360380204ef
inet6 fe80::c808:9bff:fef4:d733/64 scope link
valid_lft forever preferred_lft forever
27: 7fda66c026fcfc6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether fa:2d:aa:e9:65:81 brd ff:ff:ff:ff:ff:ff link-netns 31d61394-64f8-4362-b7de-2b6e5cc15926
inet6 fe80::f82d:aaff:fee9:6581/64 scope link
valid_lft forever preferred_lft forever
28: 7766a6bed505cce@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether a2:89:40:3b:21:33 brd ff:ff:ff:ff:ff:ff link-netns 210f4ef9-c4b2-4215-966e-9bb3281b4c91
inet6 fe80::a089:40ff:fe3b:2133/64 scope link
valid_lft forever preferred_lft forever
29: 38e542cb0f8baa7@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 66:c0:f1:9e:9b:33 brd ff:ff:ff:ff:ff:ff link-netns bdfd2d3e-be64-4516-bf54-9ee4ea97bde8
inet6 fe80::64c0:f1ff:fe9e:9b33/64 scope link
valid_lft forever preferred_lft forever
30: 7582e76bd00e770@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether b6:e0:33:5f:e0:07 brd ff:ff:ff:ff:ff:ff link-netns 439508b4-836e-4d9b-acfa-ae4c420c4518
inet6 fe80::b4e0:33ff:fe5f:e007/64 scope link
valid_lft forever preferred_lft forever
31: 18324aefb1260c6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether ce:e0:8c:5e:08:cb brd ff:ff:ff:ff:ff:ff link-netns 3f615f2a-2fd2-4884-9aa7-e34b3cc84c05
inet6 fe80::cce0:8cff:fe5e:8cb/64 scope link
valid_lft forever preferred_lft forever
32: 6f261d6f7811bf1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 4a:52:a0:5d:b4:38 brd ff:ff:ff:ff:ff:ff link-netns d5713ddf-083d-4a8a-acf7-3bb7ec28e4e0
inet6 fe80::4852:a0ff:fe5d:b438/64 scope link
valid_lft forever preferred_lft forever
33: 3697fff24c5dcc2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 0a:2c:cb:a6:b5:03 brd ff:ff:ff:ff:ff:ff link-netns 3bb0874a-a52a-46a2-8972-42c4ad07ef26
inet6 fe80::82c:cbff:fea6:b503/64 scope link
valid_lft forever preferred_lft forever
34: 61663a07e14834d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether ca:5d:90:70:b0:c8 brd ff:ff:ff:ff:ff:ff link-netns 6ed49ff7-5d29-4043-b279-668604212cd4
inet6 fe80::c85d:90ff:fe70:b0c8/64 scope link
valid_lft forever preferred_lft forever
35: cf8cdce04f8430f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether de:d2:a9:f2:a2:b7 brd ff:ff:ff:ff:ff:ff link-netns 25a1e794-00d8-4bc4-a1ba-2c92315495ac
inet6 fe80::dcd2:a9ff:fef2:a2b7/64 scope link
valid_lft forever preferred_lft forever
36: 4bab8e939519a8d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether ca:f0:6f:b2:80:82 brd ff:ff:ff:ff:ff:ff link-netns 398e0002-f711-4984-842c-23c37f1c839a
inet6 fe80::c8f0:6fff:feb2:8082/64 scope link
valid_lft forever preferred_lft forever
37: bf1816dba0f9a8b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 5e:89:e0:26:dd:1f brd ff:ff:ff:ff:ff:ff link-netns a728b655-a9e6-4f46-9c8b-d04b2ce10ff8
inet6 fe80::5c89:e0ff:fe26:dd1f/64 scope link
valid_lft forever preferred_lft forever
38: 3352261b9605743@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 36:7a:f5:f6:c4:a7 brd ff:ff:ff:ff:ff:ff link-netns 3e0ae4bf-7d67-4bbb-8d28-2c159ceb3f08
inet6 fe80::347a:f5ff:fef6:c4a7/64 scope link
valid_lft forever preferred_lft forever
39: e1f91184aef5491@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether d6:24:01:cc:09:89 brd ff:ff:ff:ff:ff:ff link-netns 4e6bb4fa-fabc-4c9d-a0cb-6e44f9ca5d62
inet6 fe80::d424:1ff:fecc:989/64 scope link
valid_lft forever preferred_lft forever
40: c8541055b2db5e9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether b2:cf:d1:7a:c2:ee brd ff:ff:ff:ff:ff:ff link-netns 93aa25b9-187a-4b1c-8831-41f24d73a610
inet6 fe80::b0cf:d1ff:fe7a:c2ee/64 scope link
valid_lft forever preferred_lft forever
42: cf65cefe06d70a4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 66:31:aa:c6:cf:61 brd ff:ff:ff:ff:ff:ff link-netns b8304603-6683-48d4-9f4c-f122af35dbe5
inet6 fe80::6431:aaff:fec6:cf61/64 scope link
valid_lft forever preferred_lft forever
43: 0d30c5ae2d6f4aa@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether b6:7d:77:63:86:d0 brd ff:ff:ff:ff:ff:ff link-netns dd301ad7-34e7-4476-919f-b39796770e37
inet6 fe80::b47d:77ff:fe63:86d0/64 scope link
valid_lft forever preferred_lft forever
44: 118de30d9435362@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UP group default
link/ether ee:98:07:85:44:a5 brd ff:ff:ff:ff:ff:ff link-netns eecbc6cb-16a0-4003-9d7f-a455d7cef165
inet6 fe80::ec98:7ff:fe85:44a5/64 scope link
valid_lft forever preferred_lft forever
45: 56a30e16a72a0ea@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 9e:89:73:af:76:4d brd ff:ff:ff:ff:ff:ff link-netns e7b89063-b916-4dec-8273-d8f856692d34
inet6 fe80::9c89:73ff:feaf:764d/64 scope link
valid_lft forever preferred_lft forever
46: 761998dc18ea934@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue master ovs-system state UP group default
link/ether 96:c4:0b:68:e7:2d brd ff:ff:ff:ff:ff:ff link-netns f56ff567-e253-41b5-8b10-901440307759
inet6 fe80::94c4:bff:fe68:e72d/64 scope link
valid_lft forever preferred_lft forever
What is the version/release of the Nmstate RPM in the tested container image? ***FIXED***
ENVIORMENT
----------
$ oc version
Client Version: 4.6.0-202010271321.p0-ffd6836
Server Version: 4.6.1
Kubernetes Version: v1.19.0+d59ce34
$ oc get csv -n openshift-cnv
NAME DISPLAY VERSION REPLACES PHASE
kubevirt-hyperconverged-operator.v2.5.0 OpenShift Virtualization 2.5.0 kubevirt-hyperconverged-operator.v2.4.2 Succeeded
$ oc describe daemonset.apps/nmstate-handler -n openshift-cnv | grep Image
Image: registry.redhat.io/container-native-virtualization/kubernetes-nmstate-handler-rhel8@sha256:997feb555f9b9ac31c41d34b45ba2a200f478058419e4096c9ad9991bceb5430
image corresponds to nmstate-handler v2.5.0-18 so bug should be fixed
STEPS
-----
1) check cluster have managed OvS bridge
2) check following nncp is configured successfully:
apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
name: bigip-bridges
spec:
nodeSelector:
node-role.kubernetes.io/worker: ""
desiredState:
interfaces:
- name: ens6.375
type: vlan
state: up
vlan:
base-iface: ens6
id: 375
- name: bigip-mgmt
description: Linux bridge with ens6 vlan375 as a port!
type: linux-bridge
state: up
bridge:
options:
stp:
enabled: false
port:
- name: ens6.375
- name: ens6.376
type: vlan
state: up
vlan:
base-iface: ens6
id: 376
- name: bigip-ha
description: Linux bridge with ens6 vlan376 as a port!
type: linux-bridge
state: up
bridge:
options:
stp:
enabled: false
port:
- name: ens6.376
3) check nmstate shows state of all interfaces configured in that nncp
RESAULTS
---------
1)
[core@25-ovn-7wr7r-worker-0-2xh4j ~]$ nmcli d
DEVICE TYPE STATE CONNECTION
br-ex ovs-interface connected ovs-if-br-ex
bigip-ha bridge connected bigip-ha
bigip-mgmt bridge connected bigip-mgmt
ens3 ethernet connected ovs-if-phys0
br-ex ovs-bridge connected br-ex
br-ex ovs-port connected ovs-port-br-ex
ens3 ovs-port connected ovs-port-phys0
ens6.375 vlan connected ens6.375
ens6.376 vlan connected ens6.376
2)
[cnv-qe-jenkins@243ovn-h276h-executor ~]$ oc get nncp
NAME STATUS
bigip-bridges SuccessfullyConfigured
[core@25-ovn-7wr7r-worker-0-2xh4j ~]$ ip a
...
45: bigip-ha: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff
46: bigip-mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff
47: ens6.375@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master bigip-mgmt state UP group default qlen 1000
link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff
48: ens6.376@ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master bigip-ha state UP group default qlen 1000
link/ether fa:16:3e:c1:cb:fb brd ff:ff:ff:ff:ff:ff
3) knmstate shows state of interfaces configured in nncp:
$ oc get nns 25-ovn-7wr7r-worker-0-swlb2 -oyaml
- bridge:
options:
group-forward-mask: 0
mac-ageing-time: 300
multicast-snooping: true
stp:
enabled: false
forward-delay: 15
hello-time: 2
max-age: 20
priority: 32768
port:
- name: ens6.376
stp-hairpin-mode: false
stp-path-cost: 100
stp-priority: 32
description: Linux bridge with ens6 vlan376 as a port!
ipv4:
dhcp: false
enabled: false
ipv6:
autoconf: false
dhcp: false
enabled: false
mac-address: FA:16:3E:C1:CB:FB
mtu: 1450
name: bigip-ha
state: up
type: linux-bridge
- bridge:
options:
group-forward-mask: 0
mac-ageing-time: 300
multicast-snooping: true
stp:
enabled: false
forward-delay: 15
hello-time: 2
max-age: 20
priority: 32768
port:
- name: ens6.375
stp-hairpin-mode: false
stp-path-cost: 100
stp-priority: 32
description: Linux bridge with ens6 vlan375 as a port!
ipv4:
dhcp: false
enabled: false
ipv6:
autoconf: false
dhcp: false
enabled: false
mac-address: FA:16:3E:C1:CB:FB
mtu: 1450
name: bigip-mgmt
state: up
type: linux-bridge
- ipv4:
dhcp: false
enabled: false
ipv6:
autoconf: false
dhcp: false
enabled: false
mac-address: FA:16:3E:C1:CB:FB
mtu: 1450
name: ens6.375
state: up
type: vlan
vlan:
base-iface: ens6
id: 375
- ipv4:
dhcp: false
enabled: false
ipv6:
autoconf: false
dhcp: false
enabled: false
mac-address: FA:16:3E:C1:CB:FB
mtu: 1450
name: ens6.376
state: up
type: vlan
vlan:
base-iface: ens6
id: 376
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:5127 |
Created attachment 1718964 [details] nmstate-handler-xvhf2.log Description of problem: On a bare metal IPI deployment creating a NodeNetworkConfigurationPolicy to configure tagged vlans over the external interface fails. I suspect this is related to the external interface getting automatically added to an OVS br-ex bridge in 4.6. Nevertheless the same configuration used to work in 4.5 so I'd like to clarify if there are any limitations introduced in 4.6 by the br-ex OVS bridge and if there are any ways to overcome them. Version-Release number of selected component (if applicable): 2.5.0 How reproducible: 100% Steps to Reproduce: 1. Create the following NodeNetworkConfigurationPolicy: --- apiVersion: nmstate.io/v1alpha1 kind: NodeNetworkConfigurationPolicy metadata: name: bigip-bridges spec: nodeSelector: node-role.kubernetes.io/worker-lb: "" desiredState: interfaces: - name: eno2np1.375 type: vlan state: up vlan: base-iface: eno2np1 id: 375 - name: bigip-mgmt description: Linux bridge with eno2np1 vlan375 as a port! type: linux-bridge state: up bridge: options: stp: enabled: false port: - name: eno2np1.375 - name: eno2np1.376 type: vlan state: up vlan: base-iface: eno2np1 id: 376 - name: bigip-ha description: Linux bridge with eno2np1 vlan376 as a port! type: linux-bridge state: up bridge: options: stp: enabled: false port: - name: eno2np1.376 Actual results: FailedToConfigure Expected results: Configuration succeeds. Additional info: Attaching nmstate-handler log from one of the nodes.