Description of problem: OVS hardware offload in OCP 4.10 When switchdev mode is enabled for Mellanox CX-5 NIC, VF netdev are not bind to mlx5_core driver after node rebooting (triggered by SR-IOV Operator), VF representor devices are not affected. Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2022-01-11-065245 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Assuming PF interface is ens8f0, VF netdev are renamed as ens8f0vX and VF representor are renamed as ens8f0_X after node rebooting by udev rules, where X is the VF index number.
Verified on version below. Enabled switchdev and created sriovnetworknodepolicy on mlx5 port ens2f0, checked 'ip link show' on worker node, ens2f0vX interfaces were observable. [root@f33-h13-000-r640 offload_test]# oc version Client Version: 4.10.0-0.nightly-2022-01-17-023213 Server Version: 4.10.0-0.nightly-2022-01-17-023213 Kubernetes Version: v1.23.0+60f5a1c [root@f33-h13-000-r640 offload_test]# oc get csv NAME DISPLAY VERSION REPLACES PHASE sriov-network-operator.4.10.0-202201181018 SR-IOV Network Operator 4.10.0-202201181018 Succeeded [root@f33-h13-000-r640 offload_test]# # more policy-mlx.yaml apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: policy-mlx namespace: openshift-sriov-network-operator spec: deviceType: netdevice eSwitchMode: "switchdev" nicSelector: deviceID: "1017" rootDevices: - 0000:5e:00.0 vendor: "15b3" pfNames: - ens2f0 nodeSelector: feature.node.kubernetes.io/sriov-capable: "true" numVfs: 16 priority: 5 resourceName: mlxnics sh-4.4# ip link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eno1np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether b0:26:28:d8:b6:a0 brd ff:ff:ff:ff:ff:ff 3: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether b0:26:28:d8:b6:a1 brd ff:ff:ff:ff:ff:ff 4: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 40:a6:b7:0c:1a:50 brd ff:ff:ff:ff:ff:ff 5: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 40:a6:b7:0c:1a:51 brd ff:ff:ff:ff:ff:ff 7: ens2f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 1c:34:da:7d:80:4b brd ff:ff:ff:ff:ff:ff 24: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether 1c:34:da:7d:80:4a brd ff:ff:ff:ff:ff:ff vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 6 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 7 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 8 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 9 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 10 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 11 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 12 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 13 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 14 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off vf 15 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off 25: ens2f0_0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 66:23:e1:3e:20:81 brd ff:ff:ff:ff:ff:ff 26: ens2f0_1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 7e:5c:7d:31:9f:e9 brd ff:ff:ff:ff:ff:ff 27: ens2f0_2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 8e:c9:ca:e4:c9:c2 brd ff:ff:ff:ff:ff:ff 28: ens2f0_3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether a6:a9:eb:ff:08:ff brd ff:ff:ff:ff:ff:ff 29: ens2f0_4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 52:4c:3e:dd:03:24 brd ff:ff:ff:ff:ff:ff 30: ens2f0_5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether da:c3:c0:f9:3b:5b brd ff:ff:ff:ff:ff:ff 31: ens2f0_6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 62:d9:26:46:8b:df brd ff:ff:ff:ff:ff:ff 32: ens2f0_7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether ea:21:28:a1:00:57 brd ff:ff:ff:ff:ff:ff 33: ens2f0_8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether da:22:e9:88:b9:10 brd ff:ff:ff:ff:ff:ff 34: ens2f0_9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether d6:5a:18:19:8d:a8 brd ff:ff:ff:ff:ff:ff 35: ens2f0_10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 7e:60:0a:d6:43:f9 brd ff:ff:ff:ff:ff:ff 36: ens2f0_11: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 4a:3f:4c:e9:f2:11 brd ff:ff:ff:ff:ff:ff 37: ens2f0_12: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether ba:ca:c5:c9:e7:ce brd ff:ff:ff:ff:ff:ff 38: ens2f0_13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 46:45:01:ae:0e:ae brd ff:ff:ff:ff:ff:ff 39: ens2f0_14: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether a6:13:4f:5d:15:4a brd ff:ff:ff:ff:ff:ff 40: ens2f0_15: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 06:e2:52:52:a4:eb brd ff:ff:ff:ff:ff:ff 41: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 0a:23:b9:8e:18:7a brd ff:ff:ff:ff:ff:ff 42: vxlan_sys_4789: <BROADCAST,MULTICAST> mtu 65000 qdisc noqueue master ovs-system state DOWN mode DEFAULT group default qlen 1000 link/ether 56:c4:39:f7:0b:1b brd ff:ff:ff:ff:ff:ff 43: br-ext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fa:89:a5:56:09:4f brd ff:ff:ff:ff:ff:ff 44: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000 link/ether ee:31:a4:17:43:4a brd ff:ff:ff:ff:ff:ff 45: ovn-k8s-mp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 1e:6c:42:66:e8:cd brd ff:ff:ff:ff:ff:ff 46: br-int: <BROADCAST,MULTICAST> mtu 1400 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 16:13:2a:76:53:cf brd ff:ff:ff:ff:ff:ff 83: ens2f0v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether ce:71:c6:d8:49:62 brd ff:ff:ff:ff:ff:ff 84: ens2f0v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 22:c1:af:c0:af:b1 brd ff:ff:ff:ff:ff:ff 85: ens2f0v10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether d2:af:5d:3e:33:ea brd ff:ff:ff:ff:ff:ff 86: ens2f0v11: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 22:86:ae:15:61:31 brd ff:ff:ff:ff:ff:ff 87: ens2f0v12: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether ee:6b:5f:a6:b6:77 brd ff:ff:ff:ff:ff:ff 88: ens2f0v13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 36:4a:72:67:0e:48 brd ff:ff:ff:ff:ff:ff 89: ens2f0v14: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 52:fb:79:5b:9b:04 brd ff:ff:ff:ff:ff:ff 90: ens2f0v15: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 16:74:5c:4e:c7:37 brd ff:ff:ff:ff:ff:ff 91: ens2f0v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether ea:61:11:ff:1f:eb brd ff:ff:ff:ff:ff:ff 92: ens2f0v3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether de:e5:7b:06:6d:b0 brd ff:ff:ff:ff:ff:ff 93: ens2f0v4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 72:c3:e5:7e:ff:7d brd ff:ff:ff:ff:ff:ff 94: ens2f0v5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 6a:8f:43:cc:58:a9 brd ff:ff:ff:ff:ff:ff 95: ens2f0v6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether b6:a4:02:68:81:59 brd ff:ff:ff:ff:ff:ff 96: ens2f0v7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether ae:34:e7:18:2d:a6 brd ff:ff:ff:ff:ff:ff 97: ens2f0v8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 96:e3:87:d5:05:d4 brd ff:ff:ff:ff:ff:ff 98: ens2f0v9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 9a:27:c6:2b:1e:ed brd ff:ff:ff:ff:ff:ff 99: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 1c:34:da:7d:80:4a brd ff:ff:ff:ff:ff:ff 100: dfa71fd24fa169c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000 Error: Peer netns reference is invalid. Error: Peer netns reference is invalid. link/ether 1a:6e:d0:b0:2a:d7 brd ff:ff:ff:ff:ff:ff link-netns 4054a031-0bfc-4701-a758-21f24a41c2e3 101: 7ea9e919ffd19bc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether ae:fd:5b:9c:ea:53 brd ff:ff:ff:ff:ff:ff link-netns 13977501-1b7e-4cd6-9a37-738785f14a97 102: 579684edff3e267@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether 16:b2:ca:89:14:57 brd ff:ff:ff:ff:ff:ff link-netns da784952-a392-43bb-8c29-196c1d8d044c 103: 30e8e733f48253e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether 6e:29:9c:10:d7:3e brd ff:ff:ff:ff:ff:ff link-netns ecba5b96-433c-4e01-beb4-3e2039a52642 104: a8b1e3395761da0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether 62:83:54:59:39:6c brd ff:ff:ff:ff:ff:ff link-netns a999c7a5-3046-47ba-b1b8-0bae7a209fb3 105: 59e94160323cfb9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether b6:ed:5a:40:39:39 brd ff:ff:ff:ff:ff:ff link-netns 24171bf5-f7f1-4d28-ba31-b831ab0fd76b 106: 7762577b21402b0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether 4e:fd:f9:08:9c:66 brd ff:ff:ff:ff:ff:ff link-netns 4b1064de-3ca1-497d-a234-4c7eea3fbb7a 107: b2a57d60bab4561@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000 link/ether 02:e0:bc:55:56:b2 brd ff:ff:ff:ff:ff:ff link-netns 6474e1b0-4108-48ca-805a-82ffd8965373 sh-4.4#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056