Bug 2041734

Summary:	ovs hwol: VFs are unbind when switchdev mode is enabled
Product:	OpenShift Container Platform	Reporter:	zenghui.shi <zshi>
Component:	Networking	Assignee:	zenghui.shi <zshi>
Networking sub component:	SR-IOV	QA Contact:	Ying Wang <yingwang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high
Version:	4.10
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-10 16:40:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description zenghui.shi 2022-01-18 08:00:44 UTC

Description of problem:

OVS hardware offload in OCP 4.10

When switchdev mode is enabled for Mellanox CX-5 NIC, VF netdev are not bind to mlx5_core driver after node rebooting (triggered by SR-IOV Operator), VF representor devices are not affected.



Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-11-065245

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 zenghui.shi 2022-01-18 08:03:47 UTC

Assuming PF interface is ens8f0, VF netdev are renamed as ens8f0vX and VF representor are renamed as ens8f0_X after node rebooting by udev rules, where X is the VF index number.

Comment 3 Ying Wang 2022-01-20 08:10:06 UTC

Verified on version below. Enabled switchdev and created sriovnetworknodepolicy on mlx5 port ens2f0, checked 'ip link show' on worker node, ens2f0vX interfaces were observable.

[root@f33-h13-000-r640 offload_test]# oc version 
Client Version: 4.10.0-0.nightly-2022-01-17-023213
Server Version: 4.10.0-0.nightly-2022-01-17-023213
Kubernetes Version: v1.23.0+60f5a1c
[root@f33-h13-000-r640 offload_test]# oc get csv
NAME                                         DISPLAY                   VERSION               REPLACES   PHASE
sriov-network-operator.4.10.0-202201181018   SR-IOV Network Operator   4.10.0-202201181018              Succeeded
[root@f33-h13-000-r640 offload_test]# 


# more policy-mlx.yaml 
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: policy-mlx
  namespace: openshift-sriov-network-operator
spec:
  deviceType: netdevice
  eSwitchMode: "switchdev"
  nicSelector:
    deviceID: "1017"
    rootDevices:
    - 0000:5e:00.0
    vendor: "15b3"
    pfNames:
    - ens2f0
  nodeSelector:
    feature.node.kubernetes.io/sriov-capable: "true"
  numVfs: 16
  priority: 5
  resourceName: mlxnics


sh-4.4# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b0:26:28:d8:b6:a0 brd ff:ff:ff:ff:ff:ff
3: eno2np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether b0:26:28:d8:b6:a1 brd ff:ff:ff:ff:ff:ff
4: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 40:a6:b7:0c:1a:50 brd ff:ff:ff:ff:ff:ff
5: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 40:a6:b7:0c:1a:51 brd ff:ff:ff:ff:ff:ff
7: ens2f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 1c:34:da:7d:80:4b brd ff:ff:ff:ff:ff:ff
24: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether 1c:34:da:7d:80:4a brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 8     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 9     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 10     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 11     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 12     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 13     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 14     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
    vf 15     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state disable, trust off, query_rss off
25: ens2f0_0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 66:23:e1:3e:20:81 brd ff:ff:ff:ff:ff:ff
26: ens2f0_1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 7e:5c:7d:31:9f:e9 brd ff:ff:ff:ff:ff:ff
27: ens2f0_2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 8e:c9:ca:e4:c9:c2 brd ff:ff:ff:ff:ff:ff
28: ens2f0_3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether a6:a9:eb:ff:08:ff brd ff:ff:ff:ff:ff:ff
29: ens2f0_4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:4c:3e:dd:03:24 brd ff:ff:ff:ff:ff:ff
30: ens2f0_5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether da:c3:c0:f9:3b:5b brd ff:ff:ff:ff:ff:ff
31: ens2f0_6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 62:d9:26:46:8b:df brd ff:ff:ff:ff:ff:ff
32: ens2f0_7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ea:21:28:a1:00:57 brd ff:ff:ff:ff:ff:ff
33: ens2f0_8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether da:22:e9:88:b9:10 brd ff:ff:ff:ff:ff:ff
34: ens2f0_9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether d6:5a:18:19:8d:a8 brd ff:ff:ff:ff:ff:ff
35: ens2f0_10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 7e:60:0a:d6:43:f9 brd ff:ff:ff:ff:ff:ff
36: ens2f0_11: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 4a:3f:4c:e9:f2:11 brd ff:ff:ff:ff:ff:ff
37: ens2f0_12: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ba:ca:c5:c9:e7:ce brd ff:ff:ff:ff:ff:ff
38: ens2f0_13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 46:45:01:ae:0e:ae brd ff:ff:ff:ff:ff:ff
39: ens2f0_14: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether a6:13:4f:5d:15:4a brd ff:ff:ff:ff:ff:ff
40: ens2f0_15: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 06:e2:52:52:a4:eb brd ff:ff:ff:ff:ff:ff
41: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 0a:23:b9:8e:18:7a brd ff:ff:ff:ff:ff:ff
42: vxlan_sys_4789: <BROADCAST,MULTICAST> mtu 65000 qdisc noqueue master ovs-system state DOWN mode DEFAULT group default qlen 1000
    link/ether 56:c4:39:f7:0b:1b brd ff:ff:ff:ff:ff:ff
43: br-ext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fa:89:a5:56:09:4f brd ff:ff:ff:ff:ff:ff
44: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ee:31:a4:17:43:4a brd ff:ff:ff:ff:ff:ff
45: ovn-k8s-mp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 1e:6c:42:66:e8:cd brd ff:ff:ff:ff:ff:ff
46: br-int: <BROADCAST,MULTICAST> mtu 1400 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 16:13:2a:76:53:cf brd ff:ff:ff:ff:ff:ff
83: ens2f0v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ce:71:c6:d8:49:62 brd ff:ff:ff:ff:ff:ff
84: ens2f0v1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 22:c1:af:c0:af:b1 brd ff:ff:ff:ff:ff:ff
85: ens2f0v10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether d2:af:5d:3e:33:ea brd ff:ff:ff:ff:ff:ff
86: ens2f0v11: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 22:86:ae:15:61:31 brd ff:ff:ff:ff:ff:ff
87: ens2f0v12: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ee:6b:5f:a6:b6:77 brd ff:ff:ff:ff:ff:ff
88: ens2f0v13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 36:4a:72:67:0e:48 brd ff:ff:ff:ff:ff:ff
89: ens2f0v14: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:fb:79:5b:9b:04 brd ff:ff:ff:ff:ff:ff
90: ens2f0v15: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 16:74:5c:4e:c7:37 brd ff:ff:ff:ff:ff:ff
91: ens2f0v2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ea:61:11:ff:1f:eb brd ff:ff:ff:ff:ff:ff
92: ens2f0v3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether de:e5:7b:06:6d:b0 brd ff:ff:ff:ff:ff:ff
93: ens2f0v4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 72:c3:e5:7e:ff:7d brd ff:ff:ff:ff:ff:ff
94: ens2f0v5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 6a:8f:43:cc:58:a9 brd ff:ff:ff:ff:ff:ff
95: ens2f0v6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether b6:a4:02:68:81:59 brd ff:ff:ff:ff:ff:ff
96: ens2f0v7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ae:34:e7:18:2d:a6 brd ff:ff:ff:ff:ff:ff
97: ens2f0v8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 96:e3:87:d5:05:d4 brd ff:ff:ff:ff:ff:ff
98: ens2f0v9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 9a:27:c6:2b:1e:ed brd ff:ff:ff:ff:ff:ff
99: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 1c:34:da:7d:80:4a brd ff:ff:ff:ff:ff:ff
100: dfa71fd24fa169c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
Error: Peer netns reference is invalid.
Error: Peer netns reference is invalid.
    link/ether 1a:6e:d0:b0:2a:d7 brd ff:ff:ff:ff:ff:ff link-netns 4054a031-0bfc-4701-a758-21f24a41c2e3
101: 7ea9e919ffd19bc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether ae:fd:5b:9c:ea:53 brd ff:ff:ff:ff:ff:ff link-netns 13977501-1b7e-4cd6-9a37-738785f14a97
102: 579684edff3e267@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether 16:b2:ca:89:14:57 brd ff:ff:ff:ff:ff:ff link-netns da784952-a392-43bb-8c29-196c1d8d044c
103: 30e8e733f48253e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether 6e:29:9c:10:d7:3e brd ff:ff:ff:ff:ff:ff link-netns ecba5b96-433c-4e01-beb4-3e2039a52642
104: a8b1e3395761da0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether 62:83:54:59:39:6c brd ff:ff:ff:ff:ff:ff link-netns a999c7a5-3046-47ba-b1b8-0bae7a209fb3
105: 59e94160323cfb9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether b6:ed:5a:40:39:39 brd ff:ff:ff:ff:ff:ff link-netns 24171bf5-f7f1-4d28-ba31-b831ab0fd76b
106: 7762577b21402b0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether 4e:fd:f9:08:9c:66 brd ff:ff:ff:ff:ff:ff link-netns 4b1064de-3ca1-497d-a234-4c7eea3fbb7a
107: b2a57d60bab4561@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
    link/ether 02:e0:bc:55:56:b2 brd ff:ff:ff:ff:ff:ff link-netns 6474e1b0-4108-48ca-805a-82ffd8965373
sh-4.4#

Comment 7 errata-xmlrpc 2022-03-10 16:40:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056