+++ This bug was initially created as a clone of Bug #1771984 +++ Description of problem: when VF is init then restart sriov operator pod and config daemon pod by manaul. after the pods are running. found the VF resource will cannot be used. Version-Release number of selected component (if applicable): quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.3.0-201911120917-ose-sriov-network-operator quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.3.0-201911120317-ose-sriov-network-config-daemon How reproducible: always Steps to Reproduce: 1. setup sriov operator 2. init vf apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: mlx277-netdevice namespace: openshift-sriov-network-operator spec: mtu: 1500 nicSelector: pfNames: - ens2f0 rootDevices: - '0000:60:00.0' vendor: '15b3' nodeSelector: feature.node.kubernetes.io/sriov-capable: 'true' numVfs: 2 resourceName: mlx277netdevice 3. create sriovnetwork apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetwork metadata: name: mlx277-sriovnetwork namespace: openshift-sriov-network-operator spec: ipam: | { "type": "host-local", "subnet": "10.56.217.0/24", "rangeStart": "10.56.217.171", "rangeEnd": "10.56.217.181", "routes": [{ "dst": "0.0.0.0/0" }], "gateway": "10.56.217.1" } vlan: 0 spoofChk: "on" trust: "off" resourceName: mlx277netdevice networkNamespace: z1 3. Create pod in namespaces z1 apiVersion: v1 kind: Pod metadata: generateName: testpod1 labels: env: test annotations: k8s.v1.cni.cncf.io/networks: mlx277-sriovnetwork spec: containers: - name: test-pod image: bmeng/centos-network resources: limits: memory: 340Mi cpu: "1" requests: cpu: 500m 4. Check pod can be running 5. Then restart the sriov operator pod and config daemon pod 6. Find the pod created in step 3 is deleted 7. Create pod again with step 3 Actual results: oc get pod -n z1 NAME READY STATUS RESTARTS AGE testpod1vkxc6 0/1 Pending 0 38m node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling <unknown> default-scheduler 0/3 nodes are available: 1 node(s) were unschedulable, 2 Insufficient openshift.io/mlx277netdevice. Warning FailedScheduling <unknown> default-scheduler 0/3 nodes are available: 1 node(s) were unschedulable, 2 Insufficient openshift.io/mlx277netdevice #oc get node dell-per740-13.rhts.eng.pek2.redhat.com -o yaml | grep "mlx277netdevice" openshift.io/mlx277netdevice: "2" openshift.io/mlx277netdevice: "2" Expected results: Additional info: --- Additional comment from zhaozhanqi on 2019-11-13 10:36:54 UTC --- oc logs sriov-network-config-daemon-lk982 -f I1113 10:35:48.805221 621466 utils.go:256] tryGetInterfaceName(): name is ens1f0 I1113 10:35:48.805290 621466 utils.go:256] tryGetInterfaceName(): name is ens1f0 I1113 10:35:48.805710 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.0 I1113 10:35:48.805753 621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v0 I1113 10:35:48.805796 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.1 I1113 10:35:48.805834 621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v1 I1113 10:35:48.805887 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.2 I1113 10:35:48.805930 621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v2 I1113 10:35:48.805969 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.3 I1113 10:35:48.806006 621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v3 I1113 10:35:48.806054 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.4 I1113 10:35:48.806089 621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v4 I1113 10:35:48.806142 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:00.1 I1113 10:35:48.806183 621466 utils.go:256] tryGetInterfaceName(): name is ens1f1 I1113 10:35:48.806242 621466 utils.go:256] tryGetInterfaceName(): name is ens1f1 I1113 10:35:48.806397 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.0 I1113 10:35:48.806445 621466 utils.go:256] tryGetInterfaceName(): name is ens3f0 I1113 10:35:48.806510 621466 utils.go:256] tryGetInterfaceName(): name is ens3f0 I1113 10:35:48.806822 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.2 I1113 10:35:48.806874 621466 utils.go:256] tryGetInterfaceName(): name is ens3f0v0 I1113 10:35:48.806916 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.3 I1113 10:35:48.806957 621466 utils.go:256] tryGetInterfaceName(): name is ens3f0v1 I1113 10:35:48.807006 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.1 I1113 10:35:48.807047 621466 utils.go:256] tryGetInterfaceName(): name is ens3f1 I1113 10:35:48.807106 621466 utils.go:256] tryGetInterfaceName(): name is ens3f1 I1113 10:35:48.807372 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.4 I1113 10:35:48.807413 621466 utils.go:256] tryGetInterfaceName(): name is ens3f1v0 I1113 10:35:48.807452 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.5 I1113 10:35:48.807494 621466 utils.go:256] tryGetInterfaceName(): name is ens3f1v1 I1113 10:35:48.807575 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:60:00.0 I1113 10:35:48.807614 621466 utils.go:256] tryGetInterfaceName(): name is ens2f0 I1113 10:35:48.807688 621466 utils.go:256] tryGetInterfaceName(): name is ens2f0 I1113 10:35:48.807955 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:60:00.2 I1113 10:35:48.807995 621466 utils.go:256] tryGetInterfaceName(): name is ens2f0v0 I1113 10:35:48.808043 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:60:00.3 I1113 10:35:48.808072 621466 utils.go:256] tryGetInterfaceName(): name is ens2f0v1 I1113 10:35:48.808118 621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:60:00.1 I1113 10:35:48.808150 621466 utils.go:256] tryGetInterfaceName(): name is ens2f1 I1113 10:35:48.808196 621466 utils.go:256] tryGetInterfaceName(): name is ens2f1 I1113 10:35:48.826016 621466 writer.go:107] setNodeStateStatus(): syncStatus: InProgress, lastSyncError: --- Additional comment from Peng Liu on 2019-11-13 13:11:36 UTC --- The sriov network config daemon hanged, and following dmsg log was found on the node. It looks the NIC was in abnormal status, which blocked the config daemon to configure the number of VF. [ 6784.667959] i40e 0000:3b:00.0: Setting VLAN 1, QOS 0x2 on VF 4 [ 6784.674635] i40e 0000:3b:00.0: Setting MAC ca:fe:c0:ff:ee:02 on VF 4 [ 6784.681128] iavf 0000:3b:02.4: Reset warning received from the PF [ 6784.687310] iavf 0000:3b:02.4: Scheduling reset task [ 6784.772599] i40e 0000:3b:00.0: Bring down and up the VF interface to make this change effective. [ 6784.781525] i40e 0000:3b:00.0: Invalid min tx rate (40) (greater than 0) specified for VF 4. [ 6784.829649] i40e 0000:3b:00.0: VF attempting to override administratively set MAC address, bring down and up the VF interface to resume normal operation [ 6784.843352] i40e 0000:3b:00.0: VF 4 failed opcode 10, retval: -1 [ 6784.849501] iavf 0000:3b:02.4: Failed to add MAC filter, error I40E_ERR_NVM [ 6784.941643] device veth88e76ab6 left promiscuous mode [ 6786.819082] SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue) [ 6792.617003] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 6792.633942] IPv6: ADDRCONF(NETDEV_UP): vetha0e6c895: link is not ready [ 6792.640770] IPv6: ADDRCONF(NETDEV_CHANGE): vetha0e6c895: link becomes ready [ 6792.647875] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6792.669191] device vetha0e6c895 entered promiscuous mode [ 6794.259815] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending. [ 6794.267535] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending. [ 6794.275221] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending. [ 6794.282904] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending. [ 6794.290590] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending. [ 6794.298422] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending. --- Additional comment from Peng Liu on 2019-11-25 14:03:37 UTC --- min tx rate is not supported by Intel NICs. Configuring this parameter will lead i40e driver to an abnormal state. https://bugzilla.redhat.com/show_bug.cgi?id=1772815 has been created to track this issue from kernel side. A note has been added to the OCP document to prevent users from configuring this parameter for Intel NICs.
Verified this bug with quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.3.0-201911272116-ose-sriov-network-config-daemon quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.3.0-201911272116-ose-sriov-network-operator
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062