Bug 1776604 - [sriov] VF resource will not be used after restart sriov operator pod and config daemon pod by manual
Summary: [sriov] VF resource will not be used after restart sriov operator pod and con...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.3.0
Hardware: All
OS: All
high
high
Target Milestone: ---
: 4.3.0
Assignee: Peng Liu
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On: 1771984
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-26 04:50 UTC by Peng Liu
Modified: 2020-01-23 11:14 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1771984
Environment:
Last Closed: 2020-01-23 11:14:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift sriov-network-operator pull 128 0 'None' closed [release-4.3] Bug 1776604: Optimize node draining condition 2020-06-11 23:22:37 UTC
Red Hat Product Errata RHBA-2020:0062 0 None None None 2020-01-23 11:14:31 UTC

Description Peng Liu 2019-11-26 04:50:44 UTC
+++ This bug was initially created as a clone of Bug #1771984 +++

Description of problem:
when VF is init then restart sriov operator pod and config daemon pod by manaul. after the pods are running. found the VF resource will cannot be used.

Version-Release number of selected component (if applicable):
quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.3.0-201911120917-ose-sriov-network-operator
quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.3.0-201911120317-ose-sriov-network-config-daemon

How reproducible:
always

Steps to Reproduce:
1. setup sriov operator
2. init vf
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: mlx277-netdevice
  namespace: openshift-sriov-network-operator
spec:
  mtu: 1500
  nicSelector:
    pfNames:
      - ens2f0
    rootDevices:
      - '0000:60:00.0'
    vendor: '15b3'
  nodeSelector:
    feature.node.kubernetes.io/sriov-capable: 'true'
  numVfs: 2
  resourceName: mlx277netdevice
3. create sriovnetwork
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: mlx277-sriovnetwork
  namespace: openshift-sriov-network-operator
spec:
  ipam: |
    {
      "type": "host-local",
      "subnet": "10.56.217.0/24",
      "rangeStart": "10.56.217.171",
      "rangeEnd": "10.56.217.181",
      "routes": [{
        "dst": "0.0.0.0/0"
      }],
      "gateway": "10.56.217.1"
    }
  vlan: 0
  spoofChk: "on"
  trust: "off"
  resourceName: mlx277netdevice
  networkNamespace: z1
3. Create pod in namespaces z1
apiVersion: v1
kind: Pod
metadata:
  generateName: testpod1
  labels:
    env: test
  annotations:
    k8s.v1.cni.cncf.io/networks: mlx277-sriovnetwork
spec:
  containers:
  - name: test-pod
    image: bmeng/centos-network
    resources:
      limits:
        memory: 340Mi
        cpu: "1"
      requests:
        cpu: 500m
4. Check pod can be running
5. Then restart the sriov operator pod and config daemon pod
6. Find the pod created in step 3 is deleted
7. Create pod again with step 3

Actual results:
 oc get pod -n z1
NAME            READY   STATUS    RESTARTS   AGE
testpod1vkxc6   0/1     Pending   0          38m

                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  0/3 nodes are available: 1 node(s) were unschedulable, 2 Insufficient openshift.io/mlx277netdevice.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/3 nodes are available: 1 node(s) were unschedulable, 2 Insufficient openshift.io/mlx277netdevice

 #oc get node dell-per740-13.rhts.eng.pek2.redhat.com -o yaml | grep "mlx277netdevice"
    openshift.io/mlx277netdevice: "2"
    openshift.io/mlx277netdevice: "2"

Expected results:


Additional info:

--- Additional comment from zhaozhanqi on 2019-11-13 10:36:54 UTC ---

oc logs sriov-network-config-daemon-lk982 -f

I1113 10:35:48.805221  621466 utils.go:256] tryGetInterfaceName(): name is ens1f0
I1113 10:35:48.805290  621466 utils.go:256] tryGetInterfaceName(): name is ens1f0
I1113 10:35:48.805710  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.0
I1113 10:35:48.805753  621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v0
I1113 10:35:48.805796  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.1
I1113 10:35:48.805834  621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v1
I1113 10:35:48.805887  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.2
I1113 10:35:48.805930  621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v2
I1113 10:35:48.805969  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.3
I1113 10:35:48.806006  621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v3
I1113 10:35:48.806054  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:02.4
I1113 10:35:48.806089  621466 utils.go:256] tryGetInterfaceName(): name is ens1f0v4
I1113 10:35:48.806142  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:3b:00.1
I1113 10:35:48.806183  621466 utils.go:256] tryGetInterfaceName(): name is ens1f1
I1113 10:35:48.806242  621466 utils.go:256] tryGetInterfaceName(): name is ens1f1
I1113 10:35:48.806397  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.0
I1113 10:35:48.806445  621466 utils.go:256] tryGetInterfaceName(): name is ens3f0
I1113 10:35:48.806510  621466 utils.go:256] tryGetInterfaceName(): name is ens3f0
I1113 10:35:48.806822  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.2
I1113 10:35:48.806874  621466 utils.go:256] tryGetInterfaceName(): name is ens3f0v0
I1113 10:35:48.806916  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.3
I1113 10:35:48.806957  621466 utils.go:256] tryGetInterfaceName(): name is ens3f0v1
I1113 10:35:48.807006  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.1
I1113 10:35:48.807047  621466 utils.go:256] tryGetInterfaceName(): name is ens3f1
I1113 10:35:48.807106  621466 utils.go:256] tryGetInterfaceName(): name is ens3f1
I1113 10:35:48.807372  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.4
I1113 10:35:48.807413  621466 utils.go:256] tryGetInterfaceName(): name is ens3f1v0
I1113 10:35:48.807452  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:5e:00.5
I1113 10:35:48.807494  621466 utils.go:256] tryGetInterfaceName(): name is ens3f1v1
I1113 10:35:48.807575  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:60:00.0
I1113 10:35:48.807614  621466 utils.go:256] tryGetInterfaceName(): name is ens2f0
I1113 10:35:48.807688  621466 utils.go:256] tryGetInterfaceName(): name is ens2f0
I1113 10:35:48.807955  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:60:00.2
I1113 10:35:48.807995  621466 utils.go:256] tryGetInterfaceName(): name is ens2f0v0
I1113 10:35:48.808043  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:60:00.3
I1113 10:35:48.808072  621466 utils.go:256] tryGetInterfaceName(): name is ens2f0v1
I1113 10:35:48.808118  621466 utils.go:261] getNetdevMTU(): get MTU for device 0000:60:00.1
I1113 10:35:48.808150  621466 utils.go:256] tryGetInterfaceName(): name is ens2f1
I1113 10:35:48.808196  621466 utils.go:256] tryGetInterfaceName(): name is ens2f1
I1113 10:35:48.826016  621466 writer.go:107] setNodeStateStatus(): syncStatus: InProgress, lastSyncError:

--- Additional comment from Peng Liu on 2019-11-13 13:11:36 UTC ---

The sriov network config daemon hanged, and following dmsg log was found on the node. It looks the NIC was in abnormal status, which blocked the config daemon to configure the number of VF.

[ 6784.667959] i40e 0000:3b:00.0: Setting VLAN 1, QOS 0x2 on VF 4
[ 6784.674635] i40e 0000:3b:00.0: Setting MAC ca:fe:c0:ff:ee:02 on VF 4
[ 6784.681128] iavf 0000:3b:02.4: Reset warning received from the PF
[ 6784.687310] iavf 0000:3b:02.4: Scheduling reset task
[ 6784.772599] i40e 0000:3b:00.0: Bring down and up the VF interface to make this change effective.
[ 6784.781525] i40e 0000:3b:00.0: Invalid min tx rate (40) (greater than 0) specified for VF 4.
[ 6784.829649] i40e 0000:3b:00.0: VF attempting to override administratively set MAC address, bring down and up the VF interface to resume normal operation
[ 6784.843352] i40e 0000:3b:00.0: VF 4 failed opcode 10, retval: -1
[ 6784.849501] iavf 0000:3b:02.4: Failed to add MAC filter, error I40E_ERR_NVM
[ 6784.941643] device veth88e76ab6 left promiscuous mode
[ 6786.819082] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[ 6792.617003] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 6792.633942] IPv6: ADDRCONF(NETDEV_UP): vetha0e6c895: link is not ready
[ 6792.640770] IPv6: ADDRCONF(NETDEV_CHANGE): vetha0e6c895: link becomes ready
[ 6792.647875] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 6792.669191] device vetha0e6c895 entered promiscuous mode
[ 6794.259815] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending.
[ 6794.267535] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending.
[ 6794.275221] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending.
[ 6794.282904] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending.
[ 6794.290590] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending.
[ 6794.298422] i40e 0000:3b:00.0: Unable to configure VFs, other operation is pending.

--- Additional comment from Peng Liu on 2019-11-25 14:03:37 UTC ---

min tx rate is not supported by Intel NICs. Configuring this parameter will lead i40e driver to an abnormal state. https://bugzilla.redhat.com/show_bug.cgi?id=1772815 has been created to track this issue from kernel side.

A note has been added to the OCP document to prevent users from configuring this parameter for Intel NICs.

Comment 2 zhaozhanqi 2019-11-28 07:09:15 UTC
Verified this bug with 
  quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.3.0-201911272116-ose-sriov-network-config-daemon
 quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.3.0-201911272116-ose-sriov-network-operator

Comment 4 errata-xmlrpc 2020-01-23 11:14:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.