Bug 1654174
Summary: | [sriov-cni] Non-existed vf will be assigned to pod if disable the sriov feature on the PF when the sriovdp is running | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Meng Bo <bmeng> |
Component: | Networking | Assignee: | zenghui.shi <zshi> |
Networking sub component: | openshift-sdn | QA Contact: | zhaozhanqi <zzhao> |
Status: | CLOSED ERRATA | Docs Contact: | |
Severity: | low | ||
Priority: | medium | CC: | aos-bugs, bbennett, cdc, fpan, zshi |
Version: | 4.1.0 | ||
Target Milestone: | --- | ||
Target Release: | 4.2.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-10-16 06:27:40 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Meng Bo
2018-11-28 07:57:34 UTC
Thanks for testing and reporting the bug! I think this is a valid bug in sriov device plugin, because it doesn't probe the actual VF states to update device healthy state, instead it probes PF operstate and report it back to kubelet as VF healthy state. In order to solve this issue, we will need to auto discover the changes of devices(num of devices, newly added/deleted devices etc) when probing device state periodically and update the newly discovered devices to kubelet. but this also has some potential problems: 1) At least with intel NIC, changing VF num requires setting VF num to 0 first, this may interrupt any running workload on existing containers which already have device allocated. I'm not sure this is a normal action of cluster admin to re-created those VFs without restarting those workloads. Also currently there is no garbage collect for devices in this circumstance which means those devices will still be considered as allocated. (device will only be released by kubelet when the pod is in terminated state) 2) When VF is allocated to pod, it's likely moved to the pod network namespace which sriov device plugin is not in. this may cause a problem for device plugin to get access to the state of actual VF device. will talk with Intel and see how we can address this issue. update: had some discussion internally about how we expect this to work in ocp-4.0: we assume the num of VFs will not be changed on the fly after sriov device plugin is launched, this excludes cases such as: 1) VF num is changed to 0 and then to the new desired num 2) new PF device is added or deleted on the host after sriov device plugin is launched. but we are exploring and planning to support changing VF configuration as part of machine config process which happens before sriov device plugin is launched. For example, network admin can specify the num of VFs they want to create for each PF on the host via machine configuration, MCD(machine config daemon) running on each host applies these VF configuration and reboots node whenever the machine configuration is changed. This allows the sriov-device-plugin to get restarted and re-discover all the changed devices without having to re-discover the devices on the fly. With SR-IOV Operator introduced in 4.2, all the configuration and provisioning of SR-IOV devices shall be done via SR-IOV Operator. SR-IOV device plugin daemon will be restarted by Operator once the number of SR-IOV devices are changed. verified this bug on quay.io/openshift-release-dev/ocp-v4.0-art-dev:v4.2.0-201910070933-ose-sriov-network-operator When the networknodepolicy is created with 4 vfNums, pod can request the specified Vf when the networknodepolicy is deleted. create pod and still request resource. the pod will be pending. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922 |