Bug 1892167
| Summary: | [SR-IOV] SriovNetworkNodePolicies apply ignoring the spec.nodeSelector. | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Robin Cernin <rcernin> | |
| Component: | Networking | Assignee: | zenghui.shi <zshi> | |
| Networking sub component: | SR-IOV | QA Contact: | zhaozhanqi <zzhao> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | low | |||
| Priority: | unspecified | CC: | akaris, bbennett | |
| Version: | 4.6 | |||
| Target Milestone: | --- | |||
| Target Release: | 4.7.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | No Doc Update | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1900736 (view as bug list) | Environment: | ||
| Last Closed: | 2021-02-24 15:28:37 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1900736 | |||
|
Comment 2
Andreas Karis
2020-10-28 12:14:59 UTC
I reproduced this with the 4.6 downstream operator, as well:
~~~
[root@openshift-jumpserver-0 ~]# oc get csv
NAME DISPLAY VERSION REPLACES PHASE
sriov-network-operator.4.6.0-202010200139.p0 SR-IOV Network Operator 4.6.0-202010200139.p0 Succeeded
[root@openshift-jumpserver-0 ~]# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.6.1 True False 20h Cluster version is 4.6.1
[root@openshift-jumpserver-0 ~]#
~~~
~~~
[root@openshift-jumpserver-0 ~]# cat networkpolicy-netdevice.yaml
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: sriov-enp5s0f0-netdevice
namespace: openshift-sriov-network-operator
spec:
resourceName: enp5s0f0Netdev
nodeSelector:
kubernetes.io/hostname: openshift-worker-0
priority: 10
mtu: 1500
numVfs: 5
nicSelector:
pfNames: ["enp5s0f0"]
deviceType: "netdevice"
isRdma: false
[root@openshift-jumpserver-0 ~]# cat networkpolicy-netdevice2.yaml
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: sriov-enp5s0f1-netdevice
namespace: openshift-sriov-network-operator
spec:
resourceName: enp5s0f1Netdev
nodeSelector:
kubernetes.io/hostname: openshift-worker-1
priority: 10
mtu: 1500
numVfs: 6
nicSelector:
pfNames: ["enp5s0f1"]
deviceType: "netdevice"
isRdma: false
~~~
~~~
oc apply -f networkpolicy-netdevice.yaml
~~~
Wait ...
~~~
[root@openshift-jumpserver-0 ~]# echo === worker0 ===; oc get nodes openshift-worker-0 -o yaml | grep -i openshift.io ; echo === worker1 === ; oc get nodes openshift-worker-1 -o yaml | grep -i openshift.io
=== worker0 ===
machineconfiguration.openshift.io/currentConfig: rendered-worker-2f51e2b12f6daef29d5255f032ff70d0
machineconfiguration.openshift.io/desiredConfig: rendered-worker-2f51e2b12f6daef29d5255f032ff70d0
machineconfiguration.openshift.io/reason: ""
machineconfiguration.openshift.io/state: Done
sriovnetwork.openshift.io/state: Idle
node.openshift.io/os_id: rhcos
f:machineconfiguration.openshift.io/currentConfig: {}
f:machineconfiguration.openshift.io/desiredConfig: {}
f:machineconfiguration.openshift.io/reason: {}
f:machineconfiguration.openshift.io/state: {}
f:sriovnetwork.openshift.io/state: {}
f:node.openshift.io/os_id: {}
f:openshift.io/enp5s0f0Netdev: {}
f:openshift.io/enp5s0f1Netdev: {}
f:openshift.io/enp5s0f0Netdev: {}
f:openshift.io/enp5s0f1Netdev: {}
openshift.io/enp5s0f0Netdev: "5"
openshift.io/enp5s0f1Netdev: "0"
openshift.io/enp5s0f0Netdev: "5"
openshift.io/enp5s0f1Netdev: "0"
=== worker1 ===
machineconfiguration.openshift.io/currentConfig: rendered-worker-2f51e2b12f6daef29d5255f032ff70d0
machineconfiguration.openshift.io/desiredConfig: rendered-worker-2f51e2b12f6daef29d5255f032ff70d0
machineconfiguration.openshift.io/reason: ""
machineconfiguration.openshift.io/state: Done
sriovnetwork.openshift.io/state: Idle
node.openshift.io/os_id: rhcos
f:machineconfiguration.openshift.io/currentConfig: {}
f:machineconfiguration.openshift.io/desiredConfig: {}
f:machineconfiguration.openshift.io/reason: {}
f:machineconfiguration.openshift.io/state: {}
f:sriovnetwork.openshift.io/state: {}
f:node.openshift.io/os_id: {}
f:openshift.io/enp5s0f0Netdev: {}
f:openshift.io/enp5s0f1Netdev: {}
f:openshift.io/enp5s0f0Netdev: {}
f:openshift.io/enp5s0f1Netdev: {}
openshift.io/enp5s0f0Netdev: "0"
openshift.io/enp5s0f1Netdev: "0"
openshift.io/enp5s0f0Netdev: "0"
openshift.io/enp5s0f1Netdev: "0"
~~~
~~~
oc apply -f networkpolicy-netdevice2.yaml
~~~
Wait ...
~~~
[root@openshift-jumpserver-0 ~]# echo === worker0 ===; oc get nodes openshift-worker-0 -o yaml | grep -i openshift.io ; echo === worker1 === ; oc get nodes openshift-worker-1 -o yaml | grep -i openshift.io
=== worker0 ===
machineconfiguration.openshift.io/currentConfig: rendered-worker-2f51e2b12f6daef29d5255f032ff70d0
machineconfiguration.openshift.io/desiredConfig: rendered-worker-2f51e2b12f6daef29d5255f032ff70d0
machineconfiguration.openshift.io/reason: ""
machineconfiguration.openshift.io/state: Done
sriovnetwork.openshift.io/state: Idle
node.openshift.io/os_id: rhcos
f:machineconfiguration.openshift.io/currentConfig: {}
f:machineconfiguration.openshift.io/desiredConfig: {}
f:machineconfiguration.openshift.io/reason: {}
f:machineconfiguration.openshift.io/state: {}
f:sriovnetwork.openshift.io/state: {}
f:node.openshift.io/os_id: {}
f:openshift.io/enp5s0f0Netdev: {}
f:openshift.io/enp5s0f1Netdev: {}
f:openshift.io/enp5s0f0Netdev: {}
f:openshift.io/enp5s0f1Netdev: {}
openshift.io/enp5s0f0Netdev: "5"
openshift.io/enp5s0f1Netdev: "1"
openshift.io/enp5s0f0Netdev: "5"
openshift.io/enp5s0f1Netdev: "1"
=== worker1 ===
machineconfiguration.openshift.io/currentConfig: rendered-worker-2f51e2b12f6daef29d5255f032ff70d0
machineconfiguration.openshift.io/desiredConfig: rendered-worker-2f51e2b12f6daef29d5255f032ff70d0
machineconfiguration.openshift.io/reason: ""
machineconfiguration.openshift.io/state: Done
sriovnetwork.openshift.io/state: Idle
node.openshift.io/os_id: rhcos
f:machineconfiguration.openshift.io/currentConfig: {}
f:machineconfiguration.openshift.io/desiredConfig: {}
f:machineconfiguration.openshift.io/reason: {}
f:machineconfiguration.openshift.io/state: {}
f:sriovnetwork.openshift.io/state: {}
f:node.openshift.io/os_id: {}
f:openshift.io/enp5s0f0Netdev: {}
f:openshift.io/enp5s0f1Netdev: {}
f:openshift.io/enp5s0f0Netdev: {}
f:openshift.io/enp5s0f1Netdev: {}
openshift.io/enp5s0f0Netdev: "1"
openshift.io/enp5s0f1Netdev: "6"
openshift.io/enp5s0f0Netdev: "1"
openshift.io/enp5s0f1Netdev: "6"
[root@openshift-jumpserver-0 ~]#
~~~
Two different policies ^^ will update Daemon Set selector:
oc get ds sriov-device-plugin -o yaml
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- openshift-worker-0
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- openshift-worker-1
They use same configMap
- configMap:
defaultMode: 420
name: device-plugin-config
oc get cm device-plugin-config -o yaml
config.json: '{"resourceList":[{"resourceName":"enp5s0f0Netdev","selectors":{"pfNames":["enp5s0f0"],"IsRdma":false},"SelectorObj":null},{"resourceName":"enp5s0f1Netdev","selectors":{"pfNames":["enp5s0f1"],"IsRdma":false},"SelectorObj":null}]}'
Then the device-plugin iterates thru it on each of the node: https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/blob/047fb351807c278d4c7dc3f5d25c876260b0560e/cmd/sriovdp/manager.go#L87
So yes nodeSelector in terms of running daemon set on certain nodes works well, but then shouldn't there be a unique config maps per each policy created?
Hi,
I talked to Robin, and he said that:
"Got response from dev, it is correct they use centralized configMap for all device plugin pods. Meaning that one node would have all the resources advertised if it contains that interface that matches with any other node policy.
3:17
So we have confirmation, but he is asking whether it is a problem. (edited)
3:18
So we can probably inform the customer this is expected and close the case."
But that honestly makes no sense. We are annotating per node the number of units of a given resource that the node has.
As you can see above, this overlap causes invalid accounting of resources:
worker-0:
openshift.io/enp5s0f0Netdev: "5"
openshift.io/enp5s0f1Netdev: "1"
worker-1:
openshift.io/enp5s0f0Netdev: "1"
openshift.io/enp5s0f1Netdev: "6"
Let's remember that this is the consequence of applying:
~~~
[root@openshift-jumpserver-0 ~]# cat networkpolicy-netdevice.yaml
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: sriov-enp5s0f0-netdevice
namespace: openshift-sriov-network-operator
spec:
resourceName: enp5s0f0Netdev
nodeSelector:
kubernetes.io/hostname: openshift-worker-0
priority: 10
mtu: 1500
numVfs: 5
nicSelector:
pfNames: ["enp5s0f0"]
deviceType: "netdevice"
isRdma: false
[root@openshift-jumpserver-0 ~]# cat networkpolicy-netdevice2.yaml
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: sriov-enp5s0f1-netdevice
namespace: openshift-sriov-network-operator
spec:
resourceName: enp5s0f1Netdev
nodeSelector:
kubernetes.io/hostname: openshift-worker-1
priority: 10
mtu: 1500
numVfs: 6
nicSelector:
pfNames: ["enp5s0f1"]
deviceType: "netdevice"
isRdma: false
~~~
So something funky is going on here and needs to be fixed. If it's not on a functional level, it does indeed affect what's reported to the operator. The result should be:
worker-0:
openshift.io/enp5s0f0Netdev: "5"
openshift.io/enp5s0f1Netdev: "0"
worker-1:
openshift.io/enp5s0f0Netdev: "0"
openshift.io/enp5s0f1Netdev: "6"
Or even better just plain:
worker-0:
openshift.io/enp5s0f0Netdev: "5"
worker-1:
openshift.io/enp5s0f0Netdev: "1"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633 |