Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1963951

Summary: cannot use node selector with node.kubernetes.io/role: <role>
Product: OpenShift Container Platform Reporter: Yuval Kashtan <ykashtan>
Component: Performance Addon OperatorAssignee: Artyom <alukiano>
Status: CLOSED ERRATA QA Contact: Shereen Haj Makhoul <shajmakh>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.8CC: alukiano, aos-bugs, eglottma, grajaiya, mniranja, vgrinber
Target Milestone: ---   
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-18 19:58:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
oc logs -n openshift-performance-addon-operator performance-operator-7b7cc4b957-zks8p | grep metal-bug > performance-operator.metal-bug.log none

Description Yuval Kashtan 2021-05-24 13:14:36 UTC 
Created attachment 1786384 [details]
oc logs -n openshift-performance-addon-operator performance-operator-7b7cc4b957-zks8p | grep metal-bug > performance-operator.metal-bug.log

Description of problem:
if, for example, I'm creating an mcp with the selector node.kubernetes.io/role: rwn
then creating a v2 PP
results in mc 50-XX with wrong label:
```
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: role
  name: 50-performance-worker-metal-rwn
spec:
  config:
    ignition:
      version: 3.2.0
```
and node not having the right cpusets 


we see nothing special in either PAO or MCO logs.

Version-Release number of selected component (if applicable):
latest 4.8

How reproducible:
everytime 


issue is summarized and described here:
https://docs.google.com/document/d/1b-DkMtW5Uk0mqR01hbaM_eJlou0GfSYb0-rM8i_hhdM

in mco controller logs we can see:
``E0524 13:10:24.048877       1 render_controller.go:217] error finding pools for machineconfig: could │···not find any MachineConfigPool set for MachineConfig 50-performance-worker-metal-bug with labels: map│···
[machineconfiguration.openshift.io/role:role]
````

attached performance-operator log
Comment 5 Niranjan Mallapadi Raghavender 2021-10-13 10:57:24 UTC 
Versiosn:
[root@dell-r640-014 1963951]# oc version
Client Version: 4.8.13
Server Version: 4.9.0-rc.6
Kubernetes Version: v1.22.0-rc.0+9d56495


PAO Version: NVR=v4.9.0-19


1. Create Machine config as shown below:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker-rwn
  name: worker-rwn
spec:
  machineConfigSelector:
    matchExpressions:
    - key: machineconfiguration.openshift.io/role
      operator: In
      values:
      - worker-rwn
      - worker
  nodeSelector:
    matchLabels:
      node.kubernetes.io/role: rwn


2. Apply the machine config:

oc apply -f mcp.yaml

NAME         CONFIG                                                 UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master       rendered-master-ab34921e97ba1a71d4bc6322b74ef604       True      False      False      3              3                   3                     0                      3h54m
worker       rendered-worker-c555deaa699c7d1b27d3e1ad3af8bf3c       True      False      False      3              3                   3                     0                      3h54m
worker-rwn   rendered-worker-rwn-c555deaa699c7d1b27d3e1ad3af8bf3c   True      False      False      0              0                   0                     0                      61m


3. Apply labels to one of the worker nodes:
oc label --overwrite  node/ocp49-worker-0.demo.lab.mniranja node-role.kubernetes.io/worker-rwn=""
oc label --overwrite  node/ocp49-worker-0.demo.lab.mniranja node-role.kubernetes.io/rwn=""

[root@dell-r640-014 1963951]# oc get nodes
NAME                               STATUS   ROLES                   AGE     VERSION
ocp49-master-0.demo.lab.mniranja   Ready    master                  3h57m   v1.22.0-rc.0+9d56495
ocp49-master-1.demo.lab.mniranja   Ready    master                  3h57m   v1.22.0-rc.0+9d56495
ocp49-master-2.demo.lab.mniranja   Ready    master                  3h56m   v1.22.0-rc.0+9d56495
ocp49-worker-0.demo.lab.mniranja   Ready    rwn,worker,worker-rwn   3h44m   v1.22.0-rc.0+9d56495
ocp49-worker-1.demo.lab.mniranja   Ready    worker                  3h47m   v1.22.0-rc.0+9d56495
ocp49-worker-2.demo.lab.mniranja   Ready    worker                  3h42m   v1.22.0-rc.0+9d56495


4. Create performance profile as show below:

[root@dell-r640-014 1963951]# cat performance_profile.yaml 
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: worker-rwn
spec:
  cpu:
    isolated: 2-12
    reserved: 0-1
  nodeSelector:
    node.kubernetes.io/role: worker-rwn
  numa:
    topologyPolicy: restricted


5. Apply the performance profile . and check the profile should not apply

oc apply -f performance_profile.yaml

6. The profile should be in degrated status with error that there are not matching labels. 
[root@dell-r640-014 1963951]# oc get performanceprofile/worker-rwn -o yaml
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  creationTimestamp: "2021-10-13T09:32:30Z"
  finalizers:
  - foreground-deletion
  generation: 1
  name: worker-rwn
  resourceVersion: "82674"
  uid: 477cc120-26fa-4c17-b808-cc3c564f8c02
spec:
  cpu:
    isolated: 2-12
    reserved: 0-1
  nodeSelector:
    node.kubernetes.io/role: worker-rwn
  numa:
    topologyPolicy: restricted
status:
  conditions:
  - lastHeartbeatTime: "2021-10-13T09:32:30Z"
    lastTransitionTime: "2021-10-13T09:32:30Z"
    status: "False"
    type: Available
  - lastHeartbeatTime: "2021-10-13T09:32:30Z"
    lastTransitionTime: "2021-10-13T09:32:30Z"
    status: "False"
    type: Upgradeable
  - lastHeartbeatTime: "2021-10-13T09:32:30Z"
    lastTransitionTime: "2021-10-13T09:32:30Z"
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2021-10-13T09:32:30Z"
    lastTransitionTime: "2021-10-13T09:32:30Z"
    message: failed to find MCP with the node selector that matches labels "node.kubernetes.io/role=worker-rwn"
    reason: GettingMachineConfigPoolFailed
    status: "True"
    type: Degraded
  runtimeClass: performance-worker-rwn
  tuned: openshift-cluster-node-tuning-operator/openshift-node-performance-worker-rwn


7. Modify the profile to use node.kubernetes.io/role: rwn and now the performance profile status should have 
proper message that performance profile doesn't match the mcp-pool. 


apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"performance.openshift.io/v2","kind":"PerformanceProfile","metadata":{"annotations":{},"name":"worker-rwn"},"spec":{"cpu":{"isolated":"2-12","reserved":"0-1"},"nodeSelector":{"node.kubernetes.io/role":"rwn"},"numa":{"topologyPolicy":"restricted"}}}
  creationTimestamp: "2021-10-13T09:32:30Z"
  finalizers:
  - foreground-deletion
  generation: 2
  name: worker-rwn
  resourceVersion: "98973"
  uid: 477cc120-26fa-4c17-b808-cc3c564f8c02
spec:
  cpu:
    isolated: 2-12
    reserved: 0-1
  nodeSelector:
    node.kubernetes.io/role: rwn
  numa:
    topologyPolicy: restricted
status:
  conditions:
  - lastHeartbeatTime: "2021-10-13T10:24:36Z"
    lastTransitionTime: "2021-10-13T10:24:36Z"
    status: "False"
    type: Available
  - lastHeartbeatTime: "2021-10-13T10:24:36Z"
    lastTransitionTime: "2021-10-13T10:24:36Z"
    status: "False"
    type: Upgradeable
  - lastHeartbeatTime: "2021-10-13T10:24:36Z"
    lastTransitionTime: "2021-10-13T10:24:36Z"
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2021-10-13T10:24:36Z"
    lastTransitionTime: "2021-10-13T10:24:36Z"
    message: the machine config labels map[machineconfiguration.openshift.io/role:role]
      generated from the profile.spec.nodeSelector map[node.kubernetes.io/role:rwn]
      do not match the MachineConfigPool "worker-rwn" machineConfigSelector "machineconfiguration.openshift.io/role
      in (worker,worker-rwn)"
    reason: BadMachineConfigLabels
    status: "True"
    type: Degraded
  runtimeClass: performance-worker-rwn
  tuned: openshift-cluster-node-tuning-operator/openshift-node-performance-worker-rwn

8. Fix the Profile and Machine configuration as shown below:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker-rwn
  name: worker-rwn
spec:
  machineConfigSelector:
    matchExpressions:
    - key: machineconfiguration.openshift.io/role
      operator: In
      values:
      - worker-rwn
      - worker
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker-rwn: ""


Profile:

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: worker-rwn
spec:
  cpu:
    isolated: 2-12
    reserved: 0-1
  nodeSelector:
    node-role.kubernetes.io/worker-rwn: ""
  numa:
    topologyPolicy: restricted


Verify that the kubeletConfig CR is created and pointing to appropriate MCP labels:

$ oc get kubeletConfig/performance-worker-rwn -o yaml | less

<snip>

    volumeStatsAggPeriod: 0s
  machineConfigPoolSelector:
    matchLabels:
      machineconfiguration.openshift.io/role: worker-rwn
</snip>
Comment 7 errata-xmlrpc 2021-10-18 19:58:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9 low-latency extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3846