Bug 1983182

Summary: After applying Machine Config, Node is stuck in "SchedulingDisabled " state forever
Product: OpenShift Container Platform Reporter: ygohar
Component: Machine Config OperatorAssignee: Yu Qi Zhang <jerzhang>
Machine Config Operator sub component: Machine Config Operator QA Contact: Rio Liu <rioliu>
Status: CLOSED INSUFFICIENT_DATA Docs Contact:
Severity: high    
Priority: unspecified CC: aos-bugs, mkrejci
Version: 4.6   
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Other   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-08 17:38:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description ygohar 2021-07-16 18:23:46 UTC
Description of problem:
machine configs failed to apply 3 node cluster. Node is stuck in SchedulingDisabled state. When do oc describe node, get the below error:
failed to drain node (5 tries): timed out waiting for the condition: error when waiting for pod "rook-ceph-osd-0-7bddc9c54f-bq52c" termina...''



Version-Release number of selected component (if applicable):
OCP 4.6.16

How reproducible:

Steps to Reproduce:
1. oc apply -f <machine.config.yaml>
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: systemreservedset
spec:
  kubeletConfig:
    cpuManagerPolicy: static
    cpuManagerReconcilePeriod: 5s
    topologyManagerPolicy: single-numa-node
    systemReserved:
      cpu: 4000m
      memory: 16Gi
  machineConfigPoolSelector:
    matchLabels:
      machineconfiguration.openshift.io/role: "master" // machine.config.yaml
2. new render generated and start processing 
3. Node is in schedulingDisabled state and never comes out of it

Actual results:
Node stuck in disabledScheduling state

Expected results:

MC render generated and implemented on all nodes
Additional info: