+++ This bug was initially created as a clone of Bug #2103786 +++
In 4.11, when we switched over the MCO to using controller to drain, we skip draining the MCC since we want it to finish the node it is working on.
In practice this means the master node the MCC runs on retains the MCC pod. The pod won't get restarted until the master node has finished os updates and reboots. In some setups, this could mean a long time for the MCO to stall while waiting for the master node to come up.
This may get resolved with graceful shutdown (?), or maybe I am misunderstanding how pod scheduling works alongside shutdown, but I think in the short term the MCC pod should get drained. With John's recent fix to leader elections, this will make the new controller pick up where the old one left off.
We decided on not mark this bug as 4.11 blocker because it doesn't blocks upgrade or new 4.11 cluster install. With this bug, upgrade may be slower for machines taking longer reboot time. This should land soon in 4.11.z.
Verified using IPI on AWS version:
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.0-0.nightly-2022-08-16-032235 True False 30m Cluster version is 4.11.0-0.nightly-2022-08-16-032235
1. check node name of mcc pod
$ oc get pod -n openshift-machine-config-operator
NAME READY STATUS RESTARTS AGE
machine-config-controller-bb5957b5d-h2g87 2/2 Running 0 21m
$ oc get pod/machine-config-controller-bb5957b5d-h2g87 -o yaml|yq -y '.spec.nodeName'
2. create mc to trigger update on master nodes
$ cat << EOF | oc create -f -
3. make sure node drain is happened on machine-config-controller's node (ip-10-0-223-27.us-east-2.compute.internal)
$ oc get events -n default --sort-by metadata.creationTimestamp --field-selector involvedObject.name=ip-10-0-223-27.us-east-2.compute.internal
111s Normal Cordon node/ip-10-0-223-27.us-east-2.compute.internal Cordoned node to apply update
>> 111s Normal Drain node/ip-10-0-223-27.us-east-2.compute.internal Draining node to update config.
103s Normal NodeNotSchedulable node/ip-10-0-223-27.us-east-2.compute.internal Node ip-10-0-223-27.us-east-2.compute.internal status is now: NodeNotSchedulable
9s Normal OSUpdateStarted node/ip-10-0-223-27.us-east-2.compute.internal
9s Normal OSUpgradeSkipped node/ip-10-0-223-27.us-east-2.compute.internal OS upgrade skipped; new MachineConfig (rendered-master-beddfd51fed4832914e143b7a5c7c8db) has same OS image (quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6e7c8e9e407ebab51eac2482d13c07d071c0be1a5755a36a64f0be1b73b3999a) as old MachineConfig (rendered-master-ac378d4e7670937fac6263c69d7a85c6)
9s Normal OSUpdateStaged node/ip-10-0-223-27.us-east-2.compute.internal Changes to OS staged
9s Normal PendingConfig node/ip-10-0-223-27.us-east-2.compute.internal Written pending config rendered-master-beddfd51fed4832914e143b7a5c7c8db
>> 9s Normal Reboot node/ip-10-0-223-27.us-east-2.compute.internal Node will reboot into config rendered-master-beddfd51fed4832914e143b7a5c7c8db
4. check pod running on above node when update is completed on master pool, no mcc pod found
$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-beddfd51fed4832914e143b7a5c7c8db True False False 3 3 3 0 73m
worker rendered-worker-dd1de4a36326291d324ed78cb2ddf723 True False False 3 3 3 0 73m
Machine-controller not running in ip-10-0-223-27.us-east-2.compute.internal anymore
$ oc get pod -n openshift-machine-config-operator --field-selector spec.nodeName=ip-10-0-223-27.us-east-2.compute.internal
NAME READY STATUS RESTARTS AGE
machine-config-daemon-5lqz4 2/2 Running 0 71m
machine-config-server-md8v6 1/1 Running 0 70m
5. mcc pod is scheduled on another node
$ oc get pod -n openshift-machine-config-operator |grep machine-config-controller
machine-config-controller-bb5957b5d-trs8h 2/2 Running 0 4m39s
$ oc get pod machine-config-controller-bb5957b5d-trs8h -n openshift-machine-config-operator -o yaml | yq -y '.spec.nodeName'
We move the status to VERIFIED.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: OpenShift Container Platform 4.11.1 bug fix and security update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.