Bug 2115358
| Summary: | control-plane-machine-set-operator pod got panic when create cpms on a single zone deployment | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | sunzhaohua <zhsun> |
| Component: | Cloud Compute | Assignee: | Joel Speed <jspeed> |
| Cloud Compute sub component: | Other Providers | QA Contact: | sunzhaohua <zhsun> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | rmanak |
| Version: | 4.12 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.12.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-01-17 19:54:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
control-plane-machine-set-operator pod dont get panic anymore, but report below error, do you want me to close this bug and open a new one, or move this to assgin?
I0815 04:58:31.508926 1 controller.go:153] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="67c15057-8087-4784-944e-026bd3f67c56"
E0815 04:58:31.508972 1 controller.go:326] "msg"="Reconciler error" "error"="error reconciling control plane machine set: error fetching machine info: could not generate machine info for machine zhsunaws815-vfkns-master-0: could not find failure domain for index: unknown index 0" "controller"="controlplanemachineset" "reconcileID"="67c15057-8087-4784-944e-026bd3f67c56"
I0815 05:15:11.510060 1 controller.go:111] "msg"="Reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="339dd978-808e-4d3d-ab7d-243506a579d6"
I0815 05:15:11.510871 1 controller.go:153] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="339dd978-808e-4d3d-ab7d-243506a579d6"
E0815 05:15:11.510926 1 controller.go:326] "msg"="Reconciler error" "error"="error reconciling control plane machine set: error fetching machine info: could not generate machine info for machine zhsunaws815-vfkns-master-1: could not find failure domain for index: unknown index 1" "controller"="controlplanemachineset" "reconcileID"="339dd978-808e-4d3d-ab7d-243506a579d6"
I0815 05:31:51.511238 1 controller.go:111] "msg"="Reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="e8887522-c447-44a7-8f35-8cbff8466f16"
I0815 05:31:51.512136 1 controller.go:153] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="e8887522-c447-44a7-8f35-8cbff8466f16"
E0815 05:31:51.512185 1 controller.go:326] "msg"="Reconciler error" "error"="error reconciling control plane machine set: error fetching machine info: could not generate machine info for machine zhsunaws815-vfkns-master-0: could not find failure domain for index: unknown index 0" "controller"="controlplanemachineset" "reconcileID"="e8887522-c447-44a7-8f35-8cbff8466f16"
I0815 05:48:31.512834 1 controller.go:111] "msg"="Reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="c8b326a1-96d3-489b-8861-151408f818dc"
I0815 05:48:31.513610 1 controller.go:153] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="c8b326a1-96d3-489b-8861-151408f818dc"
E0815 05:48:31.513659 1 controller.go:326] "msg"="Reconciler error" "error"="error reconciling control plane machine set: error fetching machine info: could not generate machine info for machine zhsunaws815-vfkns-master-2: could not find failure domain for index: unknown index 2" "controller"="controlplanemachineset" "reconcileID"="c8b326a1-96d3-489b-8861-151408f818dc"
$ oc get controlplanemachineset
NAME DESIRED CURRENT READY UPDATED UNAVAILABLE AGE
cluster 3 13m
$ oc get machine
NAME PHASE TYPE REGION ZONE AGE
zhsunaws815-vfkns-master-0 Running m6i.xlarge us-east-2 us-east-2a 3h50m
zhsunaws815-vfkns-master-1 Running m6i.xlarge us-east-2 us-east-2a 3h50m
zhsunaws815-vfkns-master-2 Running m6i.xlarge us-east-2 us-east-2a 3h50m
zhsunaws815-vfkns-worker-us-east-2a-hlb5t Running m6i.xlarge us-east-2 us-east-2a 3h47m
apiVersion: machine.openshift.io/v1
kind: ControlPlaneMachineSet
metadata:
name: cluster
namespace: openshift-machine-api
spec:
replicas: 3
strategy:
type: RollingUpdate
selector:
matchLabels:
machine.openshift.io/cluster-api-machine-role: master
machine.openshift.io/cluster-api-machine-type: master
template:
machineType: machines_v1beta1_machine_openshift_io
machines_v1beta1_machine_openshift_io:
metadata:
labels:
machine.openshift.io/cluster-api-machine-role: master
machine.openshift.io/cluster-api-machine-type: master
machine.openshift.io/cluster-api-cluster: zhsunaws815-vfkns
spec:
providerSpec:
value:
ami:
id: ami-03c8407be1de91c11
apiVersion: machine.openshift.io/v1beta1
blockDevices:
- ebs:
encrypted: true
iops: 0
kmsKey:
arn: ""
volumeSize: 120
volumeType: gp3
credentialsSecret:
name: aws-cloud-credentials
deviceIndex: 0
iamInstanceProfile:
id: zhsunaws815-vfkns-master-profile
instanceType: m6i.xlarge
kind: AWSMachineProviderConfig
loadBalancers:
- name: zhsunaws815-vfkns-int
type: network
- name: zhsunaws815-vfkns-ext
type: network
metadata:
creationTimestamp: null
metadataServiceOptions: {}
placement:
region: us-east-2
availabilityZone: us-east-2a
subnet:
filters:
- name: tag:Name
values:
- zhsunaws815-vfkns-private-us-east-2a
securityGroups:
- filters:
- name: tag:Name
values:
- zhsunaws815-vfkns-master-sg
tags:
- name: kubernetes.io/cluster/zhsunaws815-vfkns
value: owned
userDataSecret:
name: master-user-data
I'll raise a PR to resolve this new issue It report another error: I0819 10:33:19.983320 1 controller.go:153] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="8ac8d491-26af-4781-99a7-5d8148153db6" E0819 10:33:19.983365 1 controller.go:326] "msg"="Reconciler error" "error"="error reconciling control plane machine set: error reconciling machines: error reconciling machine updates: error creating new Machine for index 0: cannot inject failure domain in the provider config: failure domain is nil" "controller"="controlplanemachineset" "reconcileID"="8ac8d491-26af-4781-99a7-5d8148153db6" $ oc get controlplanemachineset NAME DESIRED CURRENT READY UPDATED UNAVAILABLE AGE cluster 3 3 3 77m $ oc get machine NAME PHASE TYPE REGION ZONE AGE zhsunaws819-gt2z4-master-0 Running m6i.xlarge us-east-2 us-east-2a 143m zhsunaws819-gt2z4-master-1 Running m6i.xlarge us-east-2 us-east-2a 143m zhsunaws819-gt2z4-master-2 Running m6i.xlarge us-east-2 us-east-2a 143m zhsunaws819-gt2z4-worker-us-east-2a-c6gj9 Running m6i.xlarge us-east-2 us-east-2a 140m zhsunaws819-gt2z4-worker-us-east-2a-q8lhr Running m6i.xlarge us-east-2 us-east-2a 140m zhsunaws819-gt2z4-worker-us-east-2a-rmlhq Running m6i.xlarge us-east-2 us-east-2a 140m control-plane-machine-set 0.0.1-snapshot True True False 79m Observed 3 replica(s) in need of update Verfied before pr merge
$ oc get controlplanemachineset
NAME DESIRED CURRENT READY UPDATED UNAVAILABLE AGE
cluster 3 3 3 3 15s
I0826 15:07:50.830630 1 http.go:96] controller-runtime/webhook/webhooks "msg"="received request" "webhook"="/validate-machine-openshift-io-v1-controlplanemachineset" "UID"="33ab4da7-a0eb-47a3-8aa1-adc12656d4b8" "kind"={"group":"machine.openshift.io","version":"v1","kind":"ControlPlaneMachineSet"} "resource"={"group":"machine.openshift.io","version":"v1","resource":"controlplanemachinesets"}
I0826 15:07:50.831581 1 http.go:135] controller-runtime/webhook/webhooks "msg"="wrote response" "code"=200 "reason"="" "webhook"="/validate-machine-openshift-io-v1-controlplanemachineset" "UID"="33ab4da7-a0eb-47a3-8aa1-adc12656d4b8" "allowed"=true
I0826 15:07:50.839664 1 controller.go:128] "msg"="Reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="fb0b2b76-f2c5-4f84-932d-ba2e53d4c177"
I0826 15:07:50.843881 1 http.go:96] controller-runtime/webhook/webhooks "msg"="received request" "webhook"="/validate-machine-openshift-io-v1-controlplanemachineset" "UID"="46e40511-02df-4a5d-89bc-038db9c8e0a3" "kind"={"group":"machine.openshift.io","version":"v1","kind":"ControlPlaneMachineSet"} "resource"={"group":"machine.openshift.io","version":"v1","resource":"controlplanemachinesets"}
I0826 15:07:50.844320 1 http.go:135] controller-runtime/webhook/webhooks "msg"="wrote response" "code"=200 "reason"="" "webhook"="/validate-machine-openshift-io-v1-controlplanemachineset" "UID"="46e40511-02df-4a5d-89bc-038db9c8e0a3" "allowed"=true
I0826 15:07:50.848650 1 controller.go:361] "msg"="Added finalizer to control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="fb0b2b76-f2c5-4f84-932d-ba2e53d4c177"
I0826 15:07:50.861144 1 controller.go:173] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="fb0b2b76-f2c5-4f84-932d-ba2e53d4c177"
I0826 15:07:50.861206 1 controller.go:128] "msg"="Reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="70bf5ed9-222d-4899-b4f7-666cc0c6dd2f"
I0826 15:07:50.861794 1 status.go:111] "msg"="Observed Machine Configuration" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="70bf5ed9-222d-4899-b4f7-666cc0c6dd2f" "observedGeneration"=2 "readyReplicas"=3 "replicas"=3 "unavailableReplicas"=0 "updatedReplicas"=3
I0826 15:07:50.890496 1 controller.go:419] "msg"="Added owner reference to machine" "controller"="controlplanemachineset" "machineName"="zhsunaws8264-sk9z2-master-0" "machineNamespace"="openshift-machine-api" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="70bf5ed9-222d-4899-b4f7-666cc0c6dd2f"
I0826 15:07:50.901835 1 controller.go:419] "msg"="Added owner reference to machine" "controller"="controlplanemachineset" "machineName"="zhsunaws8264-sk9z2-master-1" "machineNamespace"="openshift-machine-api" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="70bf5ed9-222d-4899-b4f7-666cc0c6dd2f"
I0826 15:07:50.914793 1 controller.go:419] "msg"="Added owner reference to machine" "controller"="controlplanemachineset" "machineName"="zhsunaws8264-sk9z2-master-2" "machineNamespace"="openshift-machine-api" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="70bf5ed9-222d-4899-b4f7-666cc0c6dd2f"
I0826 15:07:51.031832 1 controller.go:173] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="70bf5ed9-222d-4899-b4f7-666cc0c6dd2f"
I0826 15:07:51.032040 1 controller.go:128] "msg"="Reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="bd532fa5-ff0e-4ea0-93d0-62261111c61d"
I0826 15:07:51.033158 1 status.go:111] "msg"="Observed Machine Configuration" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="bd532fa5-ff0e-4ea0-93d0-62261111c61d" "observedGeneration"=2 "readyReplicas"=3 "replicas"=3 "unavailableReplicas"=0 "updatedReplicas"=3
I0826 15:07:51.035754 1 controller.go:173] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="bd532fa5-ff0e-4ea0-93d0-62261111c61d"
I0826 15:07:51.035806 1 controller.go:128] "msg"="Reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="79c9bf8e-a6d4-47e2-a586-604a36ce50c0"
I0826 15:07:51.036503 1 status.go:111] "msg"="Observed Machine Configuration" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="79c9bf8e-a6d4-47e2-a586-604a36ce50c0" "observedGeneration"=2 "readyReplicas"=3 "replicas"=3 "unavailableReplicas"=0 "updatedReplicas"=3
I0826 15:07:51.037578 1 controller.go:173] "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="79c9bf8e-a6d4-47e2-a586-604a36ce50c0"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399 |
Description of problem: Control-plane-machine-set-operator pod got panic when create cpms on a single zone deployment Version-Release number of selected component (if applicable): 4.12.0-0.nightly-2022-08-01-151317 How reproducible: always Steps to Reproduce: 1. Create a cluster on a single zone $ oc get machine NAME PHASE TYPE REGION ZONE AGE zhsunaws42-8fmkk-master-0 Running m6i.xlarge us-east-2 us-east-2a 12h zhsunaws42-8fmkk-master-1 Running m6i.xlarge us-east-2 us-east-2a 12h zhsunaws42-8fmkk-master-2 Running m6i.xlarge us-east-2 us-east-2a 12h zhsunaws42-8fmkk-worker-us-east-2a-sjqjg Running m6i.xlarge us-east-2 us-east-2a 12h 2. Create cpms, no spec.template.failureDomains are provided, failureDomains are provided by spec.template.spec.providerSpec. $ oc get controlplanemachineset NAME DESIRED CURRENT READY UPDATED UNAVAILABLE AGE cluster 3 4h26m $ oc get controlplanemachineset cluster -o yaml apiVersion: machine.openshift.io/v1 kind: ControlPlaneMachineSet metadata: creationTimestamp: "2022-08-04T09:10:03Z" finalizers: - controlplanemachineset.machine.openshift.io generation: 2 name: cluster namespace: openshift-machine-api resourceVersion: "181701" uid: d2f956e3-f330-4ce6-9258-e8d82a96226c spec: replicas: 3 selector: matchLabels: machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master strategy: type: RollingUpdate template: machineType: machines_v1beta1_machine_openshift_io machines_v1beta1_machine_openshift_io: failureDomains: platform: "" metadata: labels: machine.openshift.io/cluster-api-cluster: zhsunaws42-8fmkk machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master spec: lifecycleHooks: {} metadata: {} providerSpec: value: ami: id: ami-01990fc3bdf30bc13 apiVersion: machine.openshift.io/v1beta1 blockDevices: - ebs: encrypted: true iops: 0 kmsKey: arn: "" volumeSize: 120 volumeType: gp3 credentialsSecret: name: aws-cloud-credentials deviceIndex: 0 iamInstanceProfile: id: zhsunaws42-8fmkk-master-profile instanceType: m6i.xlarge kind: AWSMachineProviderConfig loadBalancers: - name: zhsunaws42-8fmkk-int type: network - name: zhsunaws42-8fmkk-ext type: network metadata: creationTimestamp: null metadataServiceOptions: {} placement: availabilityZone: us-east-2a region: us-east-2 securityGroups: - filters: - name: tag:Name values: - zhsunaws42-8fmkk-master-sg subnet: filters: - name: tag:Name values: - zhsunaws42-8fmkk-private-us-east-2a tags: - name: kubernetes.io/cluster/zhsunaws42-8fmkk value: owned userDataSecret: name: master-user-data status: {} 3. Check cpmso pod $ oc get po NAME READY STATUS RESTARTS AGE cluster-autoscaler-operator-6f975ccbcf-f8mdh 2/2 Running 0 12h cluster-baremetal-operator-578d87965-hhxw6 2/2 Running 0 12h control-plane-machine-set-operator-6b9c678f6c-p7vtq 0/1 CrashLoopBackOff 56 (43s ago) 10h machine-api-controllers-85f47957f7-zgscz 7/7 Running 0 12h machine-api-operator-7754df4b95-hlh2n 2/2 Running 0 12h Actual results: Control-plane-machine-set-operator pod got panic I0804 09:10:03.481177 1 controller.go:117] "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "controller"="controlplanemachineset" "reconcileID"="f5703494-8973-446e-b02d-2ca936321961" panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x13e4bf3] goroutine 203 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118 +0x1f4 panic({0x154d6a0, 0x2494650}) /usr/lib/golang/src/runtime/panic.go:838 +0x207 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.providerConfig.InjectFailureDomain({{0x1722b95, 0x3}, {{{{...}, {...}}, {{...}, {...}, {...}, {...}, {...}, {...}, ...}, ...}}, ...}, ...) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/providerconfig.go:128 +0xd3 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1.(*openshiftMachineProvider).CreateMachine(0xc0001b1180, {0x19945d8, 0xc00081e1b0}, {{0x19965b8?, 0xc0005d6b90?}, 0xfdf913?}, 0x923890?) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/provider.go:345 +0x27c github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.createMachine({0x19945d8?, 0xc00081e1b0?}, {{0x19965b8?, 0xc0005d6b90?}, 0x2?}, {0x1988cd8?, 0xc0001b1180?}, 0x0?, 0x60?, 0xc00055f690) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/updates.go:348 +0x9e github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).createReplacementMachines(0xc0001431d0, {0x19945d8, 0xc00081e1b0}, {{0x19965b8?, 0xc0005d6b40?}, 0xc0001b1180?}, {0x1988cd8, 0xc0001b1180}, {0xc00081f4a0, 0x1, ...}, ...) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/updates.go:311 +0x7d2 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).reconcileMachineRollingUpdate(0x0?, {0x19945d8, 0xc00081e1b0}, {{0x19965b8?, 0xc0000967d0?}, 0x4?}, 0xc0007276c0, {0x1988cd8, 0xc0001b1180}, 0xc00055fa40) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/updates.go:178 +0x388 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).reconcileMachineUpdates(0x19945d8?, {0x19945d8?, 0xc00081e1b0?}, {{0x19965b8?, 0xc0000967d0?}, 0xc0007276c0?}, 0xc0007276c0, {0x1988cd8?, 0xc0001b1180?}, 0xc00055fa40) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/updates.go:102 +0x279 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).reconcileMachines(0x24e8f58?, {0x19945d8, 0xc00081e1b0}, {{0x19965b8?, 0xc0000967d0?}, 0xc000142be0?}, 0xc0007276c0?, {0x1988cd8, 0xc0001b1180}, 0xc00055fa40) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/controller.go:217 +0x1b5 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).reconcile(0xc0001431d0, {0x19945d8, 0xc00081e1b0}, {{0x19965b8?, 0xc0000967d0?}, 0xc00058d2a0?}, 0xc0007276c0) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/controller.go:188 +0x3aa github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).Reconcile(0xc0001431d0, {0x19945d8, 0xc00081e1b0}, {{{0xc0008a5650?, 0x10?}, {0xc00058d2a0?, 0x413c87?}}}) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/controller.go:133 +0x3c5 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1994530?, {0x19945d8?, 0xc00081e1b0?}, {{{0xc0008a5650?, 0x1662c80?}, {0xc00058d2a0?, 0x409514?}}}) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121 +0xc8 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0005d83c0, {0x1994530, 0xc000164d00}, {0x15a5cc0?, 0xc00003ef80?}) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320 +0x33c sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005d83c0, {0x1994530, 0xc000164d00}) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273 +0x1d9 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234 +0x85 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:230 +0x325 Expected results: Control-plane-machine-set-operator pod running well Additional info: