Bug 2098508
| Summary: | Control-plane-machine-set-operator report panic | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | sunzhaohua <zhsun> |
| Component: | Cloud Compute | Assignee: | Joel Speed <jspeed> |
| Cloud Compute sub component: | Other Providers | QA Contact: | sunzhaohua <zhsun> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | ||
| Version: | 4.11 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.11.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-10 11:18:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
https://github.com/openshift/cluster-control-plane-machine-set-operator/blob/main/manifests/0000_31_control-plane-machine-set-operator_02_service.yaml and https://github.com/openshift/cluster-control-plane-machine-set-operator/blob/main/manifests/0000_31_control-plane-machine-set-operator_05_service.yaml are same, only port are different. $ oc create -f ~/data/master/0000_31_control-plane-machine-set-operator_05_service.yaml Error from server (AlreadyExists): error when creating "/Users/sunzhaohua/data/master/0000_31_control-plane-machine-set-operator_05_service.yaml": services "control-plane-machine-set-operator" already exists $ oc create -f ~/data/master/controlplanemachineset.yaml Error from server (InternalError): error when creating "/Users/sunzhaohua/data/master/controlplanemachineset.yaml": Internal error occurred: failed calling webhook "controlplanemachineset.machine.openshift.io": failed to call webhook: Post "https://control-plane-machine-set-operator.openshift-machine-api.svc:9443/validate-machine-openshift-io-v1-controlplanemachineset?timeout=10s": no service port 9443 found for service "control-plane-machine-set-operator" 1. Created resources in https://github.com/openshift/cluster-control-plane-machine-set-operator/tree/main/manifests except 0000_31_control-plane-machine-set-operator_02_service.yaml 2. Create controlplanemachineset can be created successful, log no panic. name,replicas and selector can't be changed. apiVersion: machine.openshift.io/v1 kind: ControlPlaneMachineSet metadata: name: cluster namespace: openshift-machine-api spec: replicas: 3 strategy: type: RollingUpdate selector: matchLabels: machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master template: machineType: machines_v1beta1_machine_openshift_io machines_v1beta1_machine_openshift_io: metadata: labels: machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master machine.openshift.io/cluster-api-cluster: zhsunaws-n52dn failureDomains: platform: AWS aws: - placement: availabilityZone: us-east-2a subnet: type: filters filters: - name: tag:Name values: - zhsunaws-n52dn-private-us-east-2a - placement: availabilityZone: us-east-2b subnet: type: filters filters: - name: tag:Name values: - zhsunaws-n52dn-private-us-east-2b - placement: availabilityZone: us-east-2c subnet: type: filters filters: - name: tag:Name values: - zhsunaws-n52dn-private-us-east-2c spec: providerSpec: value: ami: id: ami-01990fc3bdf30bc13 apiVersion: machine.openshift.io/v1beta1 blockDevices: - ebs: encrypted: true iops: 0 kmsKey: arn: "" volumeSize: 120 volumeType: gp3 credentialsSecret: name: aws-cloud-credentials deviceIndex: 0 iamInstanceProfile: id: zhsunaws-n52dn-master-profile instanceType: m6i.xlarge kind: AWSMachineProviderConfig loadBalancers: - name: zhsunaws-n52dn-int type: network - name: zhsunaws-n52dn-ext type: network metadata: creationTimestamp: null metadataServiceOptions: {} placement: region: us-east-2 securityGroups: - filters: - name: tag:Name values: - zhsunaws-n52dn-master-sg tags: - name: kubernetes.io/cluster/zhsunaws-n52dn value: owned userDataSecret: name: master-user-data $ oc get controlplanemachineset NAME DESIRED CURRENT READY UPDATED UNAVAILABLE AGE cluster 3 46s $ oc create -f ~/data/master/controlplanemachineset.yaml Error from server (name: Invalid value: "cluster1": control plane machine set name must be cluster): error when creating "/Users/sunzhaohua/data/master/controlplanemachineset.yaml": admission webhook "controlplanemachineset.machine.openshift.io" denied the request: name: Invalid value: "cluster1": control plane machine set name must be cluster $ oc edit controlplanemachineset cluster error: controlplanemachinesets.machine.openshift.io "cluster" could not be patched: admission webhook "controlplanemachineset.machine.openshift.io" denied the request: spec.replicas: Forbidden: control plane machine set replicas cannot be changed $ oc edit controlplanemachineset cluster error: controlplanemachinesets.machine.openshift.io "cluster" could not be patched: admission webhook "controlplanemachineset.machine.openshift.io" denied the request: [spec.selector: Forbidden: control plane machine set selector is immutable, spec.template.machines_v1beta1_machine_openshift_io.metadata.labels: Invalid value: map[string]string{"machine.openshift.io/cluster-api-cluster":"zhsunaws-n52dn", "machine.openshift.io/cluster-api-machine-role":"master", "machine.openshift.io/cluster-api-machine-type":"master"}: selector does not match template labels] Checked the change, move to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |
Description of problem: CPMS name can be changed, spec.replicas can be changed, selector can be changed. control-plane-machine-set-operator show panic. Version-Release number of selected component (if applicable): Not in payload yet How reproducible: Always Steps to Reproduce: 1. Create ControlPlaneMachineSet apiVersion: machine.openshift.io/v1 kind: ControlPlaneMachineSet metadata: creationTimestamp: "2022-06-17T09:54:40Z" generation: 2 name: cluster namespace: openshift-machine-api resourceVersion: "108717" uid: a82d5430-c764-4cb3-a523-4da1d3d42707 spec: replicas: 3 selector: matchLabels: machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master strategy: type: RollingUpdate template: machineType: machines_v1beta1_machine_openshift_io machines_v1beta1_machine_openshift_io: failureDomains: aws: - placement: availabilityZone: us-east-2a subnet: filters: - name: tag:Name values: - weinliu4117-h6gsk-private-us-east-2a type: filters - placement: availabilityZone: us-east-2b subnet: filters: - name: tag:Name values: - weinliu4117-h6gsk-private-us-east-2b type: filters - placement: availabilityZone: us-east-2c subnet: filters: - name: tag:Name values: - weinliu4117-h6gsk-private-us-east-2c type: filters platform: AWS metadata: {} spec: providerSpec: value: ami: id: ami-01990fc3bdf30bc13 apiVersion: machine.openshift.io/v1beta1 blockDevices: - ebs: encrypted: true iops: 0 kmsKey: arn: "" volumeSize: 120 volumeType: gp3 credentialsSecret: name: aws-cloud-credentials deviceIndex: 0 iamInstanceProfile: id: weinliu4117-h6gsk-master-profile instanceType: m6i.xlarge kind: AWSMachineProviderConfig loadBalancers: - name: weinliu4117-h6gsk-int type: network - name: weinliu4117-h6gsk-ext type: network metadata: creationTimestamp: null metadataServiceOptions: {} placement: region: us-east-2 securityGroups: - filters: - name: tag:Name values: - weinliu4117-h6gsk-master-sg tags: - name: kubernetes.io/cluster/weinliu4117-h6gsk value: owned userDataSecret: name: master-user-data 2. Change CPMS name, change spec.replicas, change selector 3. Actual results: CPMS name can be changed, spec.replicas can be changed, selector can be changed. control-plane-machine-set-operator show panic. $ oc create -f ~/data/master/controlplanemachineset.yaml controlplanemachineset.machine.openshift.io/cluster created $ oc create -f ~/data/master/controlplanemachineset.yaml controlplanemachineset.machine.openshift.io/cluster1 created $ oc get controlplanemachineset NAME DESIRED CURRENT READY UPDATED UNAVAILABLE AGE cluster 3 2m23s cluster1 5 67s 1.6554603373737745e+09 INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "controlplanemachineset", "controllerGroup": "machine.openshift.io", "controllerKind": "ControlPlaneMachineSet", "controlPlaneMachineSet": {"name":"cluster","namespace":"openshift-machine-api"}, "namespace": "openshift-machine-api", "name": "cluster", "reconcileID": "2e45d8a7-939f-480d-80af-c9b7c35548f1"} panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x140812d]goroutine 402 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118 +0x1f4 panic({0x1560a80, 0x24bb970}) /usr/lib/golang/src/runtime/panic.go:838 +0x207 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).patchClusterOperatorConditions(0xc00025a000, {0x19a8c30, 0xc00049e240}, {{0x19aa660?, 0xc00049e270?}, 0x24dde40?}, 0xc0006a6340, {0xc0003c5560, 0x1, 0x1}) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/cluster_operator.go:136 +0x64d github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).updateClusterOperatorStatus(0xc00025a000?, {0x19a8c30, 0xc00049e240}, {{0x19aa660?, 0xc00049e270?}, 0x173b01a?}, 0xc0006a6000) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/cluster_operator.go:84 +0x768 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).Reconcile(0xc00025a000, {0x19a8c30, 0xc00049e240}, {{{0x174b22d?, 0x10?}, {0x173b01a?, 0x413c87?}}}) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/controller.go:135 +0x5cc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x19a8b88?, {0x19a8c30?, 0xc00049e240?}, {{{0x174b22d?, 0x16787e0?}, {0x173b01a?, 0x409514?}}}) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121 +0xc8 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000120000, {0x19a8b88, 0xc0004add80}, {0x15b9ea0?, 0xc00002c980?}) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320 +0x33c sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000120000, {0x19a8b88, 0xc0004add80}) /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273 +0x1d9 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234 +0x85 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:230 +0x325 Expected results: Change CPMS name, spec.replicas and selector can't be changed, control-plane-machine-set-operator should be normal. Additional info: