Description of problem: AWS instance m6i.xlarge is supported [1], but it does not appear on ec2_instance_types.go [2] and the following errors show up when it is used (hidden information marked with "XXX"): $ oc logs -l k8s-app=controller -n openshift-machine-api -c machine-controller [...] I0112 14:51:10.117191 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="XXX" "namespace"="openshift-machine-api" E0112 14:51:10.117604 1 controller.go:115] Unable to set scale from zero annotations: unknown instance type: %sm6i.xlarge E0112 14:51:10.117622 1 controller.go:116] Autoscaling from zero will not work. To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU] I0112 14:51:10.117792 1 recorder.go:104] controller-runtime/manager/events "msg"="Warning" "message"="Failed to set autoscaling from zero annotations, instance type unknown" "object"={"kind":"MachineSet","namespace":"openshift-machine-api","name":"XXX","uid":"XXX","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"121131434"} "reason"="FailedUpdate" How reproducible: Always Steps to Reproduce: 1. Create a MachineSet with m6i.xlarge instances. 2. Check the logs of machine-controller container. 3. Actual results: Error messages show up, although no impact on the cluster. Expected results: No error messages because it is a supported instance. [1] https://docs.openshift.com/container-platform/4.9/installing/installing_aws/installing-aws-customizations.html#installation-supported-aws-machine-types_installing-aws-customizations [2] https://github.com/openshift-cherrypick-robot/cluster-api-provider-aws/blob/2f3f7442ef525c7eaa2177d1903cebf219a2c3fc/pkg/actuators/machineset/ec2_instance_types.go
KCS solution written: https://access.redhat.com/solutions/6640721
As mentioned in the log message, this can be easily resolved if the end user is able to add three annotations to their machineset. These are machine.openshift.io/vCPU, machine.openshift.io/memoryMb and machine.openshift.io/GPU. > To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU] This is only really required if the end user is planning to use the autoscaler, and then, only if they want the autoscaler minimum replicas to be 0 replicas. This is very much a low priority as the workaround is very easy to apply, we will update the instance list periodically but cannot update it every time AWS introduces a new instance type
Reproduce on 4.9.15: liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.15 True False 6h33m Cluster version is 4.9.15 1. Create a MachineSet with m6i.xlarge instances. liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms1.yaml machineset.machine.openshift.io/huliu-036-hqh6t-m6ix0 created 2. Check the logs of machine-controller container. liuhuali@Lius-MacBook-Pro huali-test % oc logs -l k8s-app=controller -n openshift-machine-api -c machine-controller |grep controller I0119 09:06:52.312831 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-036-hqh6t-m6ix0" "namespace"="openshift-machine-api" E0119 09:06:52.313276 1 controller.go:115] Unable to set scale from zero annotations: unknown instance type: %sm6i.xlarge E0119 09:06:52.313291 1 controller.go:116] Autoscaling from zero will not work. To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU] I0119 09:06:52.313423 1 recorder.go:104] controller-runtime/manager/events "msg"="Warning" "message"="Failed to set autoscaling from zero annotations, instance type unknown" "object"={"kind":"MachineSet","namespace":"openshift-machine-api","name":"huliu-036-hqh6t-m6ix0","uid":"98e3e628-09e8-4cbf-9074-333e7f4f198c","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"141869"} "reason"="FailedUpdate" I0119 09:06:52.325139 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-036-hqh6t-m6ix0" "namespace"="openshift-machine-api" E0119 09:06:52.325699 1 controller.go:115] Unable to set scale from zero annotations: unknown instance type: %sm6i.xlarge E0119 09:06:52.325720 1 controller.go:116] Autoscaling from zero will not work. To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU] I0119 09:06:52.326465 1 recorder.go:104] controller-runtime/manager/events "msg"="Warning" "message"="Failed to set autoscaling from zero annotations, instance type unknown" "object"={"kind":"MachineSet","namespace":"openshift-machine-api","name":"huliu-036-hqh6t-m6ix0","uid":"98e3e628-09e8-4cbf-9074-333e7f4f198c","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"141871"} "reason"="FailedUpdate" Verified on 4.10.0-0.nightly-2022-01-17-223655 liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2022-01-17-223655 True False 6h31m Cluster version is 4.10.0-0.nightly-2022-01-17-223655 1. Create a MachineSet with m6i.xlarge instances. liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms2.yaml machineset.machine.openshift.io/huliu-035-xlb2l-m6i created 2. Check the logs of machine-controller container. no "instance type unknown" error. liuhuali@Lius-MacBook-Pro huali-test % oc logs -l k8s-app=controller -n openshift-machine-api -c machine-controller I0119 09:09:48.770144 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-worker-us-east-2a" "namespace"="openshift-machine-api" I0119 09:09:48.785346 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-worker-us-east-2b" "namespace"="openshift-machine-api" I0119 09:09:48.795173 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-worker-us-east-2c" "namespace"="openshift-machine-api" I0119 09:12:26.387899 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-m6i" "namespace"="openshift-machine-api" I0119 09:12:26.410264 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-m6i" "namespace"="openshift-machine-api" liuhuali@Lius-MacBook-Pro huali-test %
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056