Bug 2040376
| Summary: | "unknown instance type" error for supported m6i.xlarge instance | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Lucas López Montero <llopezmo> |
| Component: | Cloud Compute | Assignee: | Joel Speed <jspeed> |
| Cloud Compute sub component: | Other Providers | QA Contact: | Huali Liu <huliu> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | low | ||
| Priority: | low | CC: | jspeed, oarribas, sreber |
| Version: | 4.9 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.10.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: Generated list of instance types embedded within Machine API controllers was out of date
Consequence: Some instance types were unknown and couldn't be annotated for scale from zero requirements
Fix: Update generated list to support newer instance types
Result: Scale from zero is now enabled on more instance types
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-10 16:39:13 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
KCS solution written: https://access.redhat.com/solutions/6640721 As mentioned in the log message, this can be easily resolved if the end user is able to add three annotations to their machineset.
These are machine.openshift.io/vCPU, machine.openshift.io/memoryMb and machine.openshift.io/GPU.
> To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU]
This is only really required if the end user is planning to use the autoscaler, and then, only if they want the autoscaler minimum replicas to be 0 replicas.
This is very much a low priority as the workaround is very easy to apply, we will update the instance list periodically but cannot update it every time AWS introduces a new instance type
Reproduce on 4.9.15:
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.9.15 True False 6h33m Cluster version is 4.9.15
1. Create a MachineSet with m6i.xlarge instances.
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms1.yaml
machineset.machine.openshift.io/huliu-036-hqh6t-m6ix0 created
2. Check the logs of machine-controller container.
liuhuali@Lius-MacBook-Pro huali-test % oc logs -l k8s-app=controller -n openshift-machine-api -c machine-controller |grep controller
I0119 09:06:52.312831 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-036-hqh6t-m6ix0" "namespace"="openshift-machine-api"
E0119 09:06:52.313276 1 controller.go:115] Unable to set scale from zero annotations: unknown instance type: %sm6i.xlarge
E0119 09:06:52.313291 1 controller.go:116] Autoscaling from zero will not work. To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU]
I0119 09:06:52.313423 1 recorder.go:104] controller-runtime/manager/events "msg"="Warning" "message"="Failed to set autoscaling from zero annotations, instance type unknown" "object"={"kind":"MachineSet","namespace":"openshift-machine-api","name":"huliu-036-hqh6t-m6ix0","uid":"98e3e628-09e8-4cbf-9074-333e7f4f198c","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"141869"} "reason"="FailedUpdate"
I0119 09:06:52.325139 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-036-hqh6t-m6ix0" "namespace"="openshift-machine-api"
E0119 09:06:52.325699 1 controller.go:115] Unable to set scale from zero annotations: unknown instance type: %sm6i.xlarge
E0119 09:06:52.325720 1 controller.go:116] Autoscaling from zero will not work. To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU]
I0119 09:06:52.326465 1 recorder.go:104] controller-runtime/manager/events "msg"="Warning" "message"="Failed to set autoscaling from zero annotations, instance type unknown" "object"={"kind":"MachineSet","namespace":"openshift-machine-api","name":"huliu-036-hqh6t-m6ix0","uid":"98e3e628-09e8-4cbf-9074-333e7f4f198c","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"141871"} "reason"="FailedUpdate"
Verified on 4.10.0-0.nightly-2022-01-17-223655
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.10.0-0.nightly-2022-01-17-223655 True False 6h31m Cluster version is 4.10.0-0.nightly-2022-01-17-223655
1. Create a MachineSet with m6i.xlarge instances.
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms2.yaml
machineset.machine.openshift.io/huliu-035-xlb2l-m6i created
2. Check the logs of machine-controller container. no "instance type unknown" error.
liuhuali@Lius-MacBook-Pro huali-test % oc logs -l k8s-app=controller -n openshift-machine-api -c machine-controller
I0119 09:09:48.770144 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-worker-us-east-2a" "namespace"="openshift-machine-api"
I0119 09:09:48.785346 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-worker-us-east-2b" "namespace"="openshift-machine-api"
I0119 09:09:48.795173 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-worker-us-east-2c" "namespace"="openshift-machine-api"
I0119 09:12:26.387899 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-m6i" "namespace"="openshift-machine-api"
I0119 09:12:26.410264 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-m6i" "namespace"="openshift-machine-api"
liuhuali@Lius-MacBook-Pro huali-test %
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056 |
Description of problem: AWS instance m6i.xlarge is supported [1], but it does not appear on ec2_instance_types.go [2] and the following errors show up when it is used (hidden information marked with "XXX"): $ oc logs -l k8s-app=controller -n openshift-machine-api -c machine-controller [...] I0112 14:51:10.117191 1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="XXX" "namespace"="openshift-machine-api" E0112 14:51:10.117604 1 controller.go:115] Unable to set scale from zero annotations: unknown instance type: %sm6i.xlarge E0112 14:51:10.117622 1 controller.go:116] Autoscaling from zero will not work. To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU] I0112 14:51:10.117792 1 recorder.go:104] controller-runtime/manager/events "msg"="Warning" "message"="Failed to set autoscaling from zero annotations, instance type unknown" "object"={"kind":"MachineSet","namespace":"openshift-machine-api","name":"XXX","uid":"XXX","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"121131434"} "reason"="FailedUpdate" How reproducible: Always Steps to Reproduce: 1. Create a MachineSet with m6i.xlarge instances. 2. Check the logs of machine-controller container. 3. Actual results: Error messages show up, although no impact on the cluster. Expected results: No error messages because it is a supported instance. [1] https://docs.openshift.com/container-platform/4.9/installing/installing_aws/installing-aws-customizations.html#installation-supported-aws-machine-types_installing-aws-customizations [2] https://github.com/openshift-cherrypick-robot/cluster-api-provider-aws/blob/2f3f7442ef525c7eaa2177d1903cebf219a2c3fc/pkg/actuators/machineset/ec2_instance_types.go