Bug 2040376 - "unknown instance type" error for supported m6i.xlarge instance
Summary: "unknown instance type" error for supported m6i.xlarge instance
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.9
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.10.0
Assignee: Joel Speed
QA Contact: Huali Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-13 15:35 UTC by Lucas López Montero
Modified: 2023-07-21 15:47 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Generated list of instance types embedded within Machine API controllers was out of date Consequence: Some instance types were unknown and couldn't be annotated for scale from zero requirements Fix: Update generated list to support newer instance types Result: Scale from zero is now enabled on more instance types
Clone Of:
Environment:
Last Closed: 2022-03-10 16:39:13 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-api-provider-aws pull 19 0 None open Bug 2040376: Update map of supported aws instance types 2022-01-14 11:49:58 UTC
Red Hat Knowledge Base (Solution) 6640721 0 None None None 2023-07-21 15:47:04 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:39:33 UTC

Description Lucas López Montero 2022-01-13 15:35:15 UTC
Description of problem:

AWS instance m6i.xlarge is supported [1], but it does not appear on ec2_instance_types.go [2] and the following errors show up when it is used (hidden information marked with "XXX"):

$ oc logs -l k8s-app=controller -n openshift-machine-api -c machine-controller
[...]
I0112 14:51:10.117191       1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="XXX" "namespace"="openshift-machine-api"
E0112 14:51:10.117604       1 controller.go:115] Unable to set scale from zero annotations: unknown instance type: %sm6i.xlarge
E0112 14:51:10.117622       1 controller.go:116] Autoscaling from zero will not work. To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU]
I0112 14:51:10.117792       1 recorder.go:104] controller-runtime/manager/events "msg"="Warning"  "message"="Failed to set autoscaling from zero annotations, instance type unknown" "object"={"kind":"MachineSet","namespace":"openshift-machine-api","name":"XXX","uid":"XXX","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"121131434"} "reason"="FailedUpdate"


How reproducible:
Always


Steps to Reproduce:
1. Create a MachineSet with m6i.xlarge instances.
2. Check the logs of machine-controller container.
3.


Actual results:
Error messages show up, although no impact on the cluster.


Expected results:
No error messages because it is a supported instance.



[1] https://docs.openshift.com/container-platform/4.9/installing/installing_aws/installing-aws-customizations.html#installation-supported-aws-machine-types_installing-aws-customizations
[2] https://github.com/openshift-cherrypick-robot/cluster-api-provider-aws/blob/2f3f7442ef525c7eaa2177d1903cebf219a2c3fc/pkg/actuators/machineset/ec2_instance_types.go

Comment 1 Lucas López Montero 2022-01-13 16:06:37 UTC
KCS solution written: https://access.redhat.com/solutions/6640721

Comment 2 Joel Speed 2022-01-13 16:17:07 UTC
As mentioned in the log message, this can be easily resolved if the end user is able to add three annotations to their machineset.

These are machine.openshift.io/vCPU, machine.openshift.io/memoryMb and machine.openshift.io/GPU.

> To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU]

This is only really required if the end user is planning to use the autoscaler, and then, only if they want the autoscaler minimum replicas to be 0 replicas.
This is very much a low priority as the workaround is very easy to apply, we will update the instance list periodically but cannot update it every time AWS introduces a new instance type

Comment 5 Huali Liu 2022-01-19 09:24:27 UTC
Reproduce on 4.9.15:
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.15    True        False         6h33m   Cluster version is 4.9.15
 
1. Create a MachineSet with m6i.xlarge instances.
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms1.yaml 
machineset.machine.openshift.io/huliu-036-hqh6t-m6ix0 created

2. Check the logs of machine-controller container.
liuhuali@Lius-MacBook-Pro huali-test % oc logs -l k8s-app=controller -n openshift-machine-api -c machine-controller |grep controller
I0119 09:06:52.312831       1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-036-hqh6t-m6ix0" "namespace"="openshift-machine-api" 
E0119 09:06:52.313276       1 controller.go:115] Unable to set scale from zero annotations: unknown instance type: %sm6i.xlarge
E0119 09:06:52.313291       1 controller.go:116] Autoscaling from zero will not work. To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU]
I0119 09:06:52.313423       1 recorder.go:104] controller-runtime/manager/events "msg"="Warning"  "message"="Failed to set autoscaling from zero annotations, instance type unknown" "object"={"kind":"MachineSet","namespace":"openshift-machine-api","name":"huliu-036-hqh6t-m6ix0","uid":"98e3e628-09e8-4cbf-9074-333e7f4f198c","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"141869"} "reason"="FailedUpdate"
I0119 09:06:52.325139       1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-036-hqh6t-m6ix0" "namespace"="openshift-machine-api" 
E0119 09:06:52.325699       1 controller.go:115] Unable to set scale from zero annotations: unknown instance type: %sm6i.xlarge
E0119 09:06:52.325720       1 controller.go:116] Autoscaling from zero will not work. To fix this, manually populate machine annotations for your instance type: %v[machine.openshift.io/vCPU machine.openshift.io/memoryMb machine.openshift.io/GPU]
I0119 09:06:52.326465       1 recorder.go:104] controller-runtime/manager/events "msg"="Warning"  "message"="Failed to set autoscaling from zero annotations, instance type unknown" "object"={"kind":"MachineSet","namespace":"openshift-machine-api","name":"huliu-036-hqh6t-m6ix0","uid":"98e3e628-09e8-4cbf-9074-333e7f4f198c","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"141871"} "reason"="FailedUpdate"

Verified on 4.10.0-0.nightly-2022-01-17-223655
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-01-17-223655   True        False         6h31m   Cluster version is 4.10.0-0.nightly-2022-01-17-223655
1. Create a MachineSet with m6i.xlarge instances.
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms2.yaml 
machineset.machine.openshift.io/huliu-035-xlb2l-m6i created

2. Check the logs of machine-controller container. no "instance type unknown" error.
liuhuali@Lius-MacBook-Pro huali-test % oc logs -l k8s-app=controller -n openshift-machine-api -c machine-controller
I0119 09:09:48.770144       1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-worker-us-east-2a" "namespace"="openshift-machine-api" 
I0119 09:09:48.785346       1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-worker-us-east-2b" "namespace"="openshift-machine-api" 
I0119 09:09:48.795173       1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-worker-us-east-2c" "namespace"="openshift-machine-api" 
I0119 09:12:26.387899       1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-m6i" "namespace"="openshift-machine-api" 
I0119 09:12:26.410264       1 controller.go:59] controllers/MachineSet "msg"="Reconciling" "machineset"="huliu-035-xlb2l-m6i" "namespace"="openshift-machine-api" 
liuhuali@Lius-MacBook-Pro huali-test %

Comment 11 errata-xmlrpc 2022-03-10 16:39:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.