Description of problem:
An OSD cluster was provisioned in an account where AWS limits were not sufficient to fulfill all EC2 instances requested resulting in an initial cluster with 3 masters and 2 workers. When the limits were updated the additional EC2 instances (+2) were created and associated with the Machines in cluster. But no corresponding Node was created in the cluster.
Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.1.7 True False 10h Cluster version is 4.1.7
Steps to Reproduce:
1. Provision cluster in account with EC2 limit for workers lower than desired number of machines.
2. Observe failure to provision nodes.
$ oc logs machine-api-controllers-5d957c6fd-qzxml -c controller-manager
E0725 08:39:47.357292 1 instances.go:309] Error creating EC2 instance: InstanceLimitExceeded: You have requested more instances (7) than your current instance limit of 5 allows for the specified instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.
3. Increase limit for worker instance type.
EC2 instances created and referenced by Machines.
No Nodes are created for the Machines.
Nodes are created for all Machines that have EC2 instances.
Created attachment 1593429 [details]
Created attachment 1593430 [details]
Created attachment 1593431 [details]
Probably this is similar to https://bugzilla.redhat.com/show_bug.cgi?id=1723955
look at `oc get csr`.
This is expected when aws limit is reached. You'll get a prometheus alert due to missmatch between nodes and machines and once the instance is created, you'll need to manually approve by design. If anything we could try to store the timestamp for the time the instance was actually created so machine approver will consider it legit so bumping to 4.3 to farther considering that
Since we introduced machine phases this should be reflected in the machine phase as provisioning/failed giving a more meaningful output in addition to the alerts. Also multiple fixes were merge for the machine approver which tolerates a bigger timeout now. I'm closing this please reopen if still relevant