Description of problem: An OSD cluster was provisioned in an account where AWS limits were not sufficient to fulfill all EC2 instances requested resulting in an initial cluster with 3 masters and 2 workers. When the limits were updated the additional EC2 instances (+2) were created and associated with the Machines in cluster. But no corresponding Node was created in the cluster. Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.7 True False 10h Cluster version is 4.1.7 How reproducible: Intermittent. Steps to Reproduce: 1. Provision cluster in account with EC2 limit for workers lower than desired number of machines. 2. Observe failure to provision nodes. $ oc logs machine-api-controllers-5d957c6fd-qzxml -c controller-manager ... E0725 08:39:47.357292 1 instances.go:309] Error creating EC2 instance: InstanceLimitExceeded: You have requested more instances (7) than your current instance limit of 5 allows for the specified instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit. ... 3. Increase limit for worker instance type. Actual results: EC2 instances created and referenced by Machines. No Nodes are created for the Machines. Expected results: Nodes are created for all Machines that have EC2 instances. Additional info:
Created attachment 1593429 [details] nodelink-controller log
Created attachment 1593430 [details] machine-controller log
Created attachment 1593431 [details] controller-manager log
Probably this is similar to https://bugzilla.redhat.com/show_bug.cgi?id=1723955 look at `oc get csr`.
This is expected when aws limit is reached. You'll get a prometheus alert due to missmatch between nodes and machines and once the instance is created, you'll need to manually approve by design. If anything we could try to store the timestamp for the time the instance was actually created so machine approver will consider it legit so bumping to 4.3 to farther considering that
See https://github.com/openshift/cluster-machine-approver/issues/36
Since we introduced machine phases this should be reflected in the machine phase as provisioning/failed giving a more meaningful output in addition to the alerts. Also multiple fixes were merge for the machine approver which tolerates a bigger timeout now. I'm closing this please reopen if still relevant