Description of problem: When the autoscaler is scaling down machines, it is occasionally possible for it to delete too many machines. This happens due to a race condition which can become evident in the cluster autoscaler when more than one machine is deleted. If the autoscaler is deleting multiple machines it is possible for its cache of the replica count to become stale across the multiple deletion threads, which leads to the autoscaler deleting extra machines. Version-Release number of selected component (if applicable): 4.10.0 How reproducible: Sometimes. Because this condition only happens when there is a race with the stale cache it doesn't always express itself. Steps to Reproduce: 1. Create a ClusterAutoscaler 2. Create a MachineAutoscaler for one of the MachineSets with a maximum size greater than 3 (to allow for a multiple delete scenario) 3. Create a workload that will expand the cluster through the autoscaler 4. Reduce the scale of the workload so that only one Machine would be deleted Actual results: When this bug occurs, the autoscaler will delete an extra machine and then sometime later will create a new machine. Expected results: Autoscaler deletes the appropriate number of machines. Additional info:
verified clusterversion: 4.10.0-0.nightly-2022-01-25-023600 Didn't find extra machine was deleted, all work as expected.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056