Bug 2043675 - Too many machines deleted by cluster autoscaler when scaling down
Summary: Too many machines deleted by cluster autoscaler when scaling down
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.10
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.10.0
Assignee: Michael McCune
QA Contact: sunzhaohua
Depends On:
TreeView+ depends on / blocked
Reported: 2022-01-21 17:34 UTC by Michael McCune
Modified: 2022-03-10 16:42 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2022-03-10 16:41:20 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift kubernetes-autoscaler pull 219 0 None open Bug 2043675: UPSTREAM: <drop>: Ensure ClusterAPI DeleteNodes accounts for out of band changes scale 2022-01-21 17:35:40 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:42:08 UTC

Description Michael McCune 2022-01-21 17:34:37 UTC
Description of problem:
When the autoscaler is scaling down machines, it is occasionally possible for it to delete too many machines. This happens due to a race condition which can become evident in the cluster autoscaler when more than one machine is deleted. If the autoscaler is deleting multiple machines it is possible for its cache of the replica count to become stale across the multiple deletion threads, which leads to the autoscaler deleting extra machines.

Version-Release number of selected component (if applicable):

How reproducible:
Sometimes. Because this condition only happens when there is a race with the stale cache it doesn't always express itself.

Steps to Reproduce:
1. Create a ClusterAutoscaler
2. Create a MachineAutoscaler for one of the MachineSets with a maximum size greater than 3 (to allow for a multiple delete scenario)
3. Create a workload that will expand the cluster through the autoscaler
4. Reduce the scale of the workload so that only one Machine would be deleted

Actual results:
When this bug occurs, the autoscaler will delete an extra machine and then sometime later will create a new machine.

Expected results:
Autoscaler deletes the appropriate number of machines.

Additional info:

Comment 3 sunzhaohua 2022-01-25 09:12:05 UTC
clusterversion: 4.10.0-0.nightly-2022-01-25-023600
Didn't find extra machine was deleted, all work as expected.

Comment 6 errata-xmlrpc 2022-03-10 16:41:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.