Bug 2043675

Summary: Too many machines deleted by cluster autoscaler when scaling down
Product: OpenShift Container Platform Reporter: Michael McCune <mimccune>
Component: Cloud ComputeAssignee: Michael McCune <mimccune>
Cloud Compute sub component: Cluster Autoscaler QA Contact: sunzhaohua <zhsun>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: aos-bugs
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:41:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michael McCune 2022-01-21 17:34:37 UTC 
Description of problem:
When the autoscaler is scaling down machines, it is occasionally possible for it to delete too many machines. This happens due to a race condition which can become evident in the cluster autoscaler when more than one machine is deleted. If the autoscaler is deleting multiple machines it is possible for its cache of the replica count to become stale across the multiple deletion threads, which leads to the autoscaler deleting extra machines.

Version-Release number of selected component (if applicable):
4.10.0

How reproducible:
Sometimes. Because this condition only happens when there is a race with the stale cache it doesn't always express itself.

Steps to Reproduce:
1. Create a ClusterAutoscaler
2. Create a MachineAutoscaler for one of the MachineSets with a maximum size greater than 3 (to allow for a multiple delete scenario)
3. Create a workload that will expand the cluster through the autoscaler
4. Reduce the scale of the workload so that only one Machine would be deleted

Actual results:
When this bug occurs, the autoscaler will delete an extra machine and then sometime later will create a new machine.

Expected results:
Autoscaler deletes the appropriate number of machines.

Additional info:
Comment 3 sunzhaohua 2022-01-25 09:12:05 UTC 
verified
clusterversion: 4.10.0-0.nightly-2022-01-25-023600
Didn't find extra machine was deleted, all work as expected.
Comment 6 errata-xmlrpc 2022-03-10 16:41:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056