Bug 1819029

Summary: How to handle alert ClusterAutoscalerUnschedulablePods
Product: OpenShift Container Platform Reporter: Hongkai Liu <hongkliu>
Component: Cloud ComputeAssignee: Michael McCune <mimccune>
Cloud Compute sub component: Other Providers QA Contact: sunzhaohua <zhsun>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: unspecified CC: agarcial, mgugino
Version: 4.3.0   
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1820654 (view as bug list) Environment:
Last Closed: 2020-10-27 15:56:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1827307    
Attachments:
Description Flags
openshift-machine-api.cluster-autoscaler-default-5476d56447-5ww92.24h.log
none
openshift-machine-api.machine-api-controllers-7c696b9657-m8t4c.machine-controller.24h.log
none
prometheus.query none

Description Hongkai Liu 2020-03-31 01:33:50 UTC
Description of problem:
AlertManager on OCP4.3 cluster (with autoscaler configured) fired this alert in the afternoon todoy.

[FIRING:1] ClusterAutoscalerUnschedulablePods cluster-autoscaler-default (metrics 10.130.0.16:8085 openshift-machine-api cluster-autoscaler-default-5476d56447-5ww92 openshift-monitoring/k8s cluster-autoscaler-default warning)
Cluster Autoscaler has 32 unschedulable pods

https://coreos.slack.com/archives/CHY2E1BL4/p1585600389050900


Version-Release number of selected component (if applicable):
oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.0-0.nightly-2020-03-23-130439   True        False         6d12h   Cluster version is 4.3.0-0.nightly-2020-03-23-130439

How reproducible:
Got this alert once only

Could I expect no such alerts when autoscaler works?
Or what should I do when seeing the alert?

Additional info:
Will attache the pod logs and prometheus screenshot.

Comment 1 Hongkai Liu 2020-03-31 01:35:40 UTC
Created attachment 1674917 [details]
openshift-machine-api.cluster-autoscaler-default-5476d56447-5ww92.24h.log

Comment 2 Hongkai Liu 2020-03-31 01:36:54 UTC
Created attachment 1674918 [details]
openshift-machine-api.machine-api-controllers-7c696b9657-m8t4c.machine-controller.24h.log

Comment 3 Hongkai Liu 2020-03-31 01:38:02 UTC
Created attachment 1674920 [details]
prometheus.query

Comment 4 Michael Gugino 2020-03-31 17:23:27 UTC
This alert is caused by the cluster autoscaler's inability to scale up.  This alert is normal and expected depending on cluster autoscaler's configuration.  In this particular case, there is a bug in the cluster autoscaler.  I'm going to open a new BZ and link it here.

In the mean time, this bug should remain open until we document the cause and remedy of this particular alert under normal circumstances.

Comment 5 Hongkai Liu 2020-03-31 20:06:06 UTC
Thank Michael for help me fix the autoscaler.

Comment 7 Joel Speed 2020-05-13 15:10:12 UTC
Assigning to Michael McCune as he has a Jira card to document all of the alerts over the next sprint

Comment 8 Alberto 2020-05-29 10:54:27 UTC
tagging with upcomingSprint to re-evaluate priority.

Comment 9 Michael McCune 2020-06-19 20:20:11 UTC
just adding a note here that i am starting to investigate this issue.

Comment 10 Michael McCune 2020-06-24 18:16:06 UTC
i think the next best action we can take is to start creating a document for the cluster-autoscaler-operator to document these alerts and possible guidance around them. Michael Gugino started a pull request[0] for the machine-api-operator to document those alerts, we should do the same for the cluster-autoscaler-operator.


[0] https://github.com/openshift/machine-api-operator/pull/606

Comment 11 Michael McCune 2020-06-24 18:20:37 UTC
i have created an issue on the cluster-autoscaler-operator to track this: https://github.com/openshift/cluster-autoscaler-operator/issues/153

Comment 12 Michael McCune 2020-08-17 19:11:43 UTC
ideally we will have a PR in place for the documentation in the next sprint.

Comment 17 errata-xmlrpc 2020-10-27 15:56:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196