Bug 1732929 - 4.1.4 cluster creating 400+ eviction requests a second
Summary: 4.1.4 cluster creating 400+ eviction requests a second
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.1.z
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.1.z
Assignee: Vikas Choudhary
QA Contact: sunzhaohua
URL:
Whiteboard:
Depends On: 1733708
Blocks: 1733922
TreeView+ depends on / blocked
 
Reported: 2019-07-24 18:16 UTC by Clayton Coleman
Modified: 2019-09-27 00:34 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1733708 (view as bug list)
Environment:
Last Closed: 2019-09-27 00:33:52 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-api-actuator-pkg pull 115 0 None closed Bug 1732929: Pin to machine-api-operator to release-4.1 branch 2020-03-18 02:52:13 UTC
Github openshift cluster-api-provider-aws pull 253 0 None closed Bug 1732929: Pin dependencies to 4.1 2020-03-18 02:52:13 UTC
Github openshift kubernetes-drain pull 3 0 None closed Bug 1732929: Fix eviction failure scenario 2020-03-18 02:52:13 UTC
Github openshift machine-api-operator pull 393 0 None closed cherry-picking #366: Bug 1732929: Modify machine-api-controller clusterrole for daemonsets 2020-03-18 02:52:13 UTC
Red Hat Product Errata RHBA-2019:2856 0 None None None 2019-09-27 00:34:00 UTC

Description Clayton Coleman 2019-07-24 18:16:18 UTC 
A 4.1.4 cluster identified via telemetry was reporting 500+ 429 requests a second via the API.  Upon inspection of the cluster the machine-controller-manager was observed generating 400+ pod eviction requests (that are all errors) a second continuously.

The cluster is available for inspection, the top three requests are all abnormal and a result of the very large number of eviction requests:

{client="machine-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format",code="429",endpoint="https",job="apiserver",namespace="default",resource="pods",scope="namespace",service="kubernetes",subresource="eviction",verb="CREATE"}	405.99523809523805
{client="machine-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format",code="429",contentType="application/json",endpoint="https",job="apiserver",namespace="default",resource="pods",scope="namespace",service="kubernetes",subresource="eviction",verb="POST"}	50
{client="openshift-kube-apiserver/v1.13.4+c62ce01 (linux/amd64) kubernetes/c62ce01",code="200",contentType="application/vnd.kubernetes.protobuf",endpoint="https",job="apiserver",namespace="default",resource="poddisruptionbudgets",scope="namespace",service="kubernetes",verb="LIST"}	50
{client="openshift-apiserver/v1.13.4+c62ce01 (linux/amd64) kubernetes/c62ce01",code="429",endpoint="https",job="apiserver",namespace="default",resource="subjectaccessreviews",scope="cluster",service="kubernetes",verb="CREATE"}	12.873109243697478
Comment 1 Clayton Coleman 2019-07-24 18:18:41 UTC 
There are 5 clusters reporting over 300 429-response code errors per second, which means this is likely impacting multiple.
Comment 2 Michael Gugino 2019-07-25 02:03:33 UTC 
openshift/kubernetes drain uses a goroutine for every pod that needs to be drained on a node.  If 429 is returned (can't do eviction, probably PDB), it sleeps 5 seconds before trying again.  There's a 20 second timeout on draining that we configure in the machine-controller, and we delay requeue of the machine-object delete request after a drain fails.

My guess is that a large scale-down (many machines) took place, and there's not a place to schedule all the pods with PDBs.

In 4.1, machine controller only drains when deleting, and drains (evictions) only get 429 from PDBs.
Comment 3 Vikas Choudhary 2019-07-25 03:02:57 UTC 
Behavior is as expected.

There is a pdb object in the namespace which says "minAvailable: 1".
There is a replicaset which says replica 1:
```
$ kubectl get replicaset -n istio-system
NAME                               DESIRED   CURRENT   READY     AGE
istio-pilot-77d5cd65b4             1         1         1         9d
```

PDB wont let the pod evict.
Now a machine is tried to be deleted. Corresponding node has such a pod. Machine controller calls drain api. drain api marks the node unschedulable and then it tries to evict pod. This pod eviction fails forever and it keeps retrying to evict. Node gets stuck in unschedulable state and machine never gets deleted.
Comment 4 Vikas Choudhary 2019-07-25 03:05:04 UTC 
I verified above in one of the clusters where this is happening. Also this is very easy to reproduce and verify.
Comment 5 Michael Gugino 2019-07-25 04:55:49 UTC 
(In reply to Vikas Choudhary from comment #4)
> I verified above in one of the clusters where this is happening. Also this
> is very easy to reproduce and verify.

How many machines are being deleted?  It shouldn't be spamming 300+ per second for a single pod.  There should be 5 seconds between each 429 response, timeout after 20 seconds, and then it should wait a bit longer before attempting to drain the node again.  If we're seeing 300+ requests/sec for one pod, something is broken.
Comment 6 Clayton Coleman 2019-07-25 14:52:38 UTC 
The possibility that drain is impossible needs to be part of the design of the drain code, so something akin to image pull backoff is of course necessary.  Setting severity appropriately.
Comment 7 Vikas Choudhary 2019-07-25 14:58:52 UTC 
(In reply to Michael Gugino from comment #5)
> (In reply to Vikas Choudhary from comment #4)
> > I verified above in one of the clusters where this is happening. Also this
> > is very easy to reproduce and verify.
> 
> How many machines are being deleted?  It shouldn't be spamming 300+ per
> second for a single pod.  There should be 5 seconds between each 429
> response, timeout after 20 seconds, and then it should wait a bit longer
> before attempting to drain the node again.  If we're seeing 300+
> requests/sec for one pod, something is broken.

retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.959919       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.977066       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:44.977109       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.981985       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.982317       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.982477       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.982994       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.983569       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.990721       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:44.991017       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.991579       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.002211       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.003483       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.011189       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.019750       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.020467       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.075287       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.084313       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.084433       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.085064       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.107962       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.109397       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.109540       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.109574       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.176538       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.191653       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.236642       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.236979       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.300564       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.302414       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.303310       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.303466       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.315951       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.338997       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.356249       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.474960       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.505237       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.505467       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.505537       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.506447       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.506570       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.506635       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.506721       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.506868       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.507118       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.537281       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.537305       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.613639       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.664553       1 info.go:2^C
[vikas@fed-node cluster-machine-approver](metrics)$ tail -f ~/mao.txt 
I0724 18:39:35.259138       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.259444       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.259597       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.259677       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.259757       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.259958       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.260495       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.260832       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.269729       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.274442       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.282955       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.283030       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.283106       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.288507       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.307440       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.312153       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.313205       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.423372       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424057       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424189       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424292       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424512       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424611       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.434624       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.441138       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.447383       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.482034       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.550617       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.551986       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.551993       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.552062       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.553700       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.553845       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.554660       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.574702       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.577527       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.579395       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.579522       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.579940       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.579993       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.580040       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.580090       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.580371       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.594049       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.595020       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.639244       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.672308       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.682214       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.687674       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.687979       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.757075       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.765587       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.765796       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.765912       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.765956       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.766935       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.781010       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.786203       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.792539       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.792700       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.792780       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.810536       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.810724       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.810954       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.811027       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.811188       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.811355       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.842538       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.927203       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.931289       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.932451       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.932628       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.932864       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.933253       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.933548       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.933642       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.934974       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.936654       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.981989       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.982534       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.037300       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.037402       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.037650       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.065920       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.163720       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.184569       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.194635       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.200734       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.200942       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.215508       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.216521       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.304595       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.305124       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.306302       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.310066       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.360106       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.360238       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.361121       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.361216       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.367227       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.412259       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.447784       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.455159       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.458172       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.458302       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.478403       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.478580       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.479503       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.479750       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.480155       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.541482       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.541636       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.549706       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.551879       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.552062       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.580026       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.670595       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.670713       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.670813       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.670875       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.671028       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.679959       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.685252       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.685514       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.685760       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.685886       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.689558       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.696567       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.700508       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.700669       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.701578       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.705307       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.705624       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.729992       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.749724       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.761134       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.771029       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.771041       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.771140       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.786566       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.786708       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.786811       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.787849       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.824770       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.827841       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.840346       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.840779       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.851168       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.862859       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.872753       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.876348       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.877248       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.881959       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.028822       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.178834       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.178989       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.179924       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.180045       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.180204       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.180321       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.181119       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.181317       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.181413       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.181518       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.311999       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.312103       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.337951       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.349415       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.349557       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.349844       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.355214       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.380357       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.393520       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.459721       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.460476       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.461264       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.461370       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.466953       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.474567       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.482062       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.484636       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.501154       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.514110       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.546175       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.546881       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.547122       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.547242       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.549993       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.550086       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.621112       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.621242       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.621316       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.621421       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.621493       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.621726       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.703456       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.717831       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.722451       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.787320       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.790826       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.793607       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.794001       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.795007       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.834800       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.844770       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.845808       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.907411       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.996493       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.044766       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.062074       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.075158       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.304723       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.323828       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.353856       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.424610       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.425100       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.427284       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.445566       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.445726       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.446120       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.446450       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.446743       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.446759       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.447321       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.447406       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.456249       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.456268       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.490355       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.490469       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.490587       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
Comment 8 Michael Gugino 2019-07-25 15:01:04 UTC 
(In reply to Clayton Coleman from comment #6)
> The possibility that drain is impossible needs to be part of the design of
> the drain code, so something akin to image pull backoff is of course
> necessary.  Setting severity appropriately.

There should be a delay.  If this (many requests per second) is caused by pod-count-of-one, then it's a bug in that logic.  There's no scenario where you should see hundreds (or even dozens) requests per second for only one pod drain blocked by pdb.
Comment 9 Vikas Choudhary 2019-07-27 18:46:23 UTC 
In the drain logic, a goroutine is run for evicting each pod running on the node. If the eviction fails with error "too many requests", goroutine keeps creating eviction requests forever with a sleep of 5 sec. After global timeout, Drain() returns but the goroutine for the pods continues to create eviction requests.

machine controller gets error from the Drain() and then after waiting for 20 secs again invokes Drain(). Again goroutine for eviction of each pod gets created. And this way after each Drain() return and a wait of 20 secs, goroutines for making eviction requesting keep getting created.

Created fix PR, https://github.com/openshift/kubernetes-drain/pull/3, which is ensuring that after global timeout, all the goroutines return before the return of the Drain()

This problem is so serious that cluster will die if even a single pod failed to evict.
Comment 10 Vikas Choudhary 2019-07-27 19:46:14 UTC 
Steps to reproduce the issue:

1. Create the pdb with minAvailable 1:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      "app": "nginx"


2. Create a replicaset with 1 replica and labels as `"app": "nginx"`

3. Find out the machine for the node on which pod is running.
4. Delete the machine

Machine deletion command will not return.
In another terminal, do a `tail -f` on machine-controller logs. You will see logs like below:
late the pod's disruption budget.
I0727 19:40:08.240725       1 info.go:20] error when evicting pod "pdb-workload-q24lb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0727 19:40:13.250043       1 info.go:20] error when evicting pod "pdb-workload-q24lb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0727 19:40:18.265153       1 info.go:20] error when evicting pod "pdb-workload-q24lb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
 
In couple of minutes, you will see that time difference between the two eviction failure logs is decreasing. Failure logs will keep on increasing per second with time.


After applying the fix, you will see that time difference is always constant 5 secs.
Comment 11 Vikas Choudhary 2019-07-27 19:50:42 UTC 
WORKAROUND:
Delete the pdb object which is making the eviction fail. 

PREVENTIVE STEP:
PDB configuration and replica count should not be such that it could lead to eviction failure. For example, if minAvailable is 1 in the PDB and replica count is 1, this pod eviction will fail for sure and the issue will occur.
Comment 12 Alberto 2019-07-29 08:39:22 UTC 
Since this is in POST can you please link all the relevant PRs. Targeting this to 4.1.z and using https://bugzilla.redhat.com/show_bug.cgi?id=1732929 for tracking 4.2
Comment 13 Vikas Choudhary 2019-07-30 06:03:00 UTC 
https://github.com/openshift/cluster-api/pull/54
https://github.com/openshift/cluster-api-provider-aws/pull/243
Comment 15 sunzhaohua 2019-09-23 05:00:05 UTC 
Verified
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-0.nightly-2019-09-21-195243   True        False         37m     Cluster version is 4.1.0-0.nightly-2019-09-21-195243

Eviction retry is skipped after global timeout.

I0923 04:53:25.927140       1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7"
I0923 04:53:25.927265       1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0923 04:53:25.927333       1 controller.go:193] Reconciling machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" triggers delete
I0923 04:53:25.976363       1 info.go:20] cordoned node "ip-10-0-147-193.us-east-2.compute.internal"
I0923 04:53:26.031409       1 info.go:16] ignoring DaemonSet-managed pods: tuned-4fznn, dns-default-8dg7d, node-ca-86xxw, machine-config-daemon-sswxr, node-exporter-d6v8f, multus-gnbbb, ovs-zrd97, sdn-9qm7f; deleting pods with local storage: alertmanager-main-1, kube-state-metrics-85bf9ffbbf-xkxxp, prometheus-adapter-76659fb787-p5dj2, prometheus-k8s-1
I0923 04:53:26.058436       1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0923 04:53:26.337250       1 info.go:20] pod "alertmanager-main-1" removed (evicted)
I0923 04:53:31.065423       1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0923 04:53:32.333409       1 info.go:20] pod "router-default-555fb8b466-8z7df" removed (evicted)
I0923 04:53:32.733204       1 info.go:20] pod "telemeter-client-7f5664b4bb-44xhw" removed (evicted)
I0923 04:53:33.933347       1 info.go:20] pod "community-operators-644fbcf4bb-9wnz5" removed (evicted)
I0923 04:53:34.933416       1 info.go:20] pod "prometheus-adapter-76659fb787-p5dj2" removed (evicted)
I0923 04:53:35.133178       1 info.go:20] pod "kube-state-metrics-85bf9ffbbf-xkxxp" removed (evicted)
I0923 04:53:36.071147       1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0923 04:53:39.107004       1 info.go:20] pod "prometheus-operator-67f988789d-mpfn4" removed (evicted)
I0923 04:53:39.129496       1 info.go:20] pod "image-registry-7c595cdbbf-fdck4" removed (evicted)
I0923 04:53:39.140807       1 info.go:20] pod "prometheus-k8s-1" removed (evicted)
I0923 04:53:41.077216       1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0923 04:53:46.034981       1 info.go:20] Closing stopCh
I0923 04:53:46.122128       1 info.go:16] ignoring DaemonSet-managed pods: tuned-4fznn, dns-default-8dg7d, node-ca-86xxw, machine-config-daemon-sswxr, node-exporter-d6v8f, multus-gnbbb, ovs-zrd97, sdn-9qm7f
I0923 04:53:46.122160       1 info.go:20] failed to evict pods from node "ip-10-0-147-193.us-east-2.compute.internal" (pending pods: nginx-7499b4d6d9-2kjzf): global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf"
I0923 04:53:46.122177       1 info.go:16] global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf"
I0923 04:53:46.122184       1 info.go:20] unable to drain node "ip-10-0-147-193.us-east-2.compute.internal"
I0923 04:53:46.122191       1 info.go:20] there are pending nodes to be drained: ip-10-0-147-193.us-east-2.compute.internal
W0923 04:53:46.122199       1 controller.go:286] drain failed for machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7": global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf"
E0923 04:53:46.122209       1 controller.go:202] Failed to drain node for machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7": requeue in: 20s
I0923 04:53:46.122221       1 controller.go:352] Actuator returned requeue-after error: requeue in: 20s
I0923 04:53:46.122291       1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7"
I0923 04:53:46.122314       1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0923 04:53:46.129353       1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7"
I0923 04:53:46.129375       1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0923 04:53:46.129387       1 actuator.go:378] Checking if machine exists
I0923 04:53:46.174015       1 actuator.go:386] Instance does not exist
I0923 04:53:46.174041       1 controller.go:247] Reconciling machine object zhsun41171-rhch4-worker-us-east-2b-rvnq7 triggers idempotent create.
I0923 04:53:46.174054       1 actuator.go:110] creating machine
E0923 04:53:46.174321       1 utils.go:186] NodeRef not found in machine zhsun41171-rhch4-worker-us-east-2b-rvnq7
I0923 04:53:46.185083       1 instances.go:44] No stopped instances found for machine zhsun41171-rhch4-worker-us-east-2b-rvnq7
I0923 04:53:46.185130       1 instances.go:141] Using AMI ami-0649fd5d42859bdfc
I0923 04:53:46.185141       1 instances.go:73] Describing security groups based on filters
I0923 04:53:46.328151       1 instances.go:118] Describing subnets based on filters
I0923 04:53:47.466094       1 actuator.go:475] Updating status
I0923 04:53:47.466122       1 actuator.go:519] finished calculating AWS status
I0923 04:53:47.466224       1 actuator.go:144] machine status has changed, updating
I0923 04:53:47.474756       1 actuator.go:537] Instance state still pending, returning an error to requeue
W0923 04:53:47.474788       1 controller.go:249] Failed to create machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7": requeue in: 20s
I0923 04:53:47.474810       1 controller.go:352] Actuator returned requeue-after error: requeue in: 20s
I0923 04:53:47.474863       1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7"
I0923 04:53:47.474878       1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0923 04:53:47.474893       1 actuator.go:378] Checking if machine exists
I0923 04:53:47.625378       1 actuator.go:391] Instance exists as "i-032f759ca23f14142"
I0923 04:53:47.625403       1 controller.go:238] Reconciling machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" triggers idempotent update
I0923 04:53:47.625413       1 actuator.go:305] updating machine
I0923 04:53:47.625501       1 actuator.go:313] obtaining EC2 client for region
I0923 04:53:47.671802       1 actuator.go:331] found 1 existing instances for machine
I0923 04:53:47.671850       1 actuator.go:475] Updating status
I0923 04:53:47.671915       1 actuator.go:519] finished calculating AWS status
I0923 04:53:47.672085       1 actuator.go:153] status unchanged
I0923 04:53:47.672101       1 actuator.go:537] Instance state still pending, returning an error to requeue
E0923 04:53:47.672114       1 controller.go:240] Error updating machine "openshift-machine-api/zhsun41171-rhch4-worker-us-east-2b-rvnq7": requeue in: 20s
I0923 04:53:47.672130       1 controller.go:352] Actuator returned requeue-after error: requeue in: 20s
I0923 04:54:06.122503       1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7"
I0923 04:54:06.122532       1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster
I0923 04:54:06.122544       1 controller.go:193] Reconciling machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" triggers delete
I0923 04:54:06.165829       1 info.go:16] ignoring DaemonSet-managed pods: tuned-4fznn, dns-default-8dg7d, node-ca-86xxw, machine-config-daemon-sswxr, node-exporter-d6v8f, multus-gnbbb, ovs-zrd97, sdn-9qm7f
I0923 04:54:06.174594       1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0923 04:54:11.180953       1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0923 04:54:16.187935       1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0923 04:54:21.194214       1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0923 04:54:26.169560       1 info.go:20] Closing stopCh
I0923 04:54:26.237956       1 info.go:16] ignoring DaemonSet-managed pods: tuned-4fznn, dns-default-8dg7d, node-ca-86xxw, machine-config-daemon-sswxr, node-exporter-d6v8f, multus-gnbbb, ovs-zrd97, sdn-9qm7f
I0923 04:54:26.237980       1 info.go:20] failed to evict pods from node "ip-10-0-147-193.us-east-2.compute.internal" (pending pods: nginx-7499b4d6d9-2kjzf): global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf"
I0923 04:54:26.237991       1 info.go:16] global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf"
I0923 04:54:26.237997       1 info.go:20] unable to drain node "ip-10-0-147-193.us-east-2.compute.internal"
I0923 04:54:26.238004       1 info.go:20] there are pending nodes to be drained: ip-10-0-147-193.us-east-2.compute.internal
W0923 04:54:26.238018       1 controller.go:286] drain failed for machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7": global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf"
E0923 04:54:26.238034       1 controller.go:202] Failed to drain node for machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7": requeue in: 20s
I0923 04:54:26.238048       1 controller.go:352] Actuator returned requeue-after error: requeue in: 20s
Comment 17 errata-xmlrpc 2019-09-27 00:33:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2856
Note You need to log in before you can comment on or make changes to this bug.