Bug 1733708 - 4.1.4 cluster creating 400+ eviction requests a second
Summary: 4.1.4 cluster creating 400+ eviction requests a second
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.2.0
Assignee: Alberto
QA Contact: Jianwei Hou
URL:
Whiteboard:
: 1733922 (view as bug list)
Depends On:
Blocks: 1732929 1733922
TreeView+ depends on / blocked
 
Reported: 2019-07-27 19:31 UTC by Vikas Choudhary
Modified: 2019-10-16 06:34 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1732929
: 1733922 (view as bug list)
Environment:
Last Closed: 2019-10-16 06:33:51 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-api-provider-aws pull 251 None closed Bug 1733708: Bump cluster-api to include native drain lib with fixes 2020-02-03 10:04:29 UTC
Red Hat Product Errata RHBA-2019:2922 None None None 2019-10-16 06:34:05 UTC

Description Vikas Choudhary 2019-07-27 19:31:40 UTC
+++ This bug was initially created as a clone of Bug #1732929 +++

A 4.1.4 cluster identified via telemetry was reporting 500+ 429 requests a second via the API.  Upon inspection of the cluster the machine-controller-manager was observed generating 400+ pod eviction requests (that are all errors) a second continuously.

The cluster is available for inspection, the top three requests are all abnormal and a result of the very large number of eviction requests:

{client="machine-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format",code="429",endpoint="https",job="apiserver",namespace="default",resource="pods",scope="namespace",service="kubernetes",subresource="eviction",verb="CREATE"}	405.99523809523805
{client="machine-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format",code="429",contentType="application/json",endpoint="https",job="apiserver",namespace="default",resource="pods",scope="namespace",service="kubernetes",subresource="eviction",verb="POST"}	50
{client="openshift-kube-apiserver/v1.13.4+c62ce01 (linux/amd64) kubernetes/c62ce01",code="200",contentType="application/vnd.kubernetes.protobuf",endpoint="https",job="apiserver",namespace="default",resource="poddisruptionbudgets",scope="namespace",service="kubernetes",verb="LIST"}	50
{client="openshift-apiserver/v1.13.4+c62ce01 (linux/amd64) kubernetes/c62ce01",code="429",endpoint="https",job="apiserver",namespace="default",resource="subjectaccessreviews",scope="cluster",service="kubernetes",verb="CREATE"}	12.873109243697478

--- Additional comment from Clayton Coleman on 2019-07-24 18:18:41 UTC ---

There are 5 clusters reporting over 300 429-response code errors per second, which means this is likely impacting multiple.

--- Additional comment from Michael Gugino on 2019-07-25 02:03:33 UTC ---

openshift/kubernetes drain uses a goroutine for every pod that needs to be drained on a node.  If 429 is returned (can't do eviction, probably PDB), it sleeps 5 seconds before trying again.  There's a 20 second timeout on draining that we configure in the machine-controller, and we delay requeue of the machine-object delete request after a drain fails.

My guess is that a large scale-down (many machines) took place, and there's not a place to schedule all the pods with PDBs.

In 4.1, machine controller only drains when deleting, and drains (evictions) only get 429 from PDBs.

--- Additional comment from Vikas Choudhary on 2019-07-25 03:02:57 UTC ---

Behavior is as expected.

There is a pdb object in the namespace which says "minAvailable: 1".
There is a replicaset which says replica 1:
```
$ kubectl get replicaset -n istio-system
NAME                               DESIRED   CURRENT   READY     AGE
istio-pilot-77d5cd65b4             1         1         1         9d
```

PDB wont let the pod evict.
Now a machine is tried to be deleted. Corresponding node has such a pod. Machine controller calls drain api. drain api marks the node unschedulable and then it tries to evict pod. This pod eviction fails forever and it keeps retrying to evict. Node gets stuck in unschedulable state and machine never gets deleted.

--- Additional comment from Vikas Choudhary on 2019-07-25 03:05:04 UTC ---

I verified above in one of the clusters where this is happening. Also this is very easy to reproduce and verify.

--- Additional comment from Michael Gugino on 2019-07-25 04:55:49 UTC ---

(In reply to Vikas Choudhary from comment #4)
> I verified above in one of the clusters where this is happening. Also this
> is very easy to reproduce and verify.

How many machines are being deleted?  It shouldn't be spamming 300+ per second for a single pod.  There should be 5 seconds between each 429 response, timeout after 20 seconds, and then it should wait a bit longer before attempting to drain the node again.  If we're seeing 300+ requests/sec for one pod, something is broken.

--- Additional comment from Clayton Coleman on 2019-07-25 14:52:38 UTC ---

The possibility that drain is impossible needs to be part of the design of the drain code, so something akin to image pull backoff is of course necessary.  Setting severity appropriately.

--- Additional comment from Vikas Choudhary on 2019-07-25 14:58:52 UTC ---

(In reply to Michael Gugino from comment #5)
> (In reply to Vikas Choudhary from comment #4)
> > I verified above in one of the clusters where this is happening. Also this
> > is very easy to reproduce and verify.
> 
> How many machines are being deleted?  It shouldn't be spamming 300+ per
> second for a single pod.  There should be 5 seconds between each 429
> response, timeout after 20 seconds, and then it should wait a bit longer
> before attempting to drain the node again.  If we're seeing 300+
> requests/sec for one pod, something is broken.

retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.959919       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.977066       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:44.977109       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.981985       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.982317       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.982477       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.982994       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.983569       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.990721       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:44.991017       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:44.991579       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.002211       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.003483       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.011189       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.019750       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.020467       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.075287       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.084313       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.084433       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.085064       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.107962       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.109397       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.109540       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.109574       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.176538       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.191653       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.236642       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.236979       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.300564       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.302414       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.303310       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.303466       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.315951       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.338997       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.356249       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.474960       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.505237       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.505467       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.505537       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.506447       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.506570       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.506635       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.506721       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.506868       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.507118       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.537281       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:38:45.537305       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.613639       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:38:45.664553       1 info.go:2^C
[vikas@fed-node cluster-machine-approver](metrics)$ tail -f ~/mao.txt 
I0724 18:39:35.259138       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.259444       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.259597       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.259677       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.259757       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.259958       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.260495       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.260832       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.269729       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.274442       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.282955       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.283030       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.283106       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.288507       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.307440       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.312153       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.313205       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.423372       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424057       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424189       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424292       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424512       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.424611       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.434624       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.441138       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.447383       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.482034       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.550617       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.551986       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.551993       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.552062       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.553700       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.553845       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.554660       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.574702       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.577527       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.579395       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.579522       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.579940       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.579993       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.580040       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.580090       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.580371       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.594049       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.595020       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.639244       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.672308       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.682214       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.687674       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.687979       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.757075       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.765587       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.765796       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.765912       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.765956       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.766935       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.781010       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.786203       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.792539       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.792700       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.792780       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.810536       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.810724       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.810954       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.811027       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.811188       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.811355       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.842538       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.927203       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.931289       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.932451       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.932628       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.932864       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.933253       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.933548       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.933642       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.934974       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:35.936654       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.981989       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:35.982534       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.037300       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.037402       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.037650       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.065920       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.163720       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.184569       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.194635       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.200734       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.200942       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.215508       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.216521       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.304595       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.305124       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.306302       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.310066       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.360106       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.360238       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.361121       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.361216       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.367227       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.412259       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.447784       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.455159       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.458172       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.458302       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.478403       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.478580       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.479503       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.479750       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.480155       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.541482       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.541636       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.549706       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.551879       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.552062       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.580026       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.670595       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.670713       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.670813       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.670875       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.671028       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.679959       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.685252       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.685514       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.685760       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.685886       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.689558       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.696567       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.700508       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.700669       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.701578       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.705307       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.705624       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.729992       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.749724       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.761134       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.771029       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.771041       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.771140       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.786566       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.786708       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.786811       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.787849       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.824770       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.827841       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.840346       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.840779       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.851168       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:36.862859       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.872753       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.876348       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.877248       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:36.881959       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.028822       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.178834       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.178989       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.179924       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.180045       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.180204       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.180321       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.181119       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.181317       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.181413       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.181518       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.311999       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.312103       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.337951       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.349415       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.349557       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.349844       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.355214       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.380357       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.393520       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.459721       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.460476       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.461264       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.461370       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.466953       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.474567       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.482062       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.484636       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.501154       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.514110       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.546175       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.546881       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.547122       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.547242       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.549993       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.550086       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.621112       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.621242       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.621316       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.621421       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.621493       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.621726       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.703456       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.717831       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.722451       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.787320       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.790826       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.793607       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.794001       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.795007       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.834800       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.844770       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.845808       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:37.907411       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:37.996493       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.044766       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.062074       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.075158       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.304723       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.323828       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.353856       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.424610       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.425100       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.427284       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.445566       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.445726       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.446120       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.446450       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.446743       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.446759       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.447321       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.447406       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.456249       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.456268       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.490355       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0724 18:39:38.490469       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz)
I0724 18:39:38.490587       1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

--- Additional comment from Michael Gugino on 2019-07-25 15:01:04 UTC ---

(In reply to Clayton Coleman from comment #6)
> The possibility that drain is impossible needs to be part of the design of
> the drain code, so something akin to image pull backoff is of course
> necessary.  Setting severity appropriately.

There should be a delay.  If this (many requests per second) is caused by pod-count-of-one, then it's a bug in that logic.  There's no scenario where you should see hundreds (or even dozens) requests per second for only one pod drain blocked by pdb.

--- Additional comment from Vikas Choudhary on 2019-07-27 18:46:23 UTC ---

In the drain logic, a goroutine is run for evicting each pod running on the node. If the eviction fails with error "too many requests", goroutine keeps creating eviction requests forever with a sleep of 5 sec. After global timeout, Drain() returns but the goroutine for the pods continues to create eviction requests.

machine controller gets error from the Drain() and then after waiting for 20 secs again invokes Drain(). Again goroutine for eviction of each pod gets created. And this way after each Drain() return and a wait of 20 secs, goroutines for making eviction requesting keep getting created.

Created fix PR, https://github.com/openshift/kubernetes-drain/pull/3, which is ensuring that after global timeout, all the goroutines return before the return of the Drain()

This problem is so serious that cluster will die if even a single pod failed to evict.

Comment 1 Vikas Choudhary 2019-07-27 19:44:07 UTC
Steps to reproduce the issue:

1. Create the pdb with minAvailable 1:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      "app": "nginx"


2. Create a replicaset with 1 replica and labels as `"app": "nginx"`

3. Find out the machine for the node on which pod is running.
4. Delete the machine

Machine deletion command will not return.
In another terminal, do a `tail -f` on machine-controller logs. You will see logs like below:
late the pod's disruption budget.
I0727 19:40:08.240725       1 info.go:20] error when evicting pod "pdb-workload-q24lb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0727 19:40:13.250043       1 info.go:20] error when evicting pod "pdb-workload-q24lb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
I0727 19:40:18.265153       1 info.go:20] error when evicting pod "pdb-workload-q24lb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
 
In couple of minutes, you will see that time difference between the two eviction failure logs is decreasing. Failure logs will keep on increasing per second with time.


After applying the fix, you will see that time difference is always constant 5 secs.

Comment 2 Vikas Choudhary 2019-07-27 19:49:51 UTC
WORKAROUND:
Delete the pdb object which is making the eviction fail. 

PREVENTIVE STEP:
PDB configuration and replica count should not be such that it could lead to eviction failure. For example, if minAvailable is 1 in the PDB and replica count is 1, this pod eviction will fail for sure and the issue will occur.

Comment 3 Vikas Choudhary 2019-07-27 19:50:07 UTC
WORKAROUND:
Delete the pdb object which is making the eviction fail. 

PREVENTIVE STEP:
PDB configuration and replica count should not be such that it could lead to eviction failure. For example, if minAvailable is 1 in the PDB and replica count is 1, this pod eviction will fail for sure and the issue will occur.

Comment 4 Alberto 2019-07-29 08:38:38 UTC
Since this is in POST can you please link all the relevant PRs. Targeting this to 4.2 and using https://bugzilla.redhat.com/show_bug.cgi?id=1732929 for tracking backport

Comment 5 Alberto 2019-07-29 08:39:45 UTC
*** Bug 1733922 has been marked as a duplicate of this bug. ***

Comment 8 Jianwei Hou 2019-08-28 06:07:41 UTC
Verified on 4.2.0-0.nightly-2019-08-27-105356. Eviction retry is skipped after global timeout.

```
I0828 06:03:59.442555       1 controller.go:205] Reconciling machine "jhou1-mp2nv-worker-ap-northeast-1a-mcg2t" triggers delete
I0828 06:03:59.510084       1 info.go:16] ignoring DaemonSet-managed pods: tuned-r6cfv, dns-default-4rnkl, node-ca-mj6c2, machine-config-daemon-8nqgn,
 node-exporter-28x7s, multus-cm86v, ovs-mzhv8, sdn-5br78
I0828 06:03:59.524755       1 info.go:20] error when evicting pod "rc-grgbz" (will retry after 5s): Cannot evict pod as it would violate the pod's dis
ruption budget.
I0828 06:04:04.535211       1 info.go:20] error when evicting pod "rc-grgbz" (will retry after 5s): Cannot evict pod as it would violate the pod's dis
ruption budget.
I0828 06:04:09.544775       1 info.go:20] error when evicting pod "rc-grgbz" (will retry after 5s): Cannot evict pod as it would violate the pod's dis
ruption budget.
I0828 06:04:14.554539       1 info.go:20] error when evicting pod "rc-grgbz" (will retry after 5s): Cannot evict pod as it would violate the pod's dis
ruption budget.
I0828 06:04:19.513918       1 info.go:20] Closing stopCh
I0828 06:04:19.626801       1 info.go:16] ignoring DaemonSet-managed pods: tuned-r6cfv, dns-default-4rnkl, node-ca-mj6c2, machine-config-daemon-8nqgn,
 node-exporter-28x7s, multus-cm86v, ovs-mzhv8, sdn-5br78
I0828 06:04:19.626836       1 info.go:20] failed to evict pods from node "ip-10-0-130-176.ap-northeast-1.compute.internal" (pending pods: rc-grgbz): g
lobal timeout!! Skip eviction retries for pod "rc-grgbz"
I0828 06:04:19.626853       1 info.go:16] global timeout!! Skip eviction retries for pod "rc-grgbz"
I0828 06:04:19.626861       1 info.go:20] unable to drain node "ip-10-0-130-176.ap-northeast-1.compute.internal"
I0828 06:04:19.626871       1 info.go:20] there are pending nodes to be drained: ip-10-0-130-176.ap-northeast-1.compute.internal
W0828 06:04:19.626883       1 controller.go:298] drain failed for machine "jhou1-mp2nv-worker-ap-northeast-1a-mcg2t": global timeout!! Skip eviction r
etries for pod "rc-grgbz"
E0828 06:04:19.626898       1 controller.go:214] Failed to drain node for machine "jhou1-mp2nv-worker-ap-northeast-1a-mcg2t": requeue in: 20s
```

Comment 10 errata-xmlrpc 2019-10-16 06:33:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.