A 4.1.4 cluster identified via telemetry was reporting 500+ 429 requests a second via the API. Upon inspection of the cluster the machine-controller-manager was observed generating 400+ pod eviction requests (that are all errors) a second continuously. The cluster is available for inspection, the top three requests are all abnormal and a result of the very large number of eviction requests: {client="machine-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format",code="429",endpoint="https",job="apiserver",namespace="default",resource="pods",scope="namespace",service="kubernetes",subresource="eviction",verb="CREATE"} 405.99523809523805 {client="machine-controller-manager/v0.0.0 (linux/amd64) kubernetes/$Format",code="429",contentType="application/json",endpoint="https",job="apiserver",namespace="default",resource="pods",scope="namespace",service="kubernetes",subresource="eviction",verb="POST"} 50 {client="openshift-kube-apiserver/v1.13.4+c62ce01 (linux/amd64) kubernetes/c62ce01",code="200",contentType="application/vnd.kubernetes.protobuf",endpoint="https",job="apiserver",namespace="default",resource="poddisruptionbudgets",scope="namespace",service="kubernetes",verb="LIST"} 50 {client="openshift-apiserver/v1.13.4+c62ce01 (linux/amd64) kubernetes/c62ce01",code="429",endpoint="https",job="apiserver",namespace="default",resource="subjectaccessreviews",scope="cluster",service="kubernetes",verb="CREATE"} 12.873109243697478
There are 5 clusters reporting over 300 429-response code errors per second, which means this is likely impacting multiple.
openshift/kubernetes drain uses a goroutine for every pod that needs to be drained on a node. If 429 is returned (can't do eviction, probably PDB), it sleeps 5 seconds before trying again. There's a 20 second timeout on draining that we configure in the machine-controller, and we delay requeue of the machine-object delete request after a drain fails. My guess is that a large scale-down (many machines) took place, and there's not a place to schedule all the pods with PDBs. In 4.1, machine controller only drains when deleting, and drains (evictions) only get 429 from PDBs.
Behavior is as expected. There is a pdb object in the namespace which says "minAvailable: 1". There is a replicaset which says replica 1: ``` $ kubectl get replicaset -n istio-system NAME DESIRED CURRENT READY AGE istio-pilot-77d5cd65b4 1 1 1 9d ``` PDB wont let the pod evict. Now a machine is tried to be deleted. Corresponding node has such a pod. Machine controller calls drain api. drain api marks the node unschedulable and then it tries to evict pod. This pod eviction fails forever and it keeps retrying to evict. Node gets stuck in unschedulable state and machine never gets deleted.
I verified above in one of the clusters where this is happening. Also this is very easy to reproduce and verify.
(In reply to Vikas Choudhary from comment #4) > I verified above in one of the clusters where this is happening. Also this > is very easy to reproduce and verify. How many machines are being deleted? It shouldn't be spamming 300+ per second for a single pod. There should be 5 seconds between each 429 response, timeout after 20 seconds, and then it should wait a bit longer before attempting to drain the node again. If we're seeing 300+ requests/sec for one pod, something is broken.
The possibility that drain is impossible needs to be part of the design of the drain code, so something akin to image pull backoff is of course necessary. Setting severity appropriately.
(In reply to Michael Gugino from comment #5) > (In reply to Vikas Choudhary from comment #4) > > I verified above in one of the clusters where this is happening. Also this > > is very easy to reproduce and verify. > > How many machines are being deleted? It shouldn't be spamming 300+ per > second for a single pod. There should be 5 seconds between each 429 > response, timeout after 20 seconds, and then it should wait a bit longer > before attempting to drain the node again. If we're seeing 300+ > requests/sec for one pod, something is broken. retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:44.959919 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:44.977066 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:44.977109 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:44.981985 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:44.982317 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:44.982477 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:44.982994 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:44.983569 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:44.990721 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:44.991017 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:44.991579 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.002211 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.003483 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.011189 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.019750 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.020467 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.075287 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.084313 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.084433 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.085064 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.107962 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.109397 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.109540 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.109574 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.176538 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.191653 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.236642 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.236979 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.300564 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.302414 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.303310 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.303466 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.315951 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.338997 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.356249 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.474960 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.505237 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.505467 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.505537 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.506447 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.506570 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.506635 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.506721 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.506868 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.507118 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.537281 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:38:45.537305 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.613639 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:38:45.664553 1 info.go:2^C [vikas@fed-node cluster-machine-approver](metrics)$ tail -f ~/mao.txt I0724 18:39:35.259138 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.259444 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.259597 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.259677 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.259757 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.259958 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.260495 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.260832 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.269729 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.274442 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.282955 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.283030 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.283106 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.288507 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.307440 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.312153 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.313205 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.423372 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.424057 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.424189 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.424292 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.424512 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.424611 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.434624 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.441138 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.447383 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.482034 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.550617 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.551986 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.551993 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.552062 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.553700 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.553845 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.554660 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.574702 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.577527 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.579395 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.579522 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.579940 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.579993 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.580040 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.580090 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.580371 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.594049 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.595020 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.639244 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.672308 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.682214 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.687674 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.687979 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.757075 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.765587 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.765796 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.765912 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.765956 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.766935 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.781010 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.786203 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.792539 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.792700 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.792780 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.810536 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.810724 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.810954 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.811027 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.811188 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.811355 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.842538 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.927203 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.931289 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.932451 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.932628 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.932864 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.933253 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.933548 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.933642 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.934974 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:35.936654 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.981989 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:35.982534 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.037300 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.037402 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.037650 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.065920 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.163720 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.184569 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.194635 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.200734 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.200942 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.215508 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.216521 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.304595 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.305124 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.306302 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.310066 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.360106 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.360238 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.361121 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.361216 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.367227 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.412259 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.447784 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.455159 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.458172 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.458302 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.478403 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.478580 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.479503 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.479750 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.480155 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.541482 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.541636 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.549706 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.551879 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.552062 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.580026 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.670595 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.670713 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.670813 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.670875 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.671028 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.679959 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.685252 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.685514 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.685760 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.685886 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.689558 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.696567 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.700508 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.700669 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.701578 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.705307 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.705624 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.729992 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.749724 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.761134 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.771029 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.771041 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.771140 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.786566 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.786708 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.786811 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.787849 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.824770 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.827841 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.840346 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.840779 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.851168 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:36.862859 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.872753 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.876348 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.877248 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:36.881959 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.028822 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.178834 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.178989 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.179924 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.180045 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.180204 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.180321 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.181119 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.181317 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.181413 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.181518 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.311999 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.312103 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.337951 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.349415 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.349557 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.349844 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.355214 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.380357 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.393520 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.459721 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.460476 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.461264 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.461370 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.466953 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.474567 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.482062 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.484636 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.501154 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.514110 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.546175 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.546881 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.547122 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.547242 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.549993 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.550086 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.621112 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.621242 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.621316 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.621421 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.621493 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.621726 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.703456 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.717831 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.722451 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.787320 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.790826 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.793607 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.794001 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.795007 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.834800 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.844770 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.845808 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:37.907411 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:37.996493 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.044766 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.062074 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.075158 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:38.304723 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.323828 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.353856 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.424610 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:38.425100 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.427284 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:38.445566 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:38.445726 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:38.446120 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.446450 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.446743 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.446759 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:38.447321 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:38.447406 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:38.456249 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.456268 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.490355 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0724 18:39:38.490469 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): the server has received too many requests and has asked us to try again later (post pods.policy istio-pilot-77d5cd65b4-2s9rz) I0724 18:39:38.490587 1 info.go:20] error when evicting pod "istio-pilot-77d5cd65b4-2s9rz" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
(In reply to Clayton Coleman from comment #6) > The possibility that drain is impossible needs to be part of the design of > the drain code, so something akin to image pull backoff is of course > necessary. Setting severity appropriately. There should be a delay. If this (many requests per second) is caused by pod-count-of-one, then it's a bug in that logic. There's no scenario where you should see hundreds (or even dozens) requests per second for only one pod drain blocked by pdb.
In the drain logic, a goroutine is run for evicting each pod running on the node. If the eviction fails with error "too many requests", goroutine keeps creating eviction requests forever with a sleep of 5 sec. After global timeout, Drain() returns but the goroutine for the pods continues to create eviction requests. machine controller gets error from the Drain() and then after waiting for 20 secs again invokes Drain(). Again goroutine for eviction of each pod gets created. And this way after each Drain() return and a wait of 20 secs, goroutines for making eviction requesting keep getting created. Created fix PR, https://github.com/openshift/kubernetes-drain/pull/3, which is ensuring that after global timeout, all the goroutines return before the return of the Drain() This problem is so serious that cluster will die if even a single pod failed to evict.
Steps to reproduce the issue: 1. Create the pdb with minAvailable 1: apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: nginx-pdb spec: minAvailable: 1 selector: matchLabels: "app": "nginx" 2. Create a replicaset with 1 replica and labels as `"app": "nginx"` 3. Find out the machine for the node on which pod is running. 4. Delete the machine Machine deletion command will not return. In another terminal, do a `tail -f` on machine-controller logs. You will see logs like below: late the pod's disruption budget. I0727 19:40:08.240725 1 info.go:20] error when evicting pod "pdb-workload-q24lb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0727 19:40:13.250043 1 info.go:20] error when evicting pod "pdb-workload-q24lb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0727 19:40:18.265153 1 info.go:20] error when evicting pod "pdb-workload-q24lb" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. In couple of minutes, you will see that time difference between the two eviction failure logs is decreasing. Failure logs will keep on increasing per second with time. After applying the fix, you will see that time difference is always constant 5 secs.
WORKAROUND: Delete the pdb object which is making the eviction fail. PREVENTIVE STEP: PDB configuration and replica count should not be such that it could lead to eviction failure. For example, if minAvailable is 1 in the PDB and replica count is 1, this pod eviction will fail for sure and the issue will occur.
Since this is in POST can you please link all the relevant PRs. Targeting this to 4.1.z and using https://bugzilla.redhat.com/show_bug.cgi?id=1732929 for tracking 4.2
https://github.com/openshift/cluster-api/pull/54 https://github.com/openshift/cluster-api-provider-aws/pull/243
Verified $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.1.0-0.nightly-2019-09-21-195243 True False 37m Cluster version is 4.1.0-0.nightly-2019-09-21-195243 Eviction retry is skipped after global timeout. I0923 04:53:25.927140 1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" I0923 04:53:25.927265 1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster I0923 04:53:25.927333 1 controller.go:193] Reconciling machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" triggers delete I0923 04:53:25.976363 1 info.go:20] cordoned node "ip-10-0-147-193.us-east-2.compute.internal" I0923 04:53:26.031409 1 info.go:16] ignoring DaemonSet-managed pods: tuned-4fznn, dns-default-8dg7d, node-ca-86xxw, machine-config-daemon-sswxr, node-exporter-d6v8f, multus-gnbbb, ovs-zrd97, sdn-9qm7f; deleting pods with local storage: alertmanager-main-1, kube-state-metrics-85bf9ffbbf-xkxxp, prometheus-adapter-76659fb787-p5dj2, prometheus-k8s-1 I0923 04:53:26.058436 1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0923 04:53:26.337250 1 info.go:20] pod "alertmanager-main-1" removed (evicted) I0923 04:53:31.065423 1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0923 04:53:32.333409 1 info.go:20] pod "router-default-555fb8b466-8z7df" removed (evicted) I0923 04:53:32.733204 1 info.go:20] pod "telemeter-client-7f5664b4bb-44xhw" removed (evicted) I0923 04:53:33.933347 1 info.go:20] pod "community-operators-644fbcf4bb-9wnz5" removed (evicted) I0923 04:53:34.933416 1 info.go:20] pod "prometheus-adapter-76659fb787-p5dj2" removed (evicted) I0923 04:53:35.133178 1 info.go:20] pod "kube-state-metrics-85bf9ffbbf-xkxxp" removed (evicted) I0923 04:53:36.071147 1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0923 04:53:39.107004 1 info.go:20] pod "prometheus-operator-67f988789d-mpfn4" removed (evicted) I0923 04:53:39.129496 1 info.go:20] pod "image-registry-7c595cdbbf-fdck4" removed (evicted) I0923 04:53:39.140807 1 info.go:20] pod "prometheus-k8s-1" removed (evicted) I0923 04:53:41.077216 1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0923 04:53:46.034981 1 info.go:20] Closing stopCh I0923 04:53:46.122128 1 info.go:16] ignoring DaemonSet-managed pods: tuned-4fznn, dns-default-8dg7d, node-ca-86xxw, machine-config-daemon-sswxr, node-exporter-d6v8f, multus-gnbbb, ovs-zrd97, sdn-9qm7f I0923 04:53:46.122160 1 info.go:20] failed to evict pods from node "ip-10-0-147-193.us-east-2.compute.internal" (pending pods: nginx-7499b4d6d9-2kjzf): global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf" I0923 04:53:46.122177 1 info.go:16] global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf" I0923 04:53:46.122184 1 info.go:20] unable to drain node "ip-10-0-147-193.us-east-2.compute.internal" I0923 04:53:46.122191 1 info.go:20] there are pending nodes to be drained: ip-10-0-147-193.us-east-2.compute.internal W0923 04:53:46.122199 1 controller.go:286] drain failed for machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7": global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf" E0923 04:53:46.122209 1 controller.go:202] Failed to drain node for machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7": requeue in: 20s I0923 04:53:46.122221 1 controller.go:352] Actuator returned requeue-after error: requeue in: 20s I0923 04:53:46.122291 1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" I0923 04:53:46.122314 1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster I0923 04:53:46.129353 1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" I0923 04:53:46.129375 1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster I0923 04:53:46.129387 1 actuator.go:378] Checking if machine exists I0923 04:53:46.174015 1 actuator.go:386] Instance does not exist I0923 04:53:46.174041 1 controller.go:247] Reconciling machine object zhsun41171-rhch4-worker-us-east-2b-rvnq7 triggers idempotent create. I0923 04:53:46.174054 1 actuator.go:110] creating machine E0923 04:53:46.174321 1 utils.go:186] NodeRef not found in machine zhsun41171-rhch4-worker-us-east-2b-rvnq7 I0923 04:53:46.185083 1 instances.go:44] No stopped instances found for machine zhsun41171-rhch4-worker-us-east-2b-rvnq7 I0923 04:53:46.185130 1 instances.go:141] Using AMI ami-0649fd5d42859bdfc I0923 04:53:46.185141 1 instances.go:73] Describing security groups based on filters I0923 04:53:46.328151 1 instances.go:118] Describing subnets based on filters I0923 04:53:47.466094 1 actuator.go:475] Updating status I0923 04:53:47.466122 1 actuator.go:519] finished calculating AWS status I0923 04:53:47.466224 1 actuator.go:144] machine status has changed, updating I0923 04:53:47.474756 1 actuator.go:537] Instance state still pending, returning an error to requeue W0923 04:53:47.474788 1 controller.go:249] Failed to create machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7": requeue in: 20s I0923 04:53:47.474810 1 controller.go:352] Actuator returned requeue-after error: requeue in: 20s I0923 04:53:47.474863 1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" I0923 04:53:47.474878 1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster I0923 04:53:47.474893 1 actuator.go:378] Checking if machine exists I0923 04:53:47.625378 1 actuator.go:391] Instance exists as "i-032f759ca23f14142" I0923 04:53:47.625403 1 controller.go:238] Reconciling machine "zhsun41171-rhch4-worker-us-east-2b-rvnq7" triggers idempotent update I0923 04:53:47.625413 1 actuator.go:305] updating machine I0923 04:53:47.625501 1 actuator.go:313] obtaining EC2 client for region I0923 04:53:47.671802 1 actuator.go:331] found 1 existing instances for machine I0923 04:53:47.671850 1 actuator.go:475] Updating status I0923 04:53:47.671915 1 actuator.go:519] finished calculating AWS status I0923 04:53:47.672085 1 actuator.go:153] status unchanged I0923 04:53:47.672101 1 actuator.go:537] Instance state still pending, returning an error to requeue E0923 04:53:47.672114 1 controller.go:240] Error updating machine "openshift-machine-api/zhsun41171-rhch4-worker-us-east-2b-rvnq7": requeue in: 20s I0923 04:53:47.672130 1 controller.go:352] Actuator returned requeue-after error: requeue in: 20s I0923 04:54:06.122503 1 controller.go:129] Reconciling Machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" I0923 04:54:06.122532 1 controller.go:298] Machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" in namespace "openshift-machine-api" doesn't specify "cluster.k8s.io/cluster-name" label, assuming nil cluster I0923 04:54:06.122544 1 controller.go:193] Reconciling machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7" triggers delete I0923 04:54:06.165829 1 info.go:16] ignoring DaemonSet-managed pods: tuned-4fznn, dns-default-8dg7d, node-ca-86xxw, machine-config-daemon-sswxr, node-exporter-d6v8f, multus-gnbbb, ovs-zrd97, sdn-9qm7f I0923 04:54:06.174594 1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0923 04:54:11.180953 1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0923 04:54:16.187935 1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0923 04:54:21.194214 1 info.go:20] error when evicting pod "nginx-7499b4d6d9-2kjzf" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. I0923 04:54:26.169560 1 info.go:20] Closing stopCh I0923 04:54:26.237956 1 info.go:16] ignoring DaemonSet-managed pods: tuned-4fznn, dns-default-8dg7d, node-ca-86xxw, machine-config-daemon-sswxr, node-exporter-d6v8f, multus-gnbbb, ovs-zrd97, sdn-9qm7f I0923 04:54:26.237980 1 info.go:20] failed to evict pods from node "ip-10-0-147-193.us-east-2.compute.internal" (pending pods: nginx-7499b4d6d9-2kjzf): global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf" I0923 04:54:26.237991 1 info.go:16] global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf" I0923 04:54:26.237997 1 info.go:20] unable to drain node "ip-10-0-147-193.us-east-2.compute.internal" I0923 04:54:26.238004 1 info.go:20] there are pending nodes to be drained: ip-10-0-147-193.us-east-2.compute.internal W0923 04:54:26.238018 1 controller.go:286] drain failed for machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7": global timeout!! Skip eviction retries for pod "nginx-7499b4d6d9-2kjzf" E0923 04:54:26.238034 1 controller.go:202] Failed to drain node for machine "zhsun41171-rhch4-worker-us-east-2b-4z7c7": requeue in: 20s I0923 04:54:26.238048 1 controller.go:352] Actuator returned requeue-after error: requeue in: 20s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2856