Bug 1866828 - Operator catalog Pods created by CatalogSource aren't evicted to other nodes even after their node is down
Summary: Operator catalog Pods created by CatalogSource aren't evicted to other nodes ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.z
Assignee: Ben Luddy
QA Contact: Bruno Andrade
URL:
Whiteboard:
Depends On: 1866829
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-06 13:53 UTC by Ben Luddy
Modified: 2020-09-08 10:55 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1862340
Environment:
Last Closed: 2020-09-08 10:54:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1712 None closed [release-4.5] Bug 1866828: Remove unneeded toleration from catalog source pod 2020-08-25 21:08:04 UTC
Red Hat Product Errata RHBA-2020:3510 None None None 2020-09-08 10:55:29 UTC

Description Ben Luddy 2020-08-06 13:53:53 UTC
This bug was initially created as a copy of Bug #1862340, which is private:

Operator catalog Pods created by CatalogSource aren't evicted to other nodes,
even after the node that the pods are running on get down.

Comment 4 Bruno Andrade 2020-08-21 14:36:30 UTC
Marking as VERIFIED, catalog source operator pod was evicted as expected.

Cluster Version: 4.6.0-0.nightly-2020-08-18-165040
OLM version: 0.16.0
git commit: 1fdd347ab723bf6aec30c79dfb217bcbf21a13e9

 oc get pods -n openshift-operator-lifecycle-manager -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP             NODE                                         NOMINATED NODE   READINESS GATES
catalog-operator-6944b55486-q8gkq   1/1     Running   0          46h   10.130.0.29    ip-10-0-192-165.us-east-2.compute.internal   <none>           <none>
olm-operator-587669c4dc-jmqfl       1/1     Running   0          46h   10.130.0.27    ip-10-0-192-165.us-east-2.compute.internal   <none>           <none>
packageserver-59c9f4dd68-22wds      1/1     Running   0          11h   10.129.3.187   ip-10-0-155-133.us-east-2.compute.internal   <none>           <none>
packageserver-59c9f4dd68-ml5n5      1/1     Running   0          11h   10.130.0.55    ip-10-0-192-165.us-east-2.compute.internal   <none>           <none>

oc adm cordon ip-10-0-192-165.us-east-2.compute.internal   
node/ip-10-0-192-165.us-east-2.compute.internal cordoned

oc adm drain ip-10-0-192-165.us-east-2.compute.internal --delete-local-data --ignore-daemonsets                                                          130 ↵
node/ip-10-0-192-165.us-east-2.compute.internal already cordoned
WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fpwf8, openshift-cluster-node-tuning-operator/tuned-lhsl5, openshift-controller-manager/controller-manager-2pmvr, openshift-dns/dns-default-t7nlh, openshift-image-registry/node-ca-bdx72, openshift-machine-config-operator/machine-config-daemon-r75bn, openshift-machine-config-operator/machine-config-server-4t22s, openshift-monitoring/node-exporter-nrfgq, openshift-multus/multus-admission-controller-4sjbf, openshift-multus/multus-mnlfk, openshift-multus/network-metrics-daemon-l968d, openshift-sdn/ovs-xhtf6, openshift-sdn/sdn-controller-p5t9m, openshift-sdn/sdn-v88bk
evicting pod openshift-cloud-credential-operator/cloud-credential-operator-78bbb5fddf-m9pn7
evicting pod openshift-cloud-credential-operator/pod-identity-webhook-596ff668d-qqdrc
evicting pod openshift-apiserver/apiserver-6846795c64-75xdc
evicting pod openshift-kube-apiserver/revision-pruner-14-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-cluster-samples-operator/cluster-samples-operator-7d4f777bd7-vx49z
evicting pod openshift-console-operator/console-operator-b77b78d55-scrss
evicting pod openshift-kube-apiserver/installer-12-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-authentication/oauth-openshift-68896fb778-tzhv5
evicting pod openshift-kube-apiserver-operator/kube-apiserver-operator-bd8cb9589-5lvsk
evicting pod openshift-machine-config-operator/machine-config-controller-67f577b45f-kghrc
evicting pod openshift-etcd/revision-pruner-3-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-cluster-node-tuning-operator/cluster-node-tuning-operator-5b9d49dcf8-dwnrr
evicting pod openshift-kube-apiserver/installer-13-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-controller-manager-operator/openshift-controller-manager-operator-9c66885fc-bf9zx
evicting pod openshift-kube-scheduler/revision-pruner-7-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-kube-apiserver/revision-pruner-15-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-etcd-operator/etcd-operator-7c9854d6d4-w8gwd
evicting pod openshift-image-registry/cluster-image-registry-operator-89755bc47-mnzx6
evicting pod openshift-kube-apiserver/installer-14-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-kube-apiserver/revision-pruner-16-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-kube-apiserver/installer-15-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-network-operator/network-operator-578ffc867f-vbv9n
evicting pod openshift-etcd/etcd-quorum-guard-665574fdb6-7rk7n
evicting pod openshift-marketplace/marketplace-operator-746f875dbd-hjrqj
evicting pod openshift-insights/insights-operator-6ccf4845b-r2sr7
evicting pod openshift-machine-api/cluster-autoscaler-operator-7f98c76ff9-4blvh
evicting pod openshift-machine-api/machine-api-operator-5b4685cb69-kqr7g
evicting pod openshift-console/console-7668cbf5d8-vgct9
evicting pod openshift-kube-controller-manager/revision-pruner-9-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-monitoring/cluster-monitoring-operator-848448b8f4-nt8l5
evicting pod openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-5889fcfbdd-lxr89
evicting pod openshift-monitoring/prometheus-operator-b99dbf4bb-55nbn
evicting pod openshift-kube-apiserver/revision-pruner-12-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-kube-apiserver/revision-pruner-13-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-kube-apiserver/installer-16-ip-10-0-192-165.us-east-2.compute.internal
evicting pod openshift-oauth-apiserver/apiserver-558756467-bs5tp
evicting pod openshift-operator-lifecycle-manager/packageserver-59c9f4dd68-ml5n5
evicting pod openshift-operator-lifecycle-manager/olm-operator-587669c4dc-jmqfl
evicting pod openshift-operator-lifecycle-manager/catalog-operator-6944b55486-q8gkq
evicting pod openshift-ingress-operator/ingress-operator-77d9dc9d84-97qd8
pod/revision-pruner-12-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/revision-pruner-14-ip-10-0-192-165.us-east-2.compute.internal evicted
I0821 11:27:53.341277  154405 request.go:645] Throttling request took 1.000855838s, request: POST:https://api.bandrade2231321.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-kube-scheduler/pods/revision-pruner-7-ip-10-0-192-165.us-east-2.compute.internal/eviction
pod/revision-pruner-3-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/installer-13-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/revision-pruner-7-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/installer-12-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/installer-14-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/revision-pruner-15-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/revision-pruner-16-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/revision-pruner-9-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/revision-pruner-13-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/installer-15-ip-10-0-192-165.us-east-2.compute.internal evicted
pod/installer-16-ip-10-0-192-165.us-east-2.compute.internal evicted
I0821 11:28:03.526183  154405 request.go:645] Throttling request took 5.380261238s, request: GET:https://api.bandrade2231321.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-5889fcfbdd-lxr89
pod/cloud-credential-operator-78bbb5fddf-m9pn7 evicted
pod/cluster-node-tuning-operator-5b9d49dcf8-dwnrr evicted
pod/apiserver-6846795c64-75xdc evicted
pod/marketplace-operator-746f875dbd-hjrqj evicted
pod/cluster-image-registry-operator-89755bc47-mnzx6 evicted
pod/apiserver-558756467-bs5tp evicted
pod/machine-api-operator-5b4685cb69-kqr7g evicted
pod/insights-operator-6ccf4845b-r2sr7 evicted
pod/openshift-controller-manager-operator-9c66885fc-bf9zx evicted
pod/machine-config-controller-67f577b45f-kghrc evicted
pod/packageserver-59c9f4dd68-ml5n5 evicted
pod/cluster-samples-operator-7d4f777bd7-vx49z evicted
pod/etcd-quorum-guard-665574fdb6-7rk7n evicted
pod/kube-apiserver-operator-bd8cb9589-5lvsk evicted
pod/cluster-monitoring-operator-848448b8f4-nt8l5 evicted
pod/etcd-operator-7c9854d6d4-w8gwd evicted
pod/kube-storage-version-migrator-operator-5889fcfbdd-lxr89 evicted
pod/ingress-operator-77d9dc9d84-97qd8 evicted
pod/cluster-autoscaler-operator-7f98c76ff9-4blvh evicted
pod/network-operator-578ffc867f-vbv9n evicted
pod/prometheus-operator-b99dbf4bb-55nbn evicted
pod/catalog-operator-6944b55486-q8gkq evicted
pod/olm-operator-587669c4dc-jmqfl evicted
pod/console-operator-b77b78d55-scrss evicted
pod/oauth-openshift-68896fb778-tzhv5 evicted
pod/pod-identity-webhook-596ff668d-qqdrc evicted
pod/console-7668cbf5d8-vgct9 evicted
node/ip-10-0-192-165.us-east-2.compute.internal evicted

oc get pods -n openshift-operator-lifecycle-manager -o wide                                    
NAME                                READY   STATUS    RESTARTS   AGE   IP             NODE                                         NOMINATED NODE   READINESS GATES
catalog-operator-6944b55486-jt7kj   1/1     Running   0          54s   10.129.0.44    ip-10-0-156-115.us-east-2.compute.internal   <none>           <none>
olm-operator-587669c4dc-4f6xk       1/1     Running   0          57s   10.129.0.43    ip-10-0-156-115.us-east-2.compute.internal   <none>           <none>
packageserver-59c9f4dd68-22wds      1/1     Running   0          11h   10.129.3.187   ip-10-0-155-133.us-east-2.compute.internal   <none>           <none>
packageserver-59c9f4dd68-kcn9v      1/1     Running   0          57s   10.128.0.54    ip-10-0-163-251.us-east-2.compute.internal   <none>           <none>

Comment 6 errata-xmlrpc 2020-09-08 10:54:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.8 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3510


Note You need to log in before you can comment on or make changes to this bug.