Bug 1951657

Summary: etcd consuming high amount of memory and CPU after upgrade to 4.6.17
Product: OpenShift Container Platform Reporter: Vu Dinh <vdinh>
Component: OLMAssignee: Vu Dinh <vdinh>
OLM sub component: OLM QA Contact: Bruno Andrade <bandrade>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: bandrade, tflannag
Version: 4.6.z   
Target Milestone: ---   
Target Release: 4.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-19 15:16:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1934071    
Bug Blocks: 1969412    

Description Vu Dinh 2021-04-20 16:41:54 UTC 
This bug was initially created as a copy of Bug #1934071

I am copying this bug because: 



Description of problem:

Customer upgraded cluster from 4.5 to 4.6.17

After the upgrade they started to see the following behavior on the cluster: 

- High cpu/memory usage in master nodes. 
- etcd is unstable. It gets disconnected from endpoints and it has frequent leader elections.
- master nodes are presenting very high cpu and memory usage. kube-apiserver and etcd are using all the resources.
- api is not available unless we restart the platform, after restart api is available just for a minute or two.
- When we lose access to api we are not able to log to master nodes either because cpu/memory is too high. 
- After some time we are able to login again to nodes, but again we lose connection with them because cpu/memory usage.
- etcd was defrag moving from 3GB to 957 MB but after 10m the database size is 2GB again...

- All master nodes are affected with some issue that cause etcd and api-server consume 100% memory and 100% cpu. 
- Hosts stay inaccessible even trough ssh. Restarting the nodes makes api-server return but it last just minutes. 
- Cannot send commands to nodes, crioctl and event top are not able to respond

In the muust-gather there are a lots of errors from OLM:

Starting: 06:34 - Ending 07:07
=============================
2021-02-26T06:43:03.556442096-03:00 {"level":"error","ts":1614332583.5563064,"logger":"controllers.adoption","msg":"component not found","error":"InstallPlan.operators.coreos.com \"install-gssr9\" not found","stacktrace":"github.com/go-
logr/zapr.(*zapLogger).Error\n\t/build/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators.(*AdoptionReconciler).adopt\n\t/build/pkg/controller/operators/adoption
_controller.go:269\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators.(*AdoptionReconciler).adoptInstallPlan\n\t/build/pkg/controller/operators/adoption_controller.go:383\ngithub.com/operator-framework/op
erator-lifecycle-manager/pkg/controller/operators.(*AdoptionReconciler).ReconcileSubscription\n\t/build/pkg/controller/operators/adoption_controller.go:169\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/build/vendor/si
gs.k8s.io/controller-runtime/pkg/reconcile/reconcile.go:93\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\
nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.
(*Controller).worker\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/
apimachinery/pkg/util/wait.BackoffUntil\n\t/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pk
g/util/wait.Until\n\t/build/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
============================
wc -l = 12519


* Before the upgrade the cluster was running fine, no issues. 


Version-Release number of selected component (if applicable):

OCP 4.6.17

How reproducible:

Customer specific at this moment, I wasn't able to reproduce the issue.

Additional info:
Comment 2 Bruno Andrade 2021-04-30 17:38:00 UTC 
Cluster Version: 4.7.0-0.nightly-2021-04-30-074750
OLM version: 0.17.0
git commit: 48aea63e3aa0d42c6d69215b23f6132188e2220f

1) install etcd operator

2) delete csv to trigger ResolutionFailed event.

3) check catalog-operator log

selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
E0430 17:16:11.877135       1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again
time="2021-04-30T17:16:11Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:11Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:11Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
E0430 17:16:11.889078       1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again
time="2021-04-30T17:16:11Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:11Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:12Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:12Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:12Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:12Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:12Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:12Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:12Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:12Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:12Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:13Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{redhat-marketplace openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="failed to construct operator from bundle, continuing: couldn't parse bundle path, missing provided and required apis" catalog="{community-operators openshift-marketplace}"
time="2021-04-30T17:16:14Z" level=warning msg="no installplan found with matching generation, creating new one" id=VH0dz namespace=test-operators
time="2021-04-30T17:16:14Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:14Z" level=info msg=syncing id=vnvkS ip=install-m4q88 namespace=test-operators phase=
time="2021-04-30T17:16:14Z" level=info msg="skip processing installplan without status - subscription sync responsible for initial status" id=vnvkS ip=install-m4q88 namespace=test-operators phase=
time="2021-04-30T17:16:14Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:14Z" level=info msg=syncing id=tLEhs ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:14Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:15Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:15Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
E0430 17:16:15.044118       1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again
time="2021-04-30T17:16:15Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:15Z" level=warning msg="status not equal, updating..." id=tLEhs ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:15Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:15Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
E0430 17:16:15.359552       1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again
time="2021-04-30T17:16:15Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
E0430 17:16:15.559014       1 queueinformer_operator.go:290] sync {"update" "test-operators/install-m4q88"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-m4q88": the object has been modified; please apply your changes to the latest version and try again
time="2021-04-30T17:16:15Z" level=info msg=syncing id=bznvQ ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:15Z" level=warning msg="status not equal, updating..." id=bznvQ ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:15Z" level=info msg=syncing id=yw5M+ ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:15Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:15Z" level=warning msg="status not equal, updating..." id=yw5M+ ip=install-m4q88 namespace=test-operators phase=Installing
E0430 17:16:16.361063       1 queueinformer_operator.go:290] sync {"update" "test-operators/install-m4q88"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-m4q88": the object has been modified; please apply your changes to the latest version and try again
time="2021-04-30T17:16:16Z" level=info msg=syncing id=7e7vp ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:16Z" level=info msg=syncing id=0bzMB ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:21Z" level=info msg=syncing id=gIi9f ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:26Z" level=info msg=syncing id=aq8L2 ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:31Z" level=info msg=syncing id=EcQSw ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:36Z" level=info msg=syncing id=N73eA ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdrestores.etcd.database.coreos.com.crd.yaml
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdbackups.etcd.database.coreos.com.crd.yaml
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdclusters.etcd.database.coreos.com.crd.yaml
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdoperator.v0.9.4.clusterserviceversion.yaml
time="2021-04-30T17:16:36Z" level=warning msg="status not equal, updating..." id=N73eA ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:36Z" level=info msg=syncing id=j10fs ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:36Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdrestores.etcd.database.coreos.com.crd.yaml
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdbackups.etcd.database.coreos.com.crd.yaml
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdclusters.etcd.database.coreos.com.crd.yaml
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdoperator.v0.9.4.clusterserviceversion.yaml
W0430 17:16:36.435400       1 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0430 17:16:36.454040       1 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0430 17:16:36.466276       1 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-04-30T17:16:36Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:36Z" level=info msg=syncing id=xDruL ip=install-m4q88 namespace=test-operators phase=Installing
time="2021-04-30T17:16:36Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdrestores.etcd.database.coreos.com.crd.yaml
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdbackups.etcd.database.coreos.com.crd.yaml
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdclusters.etcd.database.coreos.com.crd.yaml
time="2021-04-30T17:16:36Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdoperator.v0.9.4.clusterserviceversion.yaml
W0430 17:16:36.805906       1 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0430 17:16:36.810279       1 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0430 17:16:36.814804       1 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
time="2021-04-30T17:16:37Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:37Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:37Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:37Z" level=info msg=syncing id=GuVz8 ip=install-m4q88 namespace=test-operators phase=Complete
time="2021-04-30T17:16:37Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:37Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
E0430 17:16:37.799404       1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again
time="2021-04-30T17:16:37Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:38Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:16:41Z" level=info msg=syncing id=K5rfB ip=install-m4q88 namespace=test-operators phase=Complete
time="2021-04-30T17:16:41Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:18:37Z" level=info msg="catalog update required at 2021-04-30 17:18:37.47048493 +0000 UTC m=+13042.199636896" CatalogSource=community-operators
time="2021-04-30T17:18:50Z" level=info msg="catalog polling result: no update" CatalogSource=community-operators
time="2021-04-30T17:20:33Z" level=info msg="Adding related objects for operator-lifecycle-manager-catalog"
time="2021-04-30T17:24:01Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:24:01Z" level=info msg=syncing id=RRx6Q ip=install-m4q88 namespace=test-operators phase=Complete
time="2021-04-30T17:24:01Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:24:01Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:24:01Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-04-30T17:24:02Z" level=info msg="catalog update required at 2021-04-30 17:24:02.080403028 +0000 UTC m=+13366.809554997" CatalogSource=redhat-operators
time="2021-04-30T17:24:02Z" level=info msg="catalog update required at 2021-04-30 17:24:02.081847959 +0000 UTC m=+13366.810999925" CatalogSource=redhat-marketplace
time="2021-04-30T17:24:13Z" level=info msg="catalog polling result: no update" CatalogSource=redhat-marketplace
time="2021-04-30T17:24:16Z" level=info msg="catalog polling result: no update" CatalogSource=redhat-operators
time="2021-04-30T17:24:23Z" level=info msg="catalog update required at 2021-04-30 17:24:23.88742903 +0000 UTC m=+13388.616580995" CatalogSource=certified-operators
time="2021-04-30T17:24:42Z" level=info msg="catalog polling result: no update" CatalogSource=certified-operators
time="2021-04-30T17:25:38Z" level=info msg="Adding related objects for operator-lifecycle-manager-catalog"

There is no such log "creating installplan" after the "ResolutionFailed" event happens. 

LGTM, verified
Comment 4 Siddharth Sharma 2021-05-10 17:59:18 UTC 
This bug will be shipped as part of next z-stream release 4.7.11 on May 19th, as 4.7.10 was dropped due to a blocker https://bugzilla.redhat.com/show_bug.cgi?id=1958518.
Comment 8 errata-xmlrpc 2021-05-19 15:16:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.11 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1550