Bug 1969412 - etcd consuming high amount of memory and CPU after upgrade to 4.6.17
Summary: etcd consuming high amount of memory and CPU after upgrade to 4.6.17
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.6.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.z
Assignee: Vu Dinh
QA Contact: Bruno Andrade
URL:
Whiteboard:
Depends On: 1951657
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-08 11:52 UTC by OpenShift BugZilla Robot
Modified: 2021-07-06 13:52 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-22 08:29:48 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 2189 0 None open [release-4.6] [release-4.7] Bug 1969412: fix(catalog): Reduce namespace resync in resolution failure 2021-06-08 11:52:37 UTC
Red Hat Product Errata RHBA-2021:2410 0 None None None 2021-06-22 08:30:12 UTC

Comment 2 Bruno Andrade 2021-06-10 19:49:47 UTC
OCP: 4.6.0-0.nightly-2021-06-10-015243
OLM version: 0.16.1
git commit: ac1f9019625c5076a504c6115a725f68b4d6e494

1) install etcd operator
cat sub-etcd.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-operator
  namespace: test-operators
spec:
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace

2) delete csv  to trigger ResolutionFailed event.

3) check catalog-operator log

time="2021-06-10T19:44:54Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:44:54Z" level=info msg="catalog update required at 2021-06-10 19:44:54.455950176 +0000 UTC m=+7070.450620955" CatalogSource=community-operators
time="2021-06-10T19:44:54Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:44:54Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:44:54Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:44:54Z" level=warning msg="status not equal, updating..." id=txWmT ip=install-h9b9f namespace=test-operators phase=Installing
E0610 19:44:55.021865       1 queueinformer_operator.go:290] sync {"update" "test-operators/install-h9b9f"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-h9b9f": the object has been modified; please apply your changes to the latest version and try again
time="2021-06-10T19:44:55Z" level=info msg=syncing id=XniA4 ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:44:55Z" level=warning msg="status not equal, updating..." id=XniA4 ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:44:55Z" level=info msg=syncing id=NdqTB ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:44:55Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:44:55Z" level=warning msg="status not equal, updating..." id=NdqTB ip=install-h9b9f namespace=test-operators phase=Installing
E0610 19:44:56.022208       1 queueinformer_operator.go:290] sync {"update" "test-operators/install-h9b9f"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-h9b9f": the object has been modified; please apply your changes to the latest version and try again
time="2021-06-10T19:44:56Z" level=info msg=syncing id=DpRfY ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:44:56Z" level=info msg=syncing id=001s/ ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:45:01Z" level=info msg=syncing id=PMV4c ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:45:06Z" level=info msg=syncing id=5DhGQ ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:45:09Z" level=info msg="catalog polling result: no update" CatalogSource=community-operators
time="2021-06-10T19:45:09Z" level=info msg="catalog polling result: no update" CatalogSource=community-operators
E0610 19:45:09.263258       1 queueinformer_operator.go:290] sync {"update" "openshift-marketplace/community-operators"} failed: couldn't ensure registry server - error ensuring updated catalog source pod: : error deleting duplicate catalog polling pod: community-operators-wgcbm: error deleting pod: community-operators-wgcbm: pods "community-operators-wgcbm" not found
time="2021-06-10T19:45:11Z" level=info msg=syncing id=HYziS ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:45:16Z" level=info msg=syncing id=zrXh6 ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:45:21Z" level=info msg=syncing id=tGT0j ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdbackups.etcd.database.coreos.com.crd.yaml
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdclusters.etcd.database.coreos.com.crd.yaml
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdoperator.v0.9.4.clusterserviceversion.yaml
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdrestores.etcd.database.coreos.com.crd.yaml
time="2021-06-10T19:45:21Z" level=warning msg="status not equal, updating..." id=tGT0j ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:45:21Z" level=info msg=syncing id=7U1L5 ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdclusters.etcd.database.coreos.com.crd.yaml
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdoperator.v0.9.4.clusterserviceversion.yaml
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdrestores.etcd.database.coreos.com.crd.yaml
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdbackups.etcd.database.coreos.com.crd.yaml
time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:45:21Z" level=info msg=syncing id=sqQkm ip=install-h9b9f namespace=test-operators phase=Installing
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdbackups.etcd.database.coreos.com.crd.yaml
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdclusters.etcd.database.coreos.com.crd.yaml
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdoperator.v0.9.4.clusterserviceversion.yaml
time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdrestores.etcd.database.coreos.com.crd.yaml
time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:45:21Z" level=info msg=syncing id=JCmhV ip=install-h9b9f namespace=test-operators phase=Complete
time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:45:22Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
E0610 19:45:22.064155       1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again
time="2021-06-10T19:45:22Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:45:22Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
E0610 19:45:22.264779       1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again
time="2021-06-10T19:45:22Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:45:23Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
E0610 19:45:23.065496       1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again
time="2021-06-10T19:45:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:45:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd
time="2021-06-10T19:45:26Z" level=info msg=syncing id=5TpCw ip=install-h9b9f namespace=test-operators phase=Complete
time="2021-06-10T19:45:26Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd


There is no such log "creating installplan" after the "ResolutionFailed" event happens. 

LGTM, marking as verified.

Comment 5 errata-xmlrpc 2021-06-22 08:29:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.35 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2410


Note You need to log in before you can comment on or make changes to this bug.