Bug 1969412
| Summary: | etcd consuming high amount of memory and CPU after upgrade to 4.6.17 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> |
| Component: | OLM | Assignee: | Vu Dinh <vdinh> |
| OLM sub component: | OLM | QA Contact: | Bruno Andrade <bandrade> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | akesarka, bandrade, jiazha, tflannag, vdinh |
| Version: | 4.6.z | ||
| Target Milestone: | --- | ||
| Target Release: | 4.6.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-06-22 08:29:48 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1951657 | ||
| Bug Blocks: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.35 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2410 |
OCP: 4.6.0-0.nightly-2021-06-10-015243 OLM version: 0.16.1 git commit: ac1f9019625c5076a504c6115a725f68b4d6e494 1) install etcd operator cat sub-etcd.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: etcd-operator namespace: test-operators spec: channel: singlenamespace-alpha installPlanApproval: Automatic name: etcd source: community-operators sourceNamespace: openshift-marketplace 2) delete csv to trigger ResolutionFailed event. 3) check catalog-operator log time="2021-06-10T19:44:54Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:44:54Z" level=info msg="catalog update required at 2021-06-10 19:44:54.455950176 +0000 UTC m=+7070.450620955" CatalogSource=community-operators time="2021-06-10T19:44:54Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:44:54Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:44:54Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:44:54Z" level=warning msg="status not equal, updating..." id=txWmT ip=install-h9b9f namespace=test-operators phase=Installing E0610 19:44:55.021865 1 queueinformer_operator.go:290] sync {"update" "test-operators/install-h9b9f"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-h9b9f": the object has been modified; please apply your changes to the latest version and try again time="2021-06-10T19:44:55Z" level=info msg=syncing id=XniA4 ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:44:55Z" level=warning msg="status not equal, updating..." id=XniA4 ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:44:55Z" level=info msg=syncing id=NdqTB ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:44:55Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:44:55Z" level=warning msg="status not equal, updating..." id=NdqTB ip=install-h9b9f namespace=test-operators phase=Installing E0610 19:44:56.022208 1 queueinformer_operator.go:290] sync {"update" "test-operators/install-h9b9f"} failed: failed to update installplan bundle lookups: Operation cannot be fulfilled on installplans.operators.coreos.com "install-h9b9f": the object has been modified; please apply your changes to the latest version and try again time="2021-06-10T19:44:56Z" level=info msg=syncing id=DpRfY ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:44:56Z" level=info msg=syncing id=001s/ ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:45:01Z" level=info msg=syncing id=PMV4c ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:45:06Z" level=info msg=syncing id=5DhGQ ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:45:09Z" level=info msg="catalog polling result: no update" CatalogSource=community-operators time="2021-06-10T19:45:09Z" level=info msg="catalog polling result: no update" CatalogSource=community-operators E0610 19:45:09.263258 1 queueinformer_operator.go:290] sync {"update" "openshift-marketplace/community-operators"} failed: couldn't ensure registry server - error ensuring updated catalog source pod: : error deleting duplicate catalog polling pod: community-operators-wgcbm: error deleting pod: community-operators-wgcbm: pods "community-operators-wgcbm" not found time="2021-06-10T19:45:11Z" level=info msg=syncing id=HYziS ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:45:16Z" level=info msg=syncing id=zrXh6 ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:45:21Z" level=info msg=syncing id=tGT0j ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdbackups.etcd.database.coreos.com.crd.yaml time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdclusters.etcd.database.coreos.com.crd.yaml time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdoperator.v0.9.4.clusterserviceversion.yaml time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdrestores.etcd.database.coreos.com.crd.yaml time="2021-06-10T19:45:21Z" level=warning msg="status not equal, updating..." id=tGT0j ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:45:21Z" level=info msg=syncing id=7U1L5 ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdclusters.etcd.database.coreos.com.crd.yaml time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdoperator.v0.9.4.clusterserviceversion.yaml time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdrestores.etcd.database.coreos.com.crd.yaml time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdbackups.etcd.database.coreos.com.crd.yaml time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:45:21Z" level=info msg=syncing id=sqQkm ip=install-h9b9f namespace=test-operators phase=Installing time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdbackups.etcd.database.coreos.com.crd.yaml time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdclusters.etcd.database.coreos.com.crd.yaml time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=ClusterServiceVersion" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdoperator.v0.9.4.clusterserviceversion.yaml time="2021-06-10T19:45:21Z" level=info msg="added to bundle, Kind=CustomResourceDefinition" configmap=openshift-marketplace/c8ec2a554f508a32921efcee390f67dd68efd5c9dd0ea8e76d8795d897e8cd2 key=etcdrestores.etcd.database.coreos.com.crd.yaml time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:45:21Z" level=info msg=syncing id=JCmhV ip=install-h9b9f namespace=test-operators phase=Complete time="2021-06-10T19:45:21Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:45:22Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd E0610 19:45:22.064155 1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again time="2021-06-10T19:45:22Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:45:22Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd E0610 19:45:22.264779 1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again time="2021-06-10T19:45:22Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:45:23Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd E0610 19:45:23.065496 1 queueinformer_operator.go:290] sync {"update" "test-operators/etcd"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd": the object has been modified; please apply your changes to the latest version and try again time="2021-06-10T19:45:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:45:23Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd time="2021-06-10T19:45:26Z" level=info msg=syncing id=5TpCw ip=install-h9b9f namespace=test-operators phase=Complete time="2021-06-10T19:45:26Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/test-operators/subscriptions/etcd There is no such log "creating installplan" after the "ResolutionFailed" event happens. LGTM, marking as verified.