1924257 – controllers continuously busy reconciling operator

Bug 1924257 - controllers continuously busy reconciling operator

Summary: controllers continuously busy reconciling operator

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	OLM
Sub Component:
Version:	4.6.z
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	---
Target Release:	4.6.z
Assignee:	Haseeb Tariq
QA Contact:	xzha
Docs Contact:
URL:
Whiteboard:
Depends On:	1917537
Blocks:
TreeView+	depends on / blocked

Reported:	2021-02-02 21:12 UTC by OpenShift BugZilla Robot
Modified:	2024-06-14 00:08 UTC (History)
CC List:	12 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-02-17 19:25:10 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	operator-framework operator-lifecycle-manager pull 1991	None	closed	[release-4.6] Bug 1924257: Fix time comparison in CSV reconcile loop	2021-02-18 00:16:32 UTC
Red Hat Knowledge Base (Solution)	5743951	None	None	None	2021-06-21 08:29:18 UTC
Red Hat Product Errata	RHBA-2021:0424	None	None	None	2021-02-17 19:25:32 UTC

Comment 2 xzha 2021-02-05 05:08:17 UTC

Verify it. LGTM

1. Install a 4.6.x openshift cluster with this bug fix
[root@preserve-olm-env 1924257]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-02-04-203135   True        False         35m     Cluster version is 4.6.0-0.nightly-2021-02-04-203135
[root@preserve-olm-env 1924257]# oc adm release info registry.ci.openshift.org/ocp/release:4.6.0-0.nightly-2021-02-04-203135 --commits|grep operator-lifecycle-manager
  operator-lifecycle-manager                     https://github.com/operator-framework/operator-lifecycle-manager            9dde58210568d45cdb8dfe9b67d4f4f24458a853

2. Install Red Hat Openshift Cluster Storage (OCS) operator and Advanced Cluster Management (ACM) operator, Wait for the operator to finish installing
login openshit, click Operators -> OperatorHub, search "OpenShift Container Storage" and "Advanced Cluster Management for Kubernetes", instal them
[root@preserve-olm-env 1924257]# oc get csv -n openshift-storage 
NAME                  DISPLAY                       VERSION   REPLACES   PHASE
ocs-operator.v4.6.2   OpenShift Container Storage   4.6.2                Succeeded
[root@preserve-olm-env 1924257]# oc get csv -n open-cluster-management
NAME                                 DISPLAY                                      VERSION   REPLACES                             PHASE
advanced-cluster-management.v2.1.2   Advanced Cluster Management for Kubernetes   2.1.2     advanced-cluster-management.v2.1.0   Succeeded

3. check the OLM operator pod's logs
[root@preserve-olm-env 1924257]# oc logs --since=400s olm-operator-5cc64d7f79-m8pfs | grep -i debug | grep -i "reconciling operator"
...
2021-02-05T04:51:24.725Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
2021-02-05T04:51:24.837Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
2021-02-05T04:51:24.997Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
2021-02-05T04:51:25.061Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
...
2021-02-05T04:51:36.704Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.712Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.715Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.723Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.792Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.894Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.907Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
...

[root@preserve-olm-env 1924257]# oc logs --since=300s olm-operator-5cc64d7f79-m8pfs | grep -i debug | grep -i "reconciling operator"
[root@preserve-olm-env 1924257]# 

4. Observe the CPU core usage of the OLM operator pod from the console. 
The cpu usage is back to normal after the operator installation is finished.
https://user-images.githubusercontent.com/77608951/106992190-d3315f80-67b2-11eb-8ef4-bc4d81457725.png

5. check the CSV for the operator
[root@preserve-olm-env 1924257]# oc -n openshift-storage get csv -w
NAME                  DISPLAY                       VERSION   REPLACES   PHASE
ocs-operator.v4.6.2   OpenShift Container Storage   4.6.2                Succeeded

[root@preserve-olm-env 1924257]# oc -n openshift-storage get ip
NAME            CSV                   APPROVAL    APPROVED
install-gdgmc   ocs-operator.v4.6.2   Automatic   true
[root@preserve-olm-env 1924257]# 


Verified

Comment 6 errata-xmlrpc 2021-02-17 19:25:10 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.17 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0424

Note You need to log in before you can comment on or make changes to this bug.