Bug 1924257 - controllers continuously busy reconciling operator
Summary: controllers continuously busy reconciling operator
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.6.z
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.6.z
Assignee: Haseeb Tariq
QA Contact: xzha
URL:
Whiteboard:
Depends On: 1917537
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-02 21:12 UTC by OpenShift BugZilla Robot
Modified: 2021-02-17 19:25 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-17 19:25:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1991 0 None closed [release-4.6] Bug 1924257: Fix time comparison in CSV reconcile loop 2021-02-18 00:16:32 UTC
Red Hat Product Errata RHBA-2021:0424 0 None None None 2021-02-17 19:25:32 UTC

Comment 2 xzha 2021-02-05 05:08:17 UTC
Verify it. LGTM

1. Install a 4.6.x openshift cluster with this bug fix
[root@preserve-olm-env 1924257]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-02-04-203135   True        False         35m     Cluster version is 4.6.0-0.nightly-2021-02-04-203135
[root@preserve-olm-env 1924257]# oc adm release info registry.ci.openshift.org/ocp/release:4.6.0-0.nightly-2021-02-04-203135 --commits|grep operator-lifecycle-manager
  operator-lifecycle-manager                     https://github.com/operator-framework/operator-lifecycle-manager            9dde58210568d45cdb8dfe9b67d4f4f24458a853

2. Install Red Hat Openshift Cluster Storage (OCS) operator and Advanced Cluster Management (ACM) operator, Wait for the operator to finish installing
login openshit, click Operators -> OperatorHub, search "OpenShift Container Storage" and "Advanced Cluster Management for Kubernetes", instal them
[root@preserve-olm-env 1924257]# oc get csv -n openshift-storage 
NAME                  DISPLAY                       VERSION   REPLACES   PHASE
ocs-operator.v4.6.2   OpenShift Container Storage   4.6.2                Succeeded
[root@preserve-olm-env 1924257]# oc get csv -n open-cluster-management
NAME                                 DISPLAY                                      VERSION   REPLACES                             PHASE
advanced-cluster-management.v2.1.2   Advanced Cluster Management for Kubernetes   2.1.2     advanced-cluster-management.v2.1.0   Succeeded

3. check the OLM operator pod's logs
[root@preserve-olm-env 1924257]# oc logs --since=400s olm-operator-5cc64d7f79-m8pfs | grep -i debug | grep -i "reconciling operator"
...
2021-02-05T04:51:24.725Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
2021-02-05T04:51:24.837Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
2021-02-05T04:51:24.997Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
2021-02-05T04:51:25.061Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
...
2021-02-05T04:51:36.704Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.712Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.715Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.723Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.792Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.894Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.907Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
...

[root@preserve-olm-env 1924257]# oc logs --since=300s olm-operator-5cc64d7f79-m8pfs | grep -i debug | grep -i "reconciling operator"
[root@preserve-olm-env 1924257]# 

4. Observe the CPU core usage of the OLM operator pod from the console. 
The cpu usage is back to normal after the operator installation is finished.
https://user-images.githubusercontent.com/77608951/106992190-d3315f80-67b2-11eb-8ef4-bc4d81457725.png

5. check the CSV for the operator
[root@preserve-olm-env 1924257]# oc -n openshift-storage get csv -w
NAME                  DISPLAY                       VERSION   REPLACES   PHASE
ocs-operator.v4.6.2   OpenShift Container Storage   4.6.2                Succeeded

[root@preserve-olm-env 1924257]# oc -n openshift-storage get ip
NAME            CSV                   APPROVAL    APPROVED
install-gdgmc   ocs-operator.v4.6.2   Automatic   true
[root@preserve-olm-env 1924257]# 


Verified

Comment 6 errata-xmlrpc 2021-02-17 19:25:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.17 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0424


Note You need to log in before you can comment on or make changes to this bug.