Bug 1924257

Summary: controllers continuously busy reconciling operator
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: OLMAssignee: Haseeb Tariq <htariq>
OLM sub component: OLM QA Contact: xzha
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: agomezpr, bturner, daharmon, htariq, jiazha, krizza, oarribas, rpalathi, travi, tsmetana, vkochuku, xzha
Version: 4.6.zKeywords: Reopened, ServiceDeliveryImpact, Triaged
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-17 19:25:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1917537    
Bug Blocks:    

Comment 2 xzha 2021-02-05 05:08:17 UTC 
Verify it. LGTM

1. Install a 4.6.x openshift cluster with this bug fix
[root@preserve-olm-env 1924257]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2021-02-04-203135   True        False         35m     Cluster version is 4.6.0-0.nightly-2021-02-04-203135
[root@preserve-olm-env 1924257]# oc adm release info registry.ci.openshift.org/ocp/release:4.6.0-0.nightly-2021-02-04-203135 --commits|grep operator-lifecycle-manager
  operator-lifecycle-manager                     https://github.com/operator-framework/operator-lifecycle-manager            9dde58210568d45cdb8dfe9b67d4f4f24458a853

2. Install Red Hat Openshift Cluster Storage (OCS) operator and Advanced Cluster Management (ACM) operator, Wait for the operator to finish installing
login openshit, click Operators -> OperatorHub, search "OpenShift Container Storage" and "Advanced Cluster Management for Kubernetes", instal them
[root@preserve-olm-env 1924257]# oc get csv -n openshift-storage 
NAME                  DISPLAY                       VERSION   REPLACES   PHASE
ocs-operator.v4.6.2   OpenShift Container Storage   4.6.2                Succeeded
[root@preserve-olm-env 1924257]# oc get csv -n open-cluster-management
NAME                                 DISPLAY                                      VERSION   REPLACES                             PHASE
advanced-cluster-management.v2.1.2   Advanced Cluster Management for Kubernetes   2.1.2     advanced-cluster-management.v2.1.0   Succeeded

3. check the OLM operator pod's logs
[root@preserve-olm-env 1924257]# oc logs --since=400s olm-operator-5cc64d7f79-m8pfs | grep -i debug | grep -i "reconciling operator"
...
2021-02-05T04:51:24.725Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
2021-02-05T04:51:24.837Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
2021-02-05T04:51:24.997Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
2021-02-05T04:51:25.061Z	DEBUG	controllers.operator	reconciling operator	{"request": "/advanced-cluster-management.open-cluster-management"}
...
2021-02-05T04:51:36.704Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.712Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.715Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.723Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.792Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.894Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
2021-02-05T04:51:36.907Z	DEBUG	controllers.operator	reconciling operator	{"request": "/ocs-operator.openshift-storage"}
...

[root@preserve-olm-env 1924257]# oc logs --since=300s olm-operator-5cc64d7f79-m8pfs | grep -i debug | grep -i "reconciling operator"
[root@preserve-olm-env 1924257]# 

4. Observe the CPU core usage of the OLM operator pod from the console. 
The cpu usage is back to normal after the operator installation is finished.
https://user-images.githubusercontent.com/77608951/106992190-d3315f80-67b2-11eb-8ef4-bc4d81457725.png

5. check the CSV for the operator
[root@preserve-olm-env 1924257]# oc -n openshift-storage get csv -w
NAME                  DISPLAY                       VERSION   REPLACES   PHASE
ocs-operator.v4.6.2   OpenShift Container Storage   4.6.2                Succeeded

[root@preserve-olm-env 1924257]# oc -n openshift-storage get ip
NAME            CSV                   APPROVAL    APPROVED
install-gdgmc   ocs-operator.v4.6.2   Automatic   true
[root@preserve-olm-env 1924257]# 


Verified
Comment 6 errata-xmlrpc 2021-02-17 19:25:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.17 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0424