Bug 2043301

Summary: Upgrade: Reconcile error in container log for every upgrade/child policy (no other noticeable impact)
Product: OpenShift Container Platform Reporter: yliu1
Component: Telco EdgeAssignee: jun
Telco Edge sub component: TALO QA Contact: yliu1
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: unspecified CC: ijolliff, jun
Version: 4.10   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-26 16:43:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yliu1 2022-01-21 00:31:17 UTC
Description of problem:
In every upgrade, one or multiple reconcile errors will show in container log. The number of errors seems to map to the child policy count. 

Other than this error in log, there is no other noticeable impact from user perspective.

2022-01-21T00:14:44.697Z        INFO    controllers.ClusterGroupUpgrade Upgrade is completed
2022-01-21T00:14:44.892Z        ERROR   controller-runtime.manager.controller.clustergroupupgrade       Reconciler error        {"reconciler group": "ran.openshift.io", "reconciler kind": "ClusterGroupUpgrade", "name": "timeout", "namespace": "default", "error": "Operation cannot be fulfilled on clustergroupupgrades.ran.openshift.io \"timeout\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime.2/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime.2/pkg/internal/controller/controller.go:214


Version-Release number of selected component (if applicable):
4.10

How reproducible:
100%


Steps to Reproduce:
1. Create and enable UOCR to trigger an upgrade
2. After upgrade is completed, check container log 


Actual results:
oc logs -n openshift-cluster-group-upgrades cluster-group-upgrades-controller-manager-56c86675df-zwp2t manager | grep ERROR
....
2022-01-20T23:35:34.373Z	ERROR	controller-runtime.manager.controller.clustergroupupgrade	Reconciler error	{"reconciler group": "ran.openshift.io", "reconciler kind": "ClusterGroupUpgrade", "name": "selector", "namespace": "default", "error": "Operation cannot be fulfilled on placementrules.apps.open-cluster-management.io \"selector-common-cluster-version-policy1\": the object has been modified; please apply your changes to the latest version and try again"}
2022-01-20T23:40:35.209Z	ERROR	controller-runtime.manager.controller.clustergroupupgrade	Reconciler error	{"reconciler group": "ran.openshift.io", "reconciler kind": "ClusterGroupUpgrade", "name": "selector", "namespace": "default", "error": "Operation cannot be fulfilled on clustergroupupgrades.ran.openshift.io \"selector\": the object has been modified; please apply your changes to the latest version and try again"}
2022-01-20T23:59:00.585Z	ERROR	controller-runtime.manager.controller.clustergroupupgrade	Reconciler error	{"reconciler group": "ran.openshift.io", "reconciler kind": "ClusterGroupUpgrade", "name": "withcanaries", "namespace": "default", "error": "Operation cannot be fulfilled on clustergroupupgrades.ran.openshift.io \"withcanaries\": the object has been modified; please apply your changes to the latest version and try again"}
2022-01-21T00:09:43.820Z	ERROR	controller-runtime.manager.controller.clustergroupupgrade	Reconciler error	{"reconciler group": "ran.openshift.io", "reconciler kind": "ClusterGroupUpgrade", "name": "timeout", "namespace": "default", "error": "Operation cannot be fulfilled on placementrules.apps.open-cluster-management.io \"timeout-common-cluster-version-policy1\": the object has been modified; please apply your changes to the latest version and try again"}
2022-01-21T00:09:43.986Z	ERROR	controller-runtime.manager.controller.clustergroupupgrade	Reconciler error	{"reconciler group": "ran.openshift.io", "reconciler kind": "ClusterGroupUpgrade", "name": "timeout", "namespace": "default", "error": "Operation cannot be fulfilled on placementrules.apps.open-cluster-management.io \"timeout-common-cluster-version-policy1\": the object has been modified; please apply your changes to the latest version and try again"}
2022-01-21T00:14:44.892Z	ERROR	controller-runtime.manager.controller.clustergroupupgrade	Reconciler error	{"reconciler group": "ran.openshift.io", "reconciler kind": "ClusterGroupUpgrade", "name": "timeout", "namespace": "default", "error": "Operation cannot be fulfilled on clustergroupupgrades.ran.openshift.io \"timeout\": the object has been modified; please apply your changes to the latest version and try again"}


Expected results:
- No error in container log


Additional info:

Comment 3 yliu1 2022-07-05 19:27:31 UTC
Verified with latest 4.10 TALM build.