Bug 2105045

Summary:	OLM updates namespace labels even if they haven't changed
Product:	OpenShift Container Platform	Reporter:	Alexander Greene <agreene>
Component:	OLM	Assignee:	Alexander Greene <agreene>
OLM sub component:	OLM	QA Contact:	xzha
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	jlanford, stevsmit
Version:	4.11
Target Milestone:	---
Target Release:	4.12.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	* Previously, Operator Lifecycle Manager (OLM) would attempt to update namespaces to apply a label, even if the label was present on the namespace. Consequently, the update requests increased the workload in API and etcd services. With this update, OLM compares existing labels against the expected labels on a namespace before issuing an update. As a result, OLM no longer attempts to make unnecessary update requests on namespaces. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2105045[BZ#2105045])	Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-01-17 19:51:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2107045

Description Alexander Greene 2022-07-07 18:54:23 UTC

Description of problem:
OperatorGroups are used to scope an operator to a set of namespaces.
OLM applies a label to each of the namespaces targeted by an operatorGroup.
There is an issue in OLM where OLM updates a namespace's labels even if the labels haven't been changed. Effectively, the update request is a semantic
no-op and it generates a apiserver and etcd load. Also, other controllers 
watching these namespaces will perform redundant work in response to the
generated watch event and resourceVersion bump.

Version-Release number of selected component (if applicable):
4.11

How reproducible:
Always

Steps to Reproduce:
1. Create an OperatorGroup that targets specific namespaces.

Actual results:
OLM updates the namespaces multiple times even though the labels it applies does not change.

Expected results:
OLM updates the namespaces once and does not update the namespace if the labels haven't changed.

Additional info:

Comment 1 Alexander Greene 2022-07-07 18:54:52 UTC

Upstream PR can be found here: https://github.com/operator-framework/operator-lifecycle-manager/pull/2809/

Comment 2 xzha 2022-07-13 10:16:38 UTC

verify:

1) install cluster
zhaoxia@xzha-mac test % oc exec olm-operator-55c86c57f9-kzt4z  -- olm --version
OLM version: 0.19.0
git commit: 993f841d30a44513c792eb37a07c02e57fdb52a6

2) create ns and og
zhaoxia@xzha-mac test % oc new-project test-2
zhaoxia@xzha-mac test % cat og.yaml.2
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og-single
  namespace: test-2
spec:
  targetNamespaces:
  - test-2
zhaoxia@xzha-mac test % oc apply -f og.yaml.2

3) check olm operator log
zhaoxia@xzha-mac test % grep test-2 olm.log.new3
time="2022-07-13T10:03:04Z" level=debug msg="updated target namespaces" namespace=test-2 operatorGroup=og-single targetNamespaces="[test-2]"
time="2022-07-13T10:03:04Z" level=debug msg="OperatorGroup namespaces change detected" namespace=test-2 operatorGroup=og-single
time="2022-07-13T10:03:04Z" level=debug msg="namespace change detected" namespace=test-2 operatorGroup=og-single targets="[test-2]"
time="2022-07-13T10:03:04Z" level=debug msg="operatorgroup status updated" namespace=test-2 operatorGroup=og-single
time="2022-07-13T10:03:04Z" level=debug msg="Requeueing out of sync namespaces" namespace=test-2 operatorGroup=og-single
time="2022-07-13T10:03:04Z" level=debug msg=requeueing namespace=test-2 operatorGroup=og-single
time="2022-07-13T10:03:04Z" level=debug msg="updated target namespaces" namespace=test-2 operatorGroup=og-single targetNamespaces="[test-2]"
time="2022-07-13T10:03:04Z" level=debug msg="check that operatorgroup has updated CSV anotations" namespace=test-2 operatorGroup=og-single
time="2022-07-13T10:03:04Z" level=debug msg="OperatorGroup CSV annotation completed" namespace=test-2 operatorGroup=og-single
time="2022-07-13T10:03:04Z" level=debug msg="operatorgroup clusterroles ensured" namespace=test-2 operatorGroup=og-single

no log "* sync {"update" "test-2"} *"

LGTM, verified.

Comment 4 xzha 2022-07-15 09:53:47 UTC

verify

zhaoxia@xzha-mac test % oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-0.nightly-2022-07-15-065851   True        False         61m     Cluster version is 4.12.0-0.nightly-2022-07-15-065851
zhaoxia@xzha-mac test % oc exec catalog-operator-5d89d7f4b9-x5559  -- olm --version
OLM version: 0.19.0
git commit: 404ac3ca3f63e0e11f05e7ccb90ca8031ab8d3d5

1, create ns
zhaoxia@xzha-mac test % oc new-project test-1
Now using project "test-1" on server "https://api.xzha-0715.ibmcloud.qe.devcluster.openshift.com:6443".

2, create og
zhaoxia@xzha-mac test % cat og.yaml
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og-single
  namespace: test-1
spec:
  targetNamespaces:
  - test-1
zhaoxia@xzha-mac test % oc apply -f og.yaml
operatorgroup.operators.coreos.com/og-single created

3, check olm log
zhaoxia@xzha-mac test % oc get pod
NAME                                      READY   STATUS      RESTARTS      AGE
catalog-operator-5d89d7f4b9-x5559         1/1     Running     0             86m
collect-profiles-27631275-t9m2m           0/1     Completed   0             34m
collect-profiles-27631290-6p2m7           0/1     Completed   0             19m
collect-profiles-27631305-m76qw           0/1     Completed   0             4m59s
olm-operator-6d6f96787f-pk6jh             1/1     Running     0             86m
package-server-manager-6cb887ddb4-gvg94   1/1     Running     1 (68m ago)   86m
packageserver-757bf85df6-mkzrp            1/1     Running     0             80m
packageserver-757bf85df6-zf2hk            1/1     Running     0             80m

zhaoxia@xzha-mac test % oc logs olm-operator-6d6f96787f-pk6jh | grep test-1
zhaoxia@xzha-mac test %

no log "* sync {"update" "test-1"} *"

lgtm, verified.

Comment 9 errata-xmlrpc 2023-01-17 19:51:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399