Bug 2117324

Summary:	catalog-operator fatal error: concurrent map writes
Product:	OpenShift Container Platform	Reporter:	OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component:	OLM	Assignee:	Per da Silva <pegoncal>
OLM sub component:	OLM	QA Contact:	Jian Zhang <jiazha>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	agreene
Version:	4.11
Target Milestone:	---
Target Release:	4.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-23 15:11:19 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2101357
Bug Blocks:

Comment 1 Jian Zhang 2022-08-11 02:47:20 UTC

1, create a OCP cluster with the fixed PR via cluster-bot 
MacBook-Pro:~ jianzhang$ oc get clusterversion
NAME      VERSION                                                   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.ci.test-2022-08-11-015131-ci-ln-ss8ywtk-latest   True        False         9m47s   Cluster version is 4.11.0-0.ci.test-2022-08-11-015131-ci-ln-ss8ywtk-latest

2, check the marketplace-operator pod status

MacBook-Pro:~ jianzhang$ oc get pods -n openshift-marketplace
NAME                                   READY   STATUS    RESTARTS      AGE
certified-operators-nvqwr              1/1     Running   0             30m
community-operators-q7pwh              1/1     Running   0             30m
marketplace-operator-75b976c98-bfrgf   1/1     Running   3 (16m ago)   36m
redhat-marketplace-msjwh               1/1     Running   0             30m
redhat-operators-4h6wz                 1/1     Running   0             30m

MacBook-Pro:~ jianzhang$ oc logs marketplace-operator-75b976c98-bfrgf --previous
time="2022-08-11T02:12:51Z" level=info msg="Go Version: go1.18.1"
time="2022-08-11T02:12:51Z" level=info msg="Go OS/Arch: linux/amd64"
time="2022-08-11T02:12:51Z" level=info msg="[metrics] Registering marketplace metrics"
time="2022-08-11T02:12:51Z" level=info msg="[metrics] Serving marketplace metrics"
time="2022-08-11T02:12:51Z" level=info msg="TLS keys set, using https for metrics"
time="2022-08-11T02:12:51Z" level=info msg="Config API is available"
time="2022-08-11T02:12:51Z" level=info msg="setting up scheme"
I0811 02:12:52.554078       1 request.go:665] Waited for 1.045450354s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/cloud.network.openshift.io/v1?timeout=32s
time="2022-08-11T02:12:53Z" level=info msg="setting up health checks"
I0811 02:12:53.940925       1 leaderelection.go:248] attempting to acquire leader lease openshift-marketplace/marketplace-operator-lock...
I0811 02:12:53.977875       1 leaderelection.go:258] successfully acquired lease openshift-marketplace/marketplace-operator-lock
time="2022-08-11T02:12:53Z" level=info msg="became leader: marketplace-operator-75b976c98-bfrgf"
time="2022-08-11T02:12:53Z" level=info msg="registering components"
time="2022-08-11T02:12:53Z" level=info msg="setting up the marketplace clusteroperator status reporter"
time="2022-08-11T02:12:56Z" level=info msg="setting up controllers"
time="2022-08-11T02:12:56Z" level=info msg="ensuring the default catalogsource resources"
time="2022-08-11T02:12:59Z" level=info msg="starting the marketplace clusteroperator status reporter"
time="2022-08-11T02:12:59Z" level=info msg="starting manager"
time="2022-08-11T02:12:59Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:12:59Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:12:59Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:12:59Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:12:59Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:12:59Z" level=info msg="Reconciling ConfigMap openshift-marketplace/marketplace-trusted-ca"
time="2022-08-11T02:12:59Z" level=info msg="[ca] Certificate Authorization ConfigMap openshift-marketplace/marketplace-trusted-ca is in sync with disk." name=marketplace-trusted-ca type=ConfigMap
time="2022-08-11T02:13:02Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:13:02Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:13:02Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:13:02Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
E0811 02:15:24.554119       1 leaderelection.go:330] error retrieving resource lock openshift-marketplace/marketplace-operator-lock: Get "https://172.30.0.1:443/api/v1/namespaces/openshift-marketplace/configmaps/marketplace-operator-lock": dial tcp 172.30.0.1:443: connect: connection refused
time="2022-08-11T02:15:45Z" level=info msg="Reconciling ConfigMap openshift-marketplace/marketplace-trusted-ca"
time="2022-08-11T02:15:45Z" level=info msg="[ca] Certificate Authorization ConfigMap openshift-marketplace/marketplace-trusted-ca is in sync with disk." name=marketplace-trusted-ca type=ConfigMap
time="2022-08-11T02:15:50Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:15:50Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:15:50Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:15:50Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:15:50Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:15:50Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:15:50Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:15:50Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:15:50Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:15:50Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:16:22Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:16:22Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:16:22Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:16:22Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:16:44Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:16:44Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
E0811 02:18:24.622118       1 leaderelection.go:330] error retrieving resource lock openshift-marketplace/marketplace-operator-lock: Get "https://172.30.0.1:443/api/v1/namespaces/openshift-marketplace/configmaps/marketplace-operator-lock": dial tcp 172.30.0.1:443: connect: connection refused
E0811 02:18:54.623821       1 leaderelection.go:330] error retrieving resource lock openshift-marketplace/marketplace-operator-lock: Get "https://172.30.0.1:443/api/v1/namespaces/openshift-marketplace/configmaps/marketplace-operator-lock": dial tcp 172.30.0.1:443: connect: connection refused
time="2022-08-11T02:19:13Z" level=info msg="Reconciling ConfigMap openshift-marketplace/marketplace-trusted-ca"
time="2022-08-11T02:19:13Z" level=info msg="[ca] Certificate Authorization ConfigMap openshift-marketplace/marketplace-trusted-ca is in sync with disk." name=marketplace-trusted-ca type=ConfigMap
time="2022-08-11T02:19:18Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:19:18Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:18Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:18Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:18Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="Reconciling OperatorHub cluster"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:19Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:23Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:23Z" level=info msg="[defaults] CatalogSource redhat-marketplace is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:23Z" level=info msg="[defaults] CatalogSource redhat-operators is annotated and its spec is the same as the default spec"
time="2022-08-11T02:19:23Z" level=info msg="[defaults] CatalogSource certified-operators is annotated and its spec is the same as the default spec"
I0811 02:19:24.622864       1 leaderelection.go:283] failed to renew lease openshift-marketplace/marketplace-operator-lock: timed out waiting for the condition
E0811 02:19:24.623026       1 leaderelection.go:306] Failed to release lock: resource name may not be empty
time="2022-08-11T02:19:24Z" level=warning msg="leader election lost for marketplace-operator-75b976c98-bfrgf identity"


no "fatal error: concurrent map writes" found, the above error is as expected during the election, correct me if I'm wrong.

3, Subscribe to some operators

MacBook-Pro:~ jianzhang$ oc get sub -A
NAMESPACE        NAME             PACKAGE          SOURCE                CHANNEL
default          etcd             etcd             community-operators   singlenamespace-alpha
rhacs-operator   rhacs-operator   rhacs-operator   redhat-operators      latest
MacBook-Pro:~ jianzhang$ oc get csv -n rhacs-operator 
NAME                     DISPLAY                                    VERSION   REPLACES                 PHASE
rhacs-operator.v3.71.0   Advanced Cluster Security for Kubernetes   3.71.0    rhacs-operator.v3.70.0   Succeeded
MacBook-Pro:~ jianzhang$ oc get csv -n default
NAME                     DISPLAY                                    VERSION   REPLACES                 PHASE
etcdoperator.v0.9.4      etcd                                       0.9.4     etcdoperator.v0.9.2      Succeeded
rhacs-operator.v3.71.0   Advanced Cluster Security for Kubernetes   3.71.0    rhacs-operator.v3.70.0   Succeeded

4, Check marketplace-operator pod status again.
MacBook-Pro:~ jianzhang$ oc get pods
NAME                                                              READY   STATUS      RESTARTS      AGE
2cbaeff1fa9179d79c02e9bcedd66f96f0edd2c4f14196c2f62d834f4egc8jx   0/1     Completed   0             5m59s
certified-operators-nvqwr                                         1/1     Running     0             40m
community-operators-q7pwh                                         1/1     Running     0             40m
e8c9651078ae45ddb2807e3a07727d459b82d7def5572a7b7ccaae332bq4dq6   0/1     Completed   0             3m41s
marketplace-operator-75b976c98-bfrgf                              1/1     Running     3 (26m ago)   46m
redhat-marketplace-msjwh                                          1/1     Running     0             40m
redhat-operators-4h6wz                                            1/1     Running     0             40m

LGTM, verify it.

Comment 6 errata-xmlrpc 2022-08-23 15:11:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.11.1 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6103