Bug 2054848

Summary: OLM, fatal error: concurrent map writes
Product: OpenShift Container Platform Reporter: Alexander Greene <agreene>
Component: OLMAssignee: Alexander Greene <agreene>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: pegoncal
Version: 4.7Keywords: Triaged
Target Milestone: ---   
Target Release: 4.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-04-08 09:56:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2003164    
Bug Blocks:    

Description Alexander Greene 2022-02-15 19:53:08 UTC 
This bug was initially created as a copy of Bug #2003164

I am copying this bug because: 

It needs to be backported to 4.8.


### Description of problem

The olm pod was restarted and it's possible to see in the logs, the next error:

~~~
$ oc logs <olm operator> -p > olm_logs
...
fatal error: concurrent map writes

goroutine 577 [running]:
runtime.throw(0x1eb5227, 0x15)
        /usr/lib/golang/src/runtime/panic.go:1116 +0x72 fp=0xc0286dba60 sp=0xc0286dba30 pc=0x43c1f2
runtime.mapassign_faststr(0x1c06100, 0xc01f70d6b0, 0x1ead4ad, 0x10, 0x3067bc0)
        /usr/lib/golang/src/runtime/map_faststr.go:211 +0x3f1 fp=0xc0286dbac8 sp=0xc0286dba60 pc=0x4190f1
github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm.(*Operator).transitionCSVState(0xc0005cb2c0, 0xc004ea3480, 0x15, 0xc004ea3460, 0x1d, 0xc00d5472f0, 0x21, 0x0, 0x0, 0xc004ea39a0, ...)
        /build/pkg/controller/operators/olm/operator.go:1412 +0x6d46 fp=0xc0286de5e0 sp=0xc0286dbac8 pc=0x19546c6
github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm.(*Operator).syncClusterServiceVersion(0xc0005cb2c0, 0x1e7e6a0, 0xc00ccfc008, 0xc01b0fd110, 0x1f84690)
        /build/pkg/controller/operators/olm/operator.go:1093 +0x6d8 fp=0xc0286df548 sp=0xc0286de5e0 pc=0x194aa78
github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/olm.(*Operator).syncClusterServiceVersion-fm(0x1e7e6a0, 0xc00ccfc008, 0xc00ccfc008, 0x1)
        /build/pkg/controller/operators/olm/operator.go:1066 +0x3e fp=0xc0286df580 sp=0xc0286df548 pc=0x196fede
github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.LegacySyncHandler.ToSyncerWithDelete.func1(0x216bda0, 0xc000b92040, 0x2137520, 0xc00d44fb20, 0xc00d44fb20, 0x1d13940)
        /build/pkg/lib/queueinformer/queueinformer.go:183 +0x26e fp=0xc0286df610 sp=0xc0286df580 pc=0x15e112e
github.com/operator-framework/operator-lifecycle-manager/pkg/lib/kubestate.SyncFunc.Sync(0xc0009a2360, 0x216bda0, 0xc000b92040, 0x2137520, 0xc00d44fb20, 0xc01b0fce01, 0x0)
        /build/pkg/lib/kubestate/kubestate.go:184 +0x4e fp=0xc0286df650 sp=0xc0286df610 pc=0x15d91ee
github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*QueueInformer).Sync(...)
...
~~~

### Version-Release number of selected component (if applicable):
~~~
$ oc get clusterversion
NAME     VERSION  AVAILABLE  PROGRESSING  SINCE  STATUS
version  4.7.24   True       False        3d     Cluster version is 4.7.24
~~~


The complete stack trace of the olm will be linked to the bug.
Comment 2 Jian Zhang 2022-03-17 06:41:54 UTC 
1, Install a cluster with the fixed PR via the cluster-bot
mac:~ jianzhang$ oc get clusterversion
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.ci.test-2022-03-17-055113-ci-ln-44p0tlk-latest   True        False         5m6s    Cluster version is 4.9.0-0.ci.test-2022-03-17-055113-ci-ln-44p0tlk-latest

2, Subscribe to some operators.
mac:~ jianzhang$ oc get sub -A
NAMESPACE             NAME          PACKAGE       SOURCE                CHANNEL
default               etcd          etcd          community-operators   singlenamespace-alpha
openshift-operators   amq-streams   amq-streams   redhat-operators      stable

mac:~ jianzhang$ oc get csv -n default
NAME                  DISPLAY                             VERSION   REPLACES              PHASE
amqstreams.v2.0.1-1   Red Hat Integration - AMQ Streams   2.0.1-1   amqstreams.v2.0.1-0   Succeeded
etcdoperator.v0.9.4   etcd                                0.9.4     etcdoperator.v0.9.2   Succeeded

3, Check the olm-operator logs.
mac:~ jianzhang$ oc logs deployment/olm-operator -n openshift-operator-lifecycle-manager|grep "fatal error"
mac:~ jianzhang$ 

Not found the `fatal error`, LGTM, verify it.
Comment 5 Jian Zhang 2022-04-02 00:48:20 UTC 
Change it to VERIFIED based on comment2.
Comment 7 errata-xmlrpc 2022-04-08 09:56:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.9.27 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1158