Bug 1982252

Summary: olm Operator is in CrashLoopBackOff state with error "couldn't cleanup cross-namespace ownerreferences"
Product: OpenShift Container Platform Reporter: Kevin Rizza <krizza>
Component: OLMAssignee: Kevin Rizza <krizza>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: akashem, davegord, dsover, jiazha, sreber
Version: 4.7   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1979544
: 1982693 (view as bug list) Environment:
Last Closed: 2021-07-27 23:14:04 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1979544    
Bug Blocks: 1982693    

Comment 2 Jian Zhang 2021-07-16 03:41:45 UTC 
[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-07-15-221145   True        False         72m     Cluster version is 4.8.0-0.nightly-2021-07-15-221145

[cloud-user@preserve-olm-env jian]$ oc adm release info registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-07-15-221145 --commits|grep lifecycle
  operator-lifecycle-manager                     https://github.com/openshift/operator-framework-olm                         505fe20e27838e8f9e33c2eeb30a56fbadfd6500

This payload contains the fixed PR.

1, Install some cluster-scoped operators.

[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-operators
NAME                                DISPLAY                            VERSION             REPLACES                            PHASE
elasticsearch-operator.5.1.0-96     OpenShift Elasticsearch Operator   5.1.0-96                                                Succeeded
etcdoperator.v0.9.4-clusterwide     etcd                               0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide     Succeeded
jaeger-operator.v1.20.4             Red Hat OpenShift Jaeger           1.20.4                                                  Installing
openshift-gitops-operator.v1.1.2    Red Hat OpenShift GitOps           1.1.2               openshift-gitops-operator.v1.1.1    Succeeded
redhat-openshift-pipelines.v1.4.1   Red Hat OpenShift Pipelines        1.4.1               redhat-openshift-pipelines.v1.4.0   Succeeded
servicemeshoperator.v2.0.6.2        Red Hat OpenShift Service Mesh     2.0.6-2             servicemeshoperator.v2.0.6.1        Succeeded

2, Create 400 namespaces, but 331 in fact, 

[cloud-user@preserve-olm-env jian]$ for l in {1..399}; do oc adm new-project "test$l";sleep 1; done;
Created project test200
...
Created project test331
^C

3, Check the CSV numbers and the OLM pods.
[cloud-user@preserve-olm-env jian]$ oc get csv -A|wc -l
2392

[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-676568bc96-h6kzz   1/1     Running   0          143m
olm-operator-b85fd4689-hccf8        1/1     Running   0          144m
packageserver-84b68df595-4vkmp      1/1     Running   0          138m
packageserver-84b68df595-n9b4m      1/1     Running   0          138m

4, OLM pods works well, install some new operators. Looks good.

[cloud-user@preserve-olm-env jian]$ oc get sub -A
NAMESPACE                    NAME                              PACKAGE                           SOURCE                CHANNEL
default                      ptp-operator                      ptp-operator                      qe-app-registry       4.8
openshift-logging            cluster-logging                   cluster-logging                   qe-app-registry       stable-5.1
openshift-operators-redhat   elasticsearch-operator            elasticsearch-operator            qe-app-registry       stable-5.1
openshift-operators          etcd                              etcd                              community-operators   clusterwide-alpha
openshift-operators          jaeger-product                    jaeger-product                    redhat-operators      stable
openshift-operators          nfd                               nfd                               qe-app-registry       4.9
openshift-operators          openshift-gitops-operator         openshift-gitops-operator         redhat-operators      stable
openshift-operators          openshift-pipelines-operator-rh   openshift-pipelines-operator-rh   redhat-operators      stable
openshift-operators          servicemeshoperator               servicemeshoperator               redhat-operators      stable
[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-operators 
NAME                                     DISPLAY                            VERSION              REPLACES                            PHASE
elasticsearch-operator.5.1.0-96          OpenShift Elasticsearch Operator   5.1.0-96                                                 Succeeded
etcdoperator.v0.9.4-clusterwide          etcd                               0.9.4-clusterwide    etcdoperator.v0.9.2-clusterwide     Succeeded
jaeger-operator.v1.20.4                  Red Hat OpenShift Jaeger           1.20.4                                                   Succeeded
node-feature-discovery-operator.v4.9.0   Node Feature Discovery             4.9.0-202107090514                                       Installing
openshift-gitops-operator.v1.1.2         Red Hat OpenShift GitOps           1.1.2                openshift-gitops-operator.v1.1.1    Succeeded
redhat-openshift-pipelines.v1.4.1        Red Hat OpenShift Pipelines        1.4.1                redhat-openshift-pipelines.v1.4.0   Succeeded
servicemeshoperator.v2.0.6.2             Red Hat OpenShift Service Mesh     2.0.6-2              servicemeshoperator.v2.0.6.1        Succeeded

5, Remove all cluster-scoped operators,
[cloud-user@preserve-olm-env jian]$  oc get csv -n openshift-operators
No resources found in openshift-operators namespace.
[cloud-user@preserve-olm-env jian]$ oc get csv |wc -l
8
[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-676568bc96-h6kzz   1/1     Running   2          161m
olm-operator-b85fd4689-hccf8        1/1     Running   1          161m
packageserver-84b68df595-4vkmp      1/1     Running   2          155m
packageserver-84b68df595-n9b4m      1/1     Running   2          155m

No OLM pods crased, looks good, verify it.
Comment 5 errata-xmlrpc 2021-07-27 23:14:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438