1982252 – olm Operator is in CrashLoopBackOff state with error "couldn't cleanup cross-namespace ownerreferences"

Bug 1982252 - olm Operator is in CrashLoopBackOff state with error "couldn't cleanup cross-namespace ownerreferences"

Summary: olm Operator is in CrashLoopBackOff state with error "couldn't cleanup cross-...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	OLM
Sub Component:
Version:	4.7
Hardware:	x86_64
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Kevin Rizza
QA Contact:	Jian Zhang
Docs Contact:
URL:
Whiteboard:
Depends On:	1979544
Blocks:	1982693
TreeView+	depends on / blocked

Reported:	2021-07-14 14:26 UTC by Kevin Rizza
Modified:	2022-10-11 06:39 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1979544
Clones:	1982693 (view as bug list)
Environment:
Last Closed:	2021-07-27 23:14:04 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift operator-framework-olm pull 117	0	None	open	[release-4.8] Bug 1982252: remove cleanup from startup	2021-07-14 15:18:39 UTC
Red Hat Product Errata	RHSA-2021:2438	0	None	None	None	2021-07-27 23:14:08 UTC

Comment 2 Jian Zhang 2021-07-16 03:41:45 UTC

[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-07-15-221145   True        False         72m     Cluster version is 4.8.0-0.nightly-2021-07-15-221145

[cloud-user@preserve-olm-env jian]$ oc adm release info registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-07-15-221145 --commits|grep lifecycle
  operator-lifecycle-manager                     https://github.com/openshift/operator-framework-olm                         505fe20e27838e8f9e33c2eeb30a56fbadfd6500

This payload contains the fixed PR.

1, Install some cluster-scoped operators.

[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-operators
NAME                                DISPLAY                            VERSION             REPLACES                            PHASE
elasticsearch-operator.5.1.0-96     OpenShift Elasticsearch Operator   5.1.0-96                                                Succeeded
etcdoperator.v0.9.4-clusterwide     etcd                               0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide     Succeeded
jaeger-operator.v1.20.4             Red Hat OpenShift Jaeger           1.20.4                                                  Installing
openshift-gitops-operator.v1.1.2    Red Hat OpenShift GitOps           1.1.2               openshift-gitops-operator.v1.1.1    Succeeded
redhat-openshift-pipelines.v1.4.1   Red Hat OpenShift Pipelines        1.4.1               redhat-openshift-pipelines.v1.4.0   Succeeded
servicemeshoperator.v2.0.6.2        Red Hat OpenShift Service Mesh     2.0.6-2             servicemeshoperator.v2.0.6.1        Succeeded

2, Create 400 namespaces, but 331 in fact, 

[cloud-user@preserve-olm-env jian]$ for l in {1..399}; do oc adm new-project "test$l";sleep 1; done;
Created project test200
...
Created project test331
^C

3, Check the CSV numbers and the OLM pods.
[cloud-user@preserve-olm-env jian]$ oc get csv -A|wc -l
2392

[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-676568bc96-h6kzz   1/1     Running   0          143m
olm-operator-b85fd4689-hccf8        1/1     Running   0          144m
packageserver-84b68df595-4vkmp      1/1     Running   0          138m
packageserver-84b68df595-n9b4m      1/1     Running   0          138m

4, OLM pods works well, install some new operators. Looks good.

[cloud-user@preserve-olm-env jian]$ oc get sub -A
NAMESPACE                    NAME                              PACKAGE                           SOURCE                CHANNEL
default                      ptp-operator                      ptp-operator                      qe-app-registry       4.8
openshift-logging            cluster-logging                   cluster-logging                   qe-app-registry       stable-5.1
openshift-operators-redhat   elasticsearch-operator            elasticsearch-operator            qe-app-registry       stable-5.1
openshift-operators          etcd                              etcd                              community-operators   clusterwide-alpha
openshift-operators          jaeger-product                    jaeger-product                    redhat-operators      stable
openshift-operators          nfd                               nfd                               qe-app-registry       4.9
openshift-operators          openshift-gitops-operator         openshift-gitops-operator         redhat-operators      stable
openshift-operators          openshift-pipelines-operator-rh   openshift-pipelines-operator-rh   redhat-operators      stable
openshift-operators          servicemeshoperator               servicemeshoperator               redhat-operators      stable
[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-operators 
NAME                                     DISPLAY                            VERSION              REPLACES                            PHASE
elasticsearch-operator.5.1.0-96          OpenShift Elasticsearch Operator   5.1.0-96                                                 Succeeded
etcdoperator.v0.9.4-clusterwide          etcd                               0.9.4-clusterwide    etcdoperator.v0.9.2-clusterwide     Succeeded
jaeger-operator.v1.20.4                  Red Hat OpenShift Jaeger           1.20.4                                                   Succeeded
node-feature-discovery-operator.v4.9.0   Node Feature Discovery             4.9.0-202107090514                                       Installing
openshift-gitops-operator.v1.1.2         Red Hat OpenShift GitOps           1.1.2                openshift-gitops-operator.v1.1.1    Succeeded
redhat-openshift-pipelines.v1.4.1        Red Hat OpenShift Pipelines        1.4.1                redhat-openshift-pipelines.v1.4.0   Succeeded
servicemeshoperator.v2.0.6.2             Red Hat OpenShift Service Mesh     2.0.6-2              servicemeshoperator.v2.0.6.1        Succeeded

5, Remove all cluster-scoped operators,
[cloud-user@preserve-olm-env jian]$  oc get csv -n openshift-operators
No resources found in openshift-operators namespace.
[cloud-user@preserve-olm-env jian]$ oc get csv |wc -l
8
[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-676568bc96-h6kzz   1/1     Running   2          161m
olm-operator-b85fd4689-hccf8        1/1     Running   1          161m
packageserver-84b68df595-4vkmp      1/1     Running   2          155m
packageserver-84b68df595-n9b4m      1/1     Running   2          155m

No OLM pods crased, looks good, verify it.

Comment 5 errata-xmlrpc 2021-07-27 23:14:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Note You need to log in before you can comment on or make changes to this bug.