Bug 1982693 - olm Operator is in CrashLoopBackOff state with error "couldn't cleanup cross-namespace ownerreferences"
Summary: olm Operator is in CrashLoopBackOff state with error "couldn't cleanup cross-...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.7
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: 4.7.z
Assignee: Kevin Rizza
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1982252
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-07-15 13:12 UTC by Kevin Rizza
Modified: 2021-08-23 02:00 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1982252
Environment:
Last Closed: 2021-08-03 17:56:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 2287 0 None open [release-4.7] Bug 1982693: remove cleanup from startup 2021-07-23 21:08:49 UTC
Red Hat Product Errata RHBA-2021:2903 0 None None None 2021-08-03 17:56:49 UTC

Comment 3 Jian Zhang 2021-07-26 03:26:18 UTC
[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-07-24-034734   True        False         4m17s   Cluster version is 4.7.0-0.nightly-2021-07-24-034734

[cloud-user@preserve-olm-env jian]$ oc -n openshift-operator-lifecycle-manager exec deploy/catalog-operator -- olm --version
OLM version: 0.17.0
git commit: dcbab9aff049c39d3a8a4b544d8829b7ce93d8f8

This payload contains the fixed PR.

1, Install some cluster-scoped operators.
[cloud-user@preserve-olm-env jian]$ oc get sub -A
NAMESPACE                    NAME                     PACKAGE                  SOURCE                CHANNEL
openshift-operators-redhat   elasticsearch-operator   elasticsearch-operator   redhat-operators      stable
openshift-operators          argocd-operator          argocd-operator          community-operators   alpha
openshift-operators          etcd                     etcd                     community-operators   clusterwide-alpha
openshift-operators          jaeger                   jaeger                   community-operators   stable
openshift-operators          servicemeshoperator      servicemeshoperator      redhat-operators      stable

[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-operators
NAME                              DISPLAY                            VERSION             REPLACES                          PHASE
argocd-operator.v0.0.15           Argo CD                            0.0.15              argocd-operator.v0.0.14           Succeeded
elasticsearch-operator.5.1.0-96   OpenShift Elasticsearch Operator   5.1.0-96                                              Succeeded
etcdoperator.v0.9.4-clusterwide   etcd                               0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide   Succeeded
jaeger-operator.v1.24.0           Community Jaeger Operator          1.24.0              jaeger-operator.v1.21.3           Succeeded
servicemeshoperator.v2.0.6.2      Red Hat OpenShift Service Mesh     2.0.6-2             servicemeshoperator.v2.0.6.1      Succeeded
[cloud-user@preserve-olm-env jian]$ oc get csv -A|wc -l
322

[cloud-user@preserve-olm-env jian]$ oc get ns |wc -l
65

2, Create 300 namespaces
[cloud-user@preserve-olm-env jian]$ for l in {1..300}; do oc adm new-project "test$l";sleep 1; done;
...

3, Check the CSV numbers and the OLM pods.
[cloud-user@preserve-olm-env jian]$ oc get csv -A|wc -l
1822
[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-operator-lifecycle-manager
NAME                               READY   STATUS    RESTARTS   AGE
catalog-operator-f966bdcbf-gtjr6   1/1     Running   0          64m
olm-operator-5696d565d7-jvnjx      1/1     Running   0          64m
packageserver-7d67fd99cb-b4ds7     1/1     Running   0          58m
packageserver-7d67fd99cb-j8mw4     1/1     Running   0          58m

4, OLM pods works well, install some new operators. Looks good.

[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-operators
NAME                              DISPLAY                            VERSION              REPLACES                          PHASE
argocd-operator.v0.0.15           Argo CD                            0.0.15               argocd-operator.v0.0.14           Succeeded
elasticsearch-operator.5.1.0-96   OpenShift Elasticsearch Operator   5.1.0-96                                               Succeeded
etcdoperator.v0.9.4-clusterwide   etcd                               0.9.4-clusterwide    etcdoperator.v0.9.2-clusterwide   Succeeded
jaeger-operator.v1.24.0           Community Jaeger Operator          1.24.0               jaeger-operator.v1.21.3           Succeeded
nfd.4.7.0-202106090743            Node Feature Discovery             4.7.0-202106090743                                     Installing
servicemeshoperator.v2.0.6.2      Red Hat OpenShift Service Mesh     2.0.6-2              servicemeshoperator.v2.0.6.1      Succeeded

[cloud-user@preserve-olm-env jian]$ oc get csv -A |wc -l
2187

5, Remove all cluster-scoped operators,

[cloud-user@preserve-olm-env jian]$  oc get csv -n openshift-operators
No resources found in openshift-operators namespace.

[cloud-user@preserve-olm-env jian]$ oc get csv -A |wc -l
3

[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-operator-lifecycle-manager
NAME                               READY   STATUS    RESTARTS   AGE
catalog-operator-f966bdcbf-gtjr6   1/1     Running   0          72m
olm-operator-5696d565d7-jvnjx      1/1     Running   0          72m
packageserver-7d67fd99cb-b4ds7     1/1     Running   0          65m
packageserver-7d67fd99cb-j8mw4     1/1     Running   0          65m

No OLM pods crased, looks good, verify it.

Comment 6 errata-xmlrpc 2021-08-03 17:56:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.22 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2903


Note You need to log in before you can comment on or make changes to this bug.