Bug 1982252 - olm Operator is in CrashLoopBackOff state with error "couldn't cleanup cross-namespace ownerreferences"
Summary: olm Operator is in CrashLoopBackOff state with error "couldn't cleanup cross-...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.7
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: 4.8.0
Assignee: Kevin Rizza
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On: 1979544
Blocks: 1982693
TreeView+ depends on / blocked
 
Reported: 2021-07-14 14:26 UTC by Kevin Rizza
Modified: 2021-07-27 23:14 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1979544
: 1982693 (view as bug list)
Environment:
Last Closed: 2021-07-27 23:14:04 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift operator-framework-olm pull 117 0 None open [release-4.8] Bug 1982252: remove cleanup from startup 2021-07-14 15:18:39 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:14:08 UTC

Comment 2 Jian Zhang 2021-07-16 03:41:45 UTC
[cloud-user@preserve-olm-env jian]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-07-15-221145   True        False         72m     Cluster version is 4.8.0-0.nightly-2021-07-15-221145

[cloud-user@preserve-olm-env jian]$ oc adm release info registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-07-15-221145 --commits|grep lifecycle
  operator-lifecycle-manager                     https://github.com/openshift/operator-framework-olm                         505fe20e27838e8f9e33c2eeb30a56fbadfd6500

This payload contains the fixed PR.

1, Install some cluster-scoped operators.

[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-operators
NAME                                DISPLAY                            VERSION             REPLACES                            PHASE
elasticsearch-operator.5.1.0-96     OpenShift Elasticsearch Operator   5.1.0-96                                                Succeeded
etcdoperator.v0.9.4-clusterwide     etcd                               0.9.4-clusterwide   etcdoperator.v0.9.2-clusterwide     Succeeded
jaeger-operator.v1.20.4             Red Hat OpenShift Jaeger           1.20.4                                                  Installing
openshift-gitops-operator.v1.1.2    Red Hat OpenShift GitOps           1.1.2               openshift-gitops-operator.v1.1.1    Succeeded
redhat-openshift-pipelines.v1.4.1   Red Hat OpenShift Pipelines        1.4.1               redhat-openshift-pipelines.v1.4.0   Succeeded
servicemeshoperator.v2.0.6.2        Red Hat OpenShift Service Mesh     2.0.6-2             servicemeshoperator.v2.0.6.1        Succeeded

2, Create 400 namespaces, but 331 in fact, 

[cloud-user@preserve-olm-env jian]$ for l in {1..399}; do oc adm new-project "test$l";sleep 1; done;
Created project test200
...
Created project test331
^C

3, Check the CSV numbers and the OLM pods.
[cloud-user@preserve-olm-env jian]$ oc get csv -A|wc -l
2392

[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-676568bc96-h6kzz   1/1     Running   0          143m
olm-operator-b85fd4689-hccf8        1/1     Running   0          144m
packageserver-84b68df595-4vkmp      1/1     Running   0          138m
packageserver-84b68df595-n9b4m      1/1     Running   0          138m

4, OLM pods works well, install some new operators. Looks good.

[cloud-user@preserve-olm-env jian]$ oc get sub -A
NAMESPACE                    NAME                              PACKAGE                           SOURCE                CHANNEL
default                      ptp-operator                      ptp-operator                      qe-app-registry       4.8
openshift-logging            cluster-logging                   cluster-logging                   qe-app-registry       stable-5.1
openshift-operators-redhat   elasticsearch-operator            elasticsearch-operator            qe-app-registry       stable-5.1
openshift-operators          etcd                              etcd                              community-operators   clusterwide-alpha
openshift-operators          jaeger-product                    jaeger-product                    redhat-operators      stable
openshift-operators          nfd                               nfd                               qe-app-registry       4.9
openshift-operators          openshift-gitops-operator         openshift-gitops-operator         redhat-operators      stable
openshift-operators          openshift-pipelines-operator-rh   openshift-pipelines-operator-rh   redhat-operators      stable
openshift-operators          servicemeshoperator               servicemeshoperator               redhat-operators      stable
[cloud-user@preserve-olm-env jian]$ oc get csv -n openshift-operators 
NAME                                     DISPLAY                            VERSION              REPLACES                            PHASE
elasticsearch-operator.5.1.0-96          OpenShift Elasticsearch Operator   5.1.0-96                                                 Succeeded
etcdoperator.v0.9.4-clusterwide          etcd                               0.9.4-clusterwide    etcdoperator.v0.9.2-clusterwide     Succeeded
jaeger-operator.v1.20.4                  Red Hat OpenShift Jaeger           1.20.4                                                   Succeeded
node-feature-discovery-operator.v4.9.0   Node Feature Discovery             4.9.0-202107090514                                       Installing
openshift-gitops-operator.v1.1.2         Red Hat OpenShift GitOps           1.1.2                openshift-gitops-operator.v1.1.1    Succeeded
redhat-openshift-pipelines.v1.4.1        Red Hat OpenShift Pipelines        1.4.1                redhat-openshift-pipelines.v1.4.0   Succeeded
servicemeshoperator.v2.0.6.2             Red Hat OpenShift Service Mesh     2.0.6-2              servicemeshoperator.v2.0.6.1        Succeeded

5, Remove all cluster-scoped operators,
[cloud-user@preserve-olm-env jian]$  oc get csv -n openshift-operators
No resources found in openshift-operators namespace.
[cloud-user@preserve-olm-env jian]$ oc get csv |wc -l
8
[cloud-user@preserve-olm-env jian]$ oc get pods -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-676568bc96-h6kzz   1/1     Running   2          161m
olm-operator-b85fd4689-hccf8        1/1     Running   1          161m
packageserver-84b68df595-4vkmp      1/1     Running   2          155m
packageserver-84b68df595-n9b4m      1/1     Running   2          155m

No OLM pods crased, looks good, verify it.

Comment 5 errata-xmlrpc 2021-07-27 23:14:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.