Bug 1891993 - 4.5 to 4.6 upgrade doesn't remove deployments created by marketplace
Summary: 4.5 to 4.6 upgrade doesn't remove deployments created by marketplace
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.6
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.7.0
Assignee: Ankita Thomas
QA Contact: Bruno Andrade
URL:
Whiteboard: non-multi-arch
Depends On:
Blocks: 1895392
TreeView+ depends on / blocked
 
Reported: 2020-10-27 20:10 UTC by Kevin Rizza
Modified: 2021-02-24 15:29 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:28:35 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-marketplace pull 358 0 None closed Bug 1891993: clean up child resources for old opsrcs 2021-02-16 03:19:57 UTC
Red Hat Knowledge Base (Solution) 5532201 0 None None None 2020-11-04 04:09:11 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:29:09 UTC

Description Kevin Rizza 2020-10-27 20:10:52 UTC
Description of problem:

In 4.6 the marketplace no longer creates operator sources, instead it creates image type catalog sources directly. Those 4.5 operator sources generated deployments that were connected to catalog sources as grpc type catalogs while the 4.6 catalogs are image type. At the moment, the upgrade from 4.5 to 4.6 does not remove the old deployments that were used for the grpc type catalogs. Though they are not wired to anything and don't affect content on the cluster, they are still sitting around wasting resources.


Version-Release number of selected component (if applicable):
4.6

How reproducible:
Always

Steps to Reproduce:
1. Create a 4.5 cluster.
2. Upgrade the cluster to 4.6

Actual results:

`oc get deployments -n openshift-marketplace`

That query returns deployments `community-operators` `certified-operators` `redhat-operators` and `redhat-marketplace` that were created by the marketplace-operator.


Expected results:

`oc get deployments -n openshift-marketplace` should not return the deployments created by the operator described above.

Additional info:

Comment 1 Heiko Braun 2020-10-30 07:13:05 UTC
Is there a workaround for the OCP 4.6 operators to show up?

Comment 4 Masaki Furuta ( RH ) 2020-11-02 09:25:34 UTC
Hello,

I have received following report from our partner NEC which states similar issue, would you please find it below ?

Feedback from NEC:
~~~
We have found the cause of the problem. If RH has already created Bugzilla, Would you please share the following with engineers ?

After the OCP4.6 upgrade, we still have old pods on openshift-marketplace.

  $ oc get pods -n openshift-marketplace
  NAME                                    READY   STATUS    RESTARTS   AGE
  certified-operators-69c7584d8c-z9vjr    1/1     Running   0          3d22h <-- For OCP4.5
  certified-operators-jdbt2               1/1     Running   0          2d7h
  community-operators-54577646b6-qn9l5    1/1     Running   0          3d22h <-- For OCP4.5
  community-operators-bttkx               1/1     Running   0          14h
  marketplace-operator-67d858bf7d-24r29   1/1     Running   0          3d22h
  redhat-marketplace-4wsts                1/1     Running   0          2d7h <-- For OCP4.5
  redhat-marketplace-55c6c798f9-2twk6     1/1     Running   0          3d22h
  redhat-operators-6d98f9c45c-xqht7       1/1     Running   0          3d22h <-- For OCP4.5
  redhat-operators-8nscg                  1/1     Running   0          3d22h

Apparently, since OCP4.6, these pods are controlled directly by the Operator, but the The old deployment information used in OCOP4.5 and earlier is still there, and this makes it possible to use the seems to be occurring

  $ oc get deployment -n openshift-marketplace
  NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
  certified-operators    1/1     1            1           117d
  community-operators    1/1     1            1           117d
  marketplace-operator   1/1     1            1           490d <== OCP4.6 fresh env has only this
  redhat-marketplace     1/1     1            1           117d
  redhat-operators       1/1     1            1           117d

After the following actions, the 4.6 operators are now available: 1.

  1. remove all openshift-marketplace deployments

    $ oc delete deployment -n openshift-marketplace certified-operators community-operators redhat-marketplace redhat-operators marketplace-operator

  2. wait until the marketplace-operator is created

  3. disable all CatalogSource and then re-enable them.

    $ oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
    operatorhub.config.openshift.io/cluster patched
    $ oc patch OperatorHub cluster --type json  -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": false}]'
    operatorhub.config.openshift.io/cluster patched

~~~

Comment 5 Masaki Hatada 2020-11-02 09:48:39 UTC
Dear Red Hat,

Please keep in mind that it's not only a problem which consumes resources.
User cannot install any Operators of 4.6 version due to this issue, since '4.6' channel is hidden by the old resources.

 $ oc get packagemanifest cluster-logging -n openshift-marketplace -o jsonpath='{.status.channels[*].name}'
 4.2 4.2-s390x 4.3 4.4 4.5 preview

We don't know why Red Hat couldn't find this issue before releasing 4.6 as GA, since it would be able to find easily with a basic upgrade test.
Anyway, we hope that Red Hat fix this issue early.

Best Regards,
Masaki Hatada

Comment 11 errata-xmlrpc 2021-02-24 15:28:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.