Bug 1831818 - Switching an OpearatorSource to a CatalogSource as default for marketplace does not clean old OperatorSource
Summary: Switching an OpearatorSource to a CatalogSource as default for marketplace do...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.0
Assignee: Anik
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-05 17:12 UTC by Anik
Modified: 2020-07-13 17:35 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-13 17:35:12 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-marketplace pull 308 0 None closed Bug 1831818: Remove Obsolete default OperatorSource 2020-11-23 15:44:55 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:35:32 UTC

Description Anik 2020-05-05 17:12:05 UTC
Description of problem:

In OCP 4.5, the OperatorHub API has been enhanced to accept both OperatorSource and CatalogSource as default operator sources for marketplace. 

However, when an OperatorSource is switched to a CatalogSource, the old OperatorSource is not cleared.  

Version-Release number of selected component (if applicable):

4.5 

How reproducible:

Always 

Steps to Reproduce:
1. Launch OCP 4.5 cluster
2. Patch and delete marketplace operator 

$ oc patch clusterversion version --type=merge -p '{"spec": {"overrides":[{"kind": "Deployment", "name": "marketplace-operator", "namespace": "openshift-marketplace", "unmanaged": true, "group": "apps"}]}}'

$ oc delete deployment marketplace-operator -n openshift-marketplace 

3. Switch https://github.com/operator-framework/operator-marketplace/blob/master/defaults/03_community_operators.yaml to the following Catsrc: 

```catsrc.yaml
apiVersion: "operators.coreos.com/v1"
kind: "CatalogSource"
metadata:
  name: "community-operators"
  namespace: "openshift-marketplace"
spec:
  sourceType: grpc
  image: quay.io/operator-framework/upstream-community-operators:latest
  displayName: "Community Operators"
  publisher: "Red Hat"
  updateStrategy:
    registryPoll:
      interval: 45m

4. Build and push an image to a registry with the above changes 

5. Point to the image built and pushed above here https://github.com/operator-framework/operator-marketplace/blob/master/manifests/09_operator.yaml#L36 

6. oc apply -f manifests/ 

Actual results:

There's 6 registry pods: 

$ oc get pods 
NAME                                    READY   STATUS    RESTARTS   AGE
certified-operators-674686b9cd-ll7h9    1/1     Running   0          42m
community-operators-66469948f8-pv6h7    1/1     Running   0          41m
marketplace-operator-57fd59469f-tb85r   1/1     Running   0          42m
redhat-marketplace-565549d5cc-kxddh     1/1     Running   0          42m
redhat-operators-6f6bf8bcf7-5tnvw       1/1     Running   0          42m
 

Expected results:

There's 3 opsrc, 1 catalogSource, and 4 registry pods, one for each source.

Additional info:

https://github.com/operator-framework/operator-marketplace/pull/300

Comment 4 Jian Zhang 2020-05-11 09:48:55 UTC
Hi Anik,

I'm sorry, but I'm confused. I have some questions below:
> 4. Build and push an image to a registry with the above changes 

Do you mean this CatalogSource image: quay.io/operator-framework/upstream-community-operators:latest?


> 5. Point to the image built and pushed above here https://github.com/operator-framework/operator-marketplace/blob/master/manifests/09_operator.yaml#L36 

> 6. oc apply -f manifests/ 

Seems like we're reinstalling the marketplace-operator itself, why not recover it by using the CVO?

$ oc patch clusterversion version --type=merge -p '{"spec": {"overrides":[{"kind": "Deployment", "name": "marketplace-operator", "namespace": "openshift-marketplace", "managed": true, "group": "apps"}]}}'

One more thing, even if we didn't do any actions for the marketplace-operator itself, the old operator-source still active, as I described in https://issues.redhat.com/browse/OLM-1615. Is it the same issue? Thanks!

Comment 5 Anik 2020-05-13 15:46:06 UTC
Hi Jian,

Yes it's the same issue you mentioned in JIRA. This bz is to fix that issue. 

As for the patching and restarting of Marketplace, apologizes if the way I framed it is confusing. 

Basically, you just have to switch one of the operatorsource to a catsrc, and see that the old operatorsource was not being removed. 

With this PR in, the old operatorsource will be removed. 

Hope that clarifies things.

Comment 6 Jian Zhang 2020-05-15 10:23:19 UTC
Hi Anik,

Thanks for your clarification! Test steps as follows:

Cluster version is 4.5.0-0.nightly-2020-05-13-221558
Marketplace version:
mac:~ jianzhang$ oc exec marketplace-operator-85fbdccd7-llcnx  -- marketplace-operator --version
time="2020-05-15T10:00:25Z" level=info msg="Go Version: go1.13.4"
time="2020-05-15T10:00:25Z" level=info msg="Go OS/Arch: linux/amd64"
time="2020-05-15T10:00:25Z" level=info msg="operator-sdk Version: v0.8.0"
Marketplace source git commit: b1ba97ff4cf3999fd8fcdc2c97700d5291dca1f0

1, Check the default OperatorSource, they worked well.
mac:~ jianzhang$ oc get operatorsource
NAME                  TYPE          ENDPOINT              REGISTRY                          DISPLAYNAME           PUBLISHER   STATUS      MESSAGE                                       AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators               Certified Operators   Red Hat     Succeeded   The object has been successfully reconciled   28h
community-operators   appregistry   https://quay.io/cnr   community-operators               Community Operators   Red Hat     Succeeded   The object has been successfully reconciled   28h
redhat-marketplace    appregistry   https://quay.io/cnr   redhat-marketplace                Red Hat Marketplace   Red Hat     Succeeded   The object has been successfully reconciled   28h
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators                  Red Hat Operators     Red Hat     Succeeded   The object has been successfully reconciled   28h

mac:~ jianzhang$ oc get catalogsource
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
certified-operators   Certified Operators   grpc   Red Hat     28h
community-operators   Community Operators   grpc   Red Hat     28h
redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     28h
redhat-operators      Red Hat Operators     grpc   Red Hat     28h

mac:~ jianzhang$ oc get pods
NAME                                   READY   STATUS    RESTARTS   AGE
certified-operators-68f8c69857-hmrc9   1/1     Running   0          23h
community-operators-5f77c8cf4d-t5rpv   1/1     Running   0          23h
marketplace-operator-85fbdccd7-llcnx   1/1     Running   0          28h
redhat-marketplace-cc46cd9bb-cq4d6     1/1     Running   0          23h
redhat-operators-57dbd6fcb-b58xw       1/1     Running   0          28h

2, Convert the one of OperatorSource (community-operator) to CatalogSource.

mac:~ jianzhang$ cat cs-community-45.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: operatorhubio-catalog
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/operator-framework/upstream-community-operators:latest
  displayName: Community Test Operators
  publisher: OperatorHub.io

mac:~ jianzhang$ oc create -f cs-community-45.yaml 
catalogsource.operators.coreos.com/operatorhubio-catalog created

3, Check the old community-operator OperatorSource status. But, it still exists.
[root@preserve-olm-env ~]# oc get catalogsource
NAME                    DISPLAY                    TYPE   PUBLISHER        AGE
certified-operators     Certified Operators        grpc   Red Hat          29h
community-operators     Community Operators        grpc   Red Hat          29h
operatorhubio-catalog   Community Test Operators   grpc   OperatorHub.io   6m47s
redhat-marketplace      Red Hat Marketplace        grpc   Red Hat          29h
redhat-operators        Red Hat Operators          grpc   Red Hat          29h

[root@preserve-olm-env ~]# oc get pods
NAME                                   READY   STATUS    RESTARTS   AGE
certified-operators-68f8c69857-hmrc9   1/1     Running   0          24h
community-operators-5f77c8cf4d-t5rpv   1/1     Running   0          24h
marketplace-operator-85fbdccd7-llcnx   1/1     Running   0          29h
operatorhubio-catalog-hd7zn            1/1     Running   0          6m53s
redhat-marketplace-cc46cd9bb-cq4d6     1/1     Running   0          24h
redhat-operators-57dbd6fcb-b58xw       1/1     Running   0          29h


[root@preserve-olm-env ~]# oc get operatorsource
NAME                  TYPE          ENDPOINT              REGISTRY                          DISPLAYNAME           PUBLISHER   STATUS      MESSAGE                                       AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators               Certified Operators   Red Hat     Succeeded   The object has been successfully reconciled   29h
community-operators   appregistry   https://quay.io/cnr   community-operators               Community Operators   Red Hat     Succeeded   The object has been successfully reconciled   29h
redhat-marketplace    appregistry   https://quay.io/cnr   redhat-marketplace                Red Hat Marketplace   Red Hat     Succeeded   The object has been successfully reconciled   29h
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators                  Red Hat Operators     Red Hat     Succeeded   The object has been successfully reconciled   29h

Comment 7 Anik 2020-05-18 14:01:13 UTC
The test case listed above is incorrect. Please see the description of the problem above for the correct steps to test this. 

Please use this doc as reference: https://github.com/operator-framework/operator-marketplace/blob/master/CONTRIBUTING.md#testing-changes-to-rbac-policy 

NOTE: You need to rebuild and redeploy marketplace operator. The defaults are bootstrapped by marketplace operator from here: https://github.com/operator-framework/operator-marketplace/blob/master/defaults/

Simply changing an operatorsource to a catsrc in cluster will not work since marketplace will maintain the state of the cluster it received during bootstrap.

Comment 8 Jian Zhang 2020-05-19 09:38:12 UTC
Hi Anik,

Thanks! Sorry, I misunderstood it. For this bug, the test steps as follows:
1, Remove the default community-operator OperatorSource YAML file and add a CatalogSource YAML file, as follows:
https://github.com/jianzhangbjz/operator-marketplace/blob/cs/defaults/03_community_operators.yaml

2, Build the marketplace-operator image.
[root@preserve-olm-env operator-marketplace]# docker build . --tag=quay.io/olmqe/marketplace-operator
Sending build context to Docker daemon  261.4MB
Step 1/12 : FROM registry.svc.ci.openshift.org/openshift/release:golang-1.10 AS builder
golang-1.10: Pulling from openshift/release
524b0c1e57f8: Already exists 
7201192efb82: Pull complete 
Digest: sha256:0378b72464970785d0b83b400d5a83b7c4cb2d0a997fe35d087f77ce3997ad0f
Status: Downloaded newer image for registry.svc.ci.openshift.org/openshift/release:golang-1.10
 ---> eae4d37ce64f
Step 2/12 : WORKDIR /go/src/github.com/operator-framework/operator-marketplace
 ---> Running in 1f65310a0ce9
Removing intermediate container 1f65310a0ce9
 ---> 5a1a88298e0f
Step 3/12 : COPY . .
 ---> 0cb0bd22a678
Step 4/12 : RUN make osbs-build
 ---> Running in af4f869c29a1
# hack/build.sh
./build/build.sh
building marketplace-operator...
Removing intermediate container af4f869c29a1
 ---> 09cfa76fe23a
Step 5/12 : FROM registry.svc.ci.openshift.org/openshift/origin-v4.0:base
base: Pulling from openshift/origin-v4.0
c2340472a0fa: Pull complete 
6e55351c18ff: Pull complete 
b2d2704dda6c: Pull complete 
a2c10be042b9: Pull complete 
Digest: sha256:cbd0e2931e6ae8cbd1522cd7226cff89629c21a3652f6fd57d84f405d432c07a
Status: Downloaded newer image for registry.svc.ci.openshift.org/openshift/origin-v4.0:base
 ---> f779ab8af41a
Step 6/12 : RUN useradd marketplace-operator
 ---> Running in 7396c74faf02
Removing intermediate container 7396c74faf02
 ---> 9e808118e7b3
Step 7/12 : USER marketplace-operator
 ---> Running in 7a9fe184e9f6
Removing intermediate container 7a9fe184e9f6
 ---> a1297ffa8ddd
Step 8/12 : COPY --from=builder /go/src/github.com/operator-framework/operator-marketplace/build/_output/bin/marketplace-operator /usr/bin
 ---> 82a3cd539038
Step 9/12 : ADD manifests /manifests
 ---> 70e616c6a11b
Step 10/12 : ADD defaults /defaults
 ---> 98f2702dc2b6
Step 11/12 : LABEL io.k8s.display-name="OpenShift Marketplace Operator"       io.k8s.description="This is a component of OpenShift Container Platform and manages the OpenShift Marketplace."       io.openshift.tags="openshift,marketplace"       io.openshift.release.operator=true       maintainer="AOS Marketplace <aos-marketplace@redhat.com>"
 ---> Running in 52904f1fd7c9
Removing intermediate container 52904f1fd7c9
 ---> fc5c91dd2785
Step 12/12 : CMD ["/usr/bin/marketplace-operator"]
 ---> Running in 6d7514d7c51c
Removing intermediate container 6d7514d7c51c
 ---> 25cdb6fe6af2
Successfully built 25cdb6fe6af2
Successfully tagged quay.io/olmqe/marketplace-operator:latest

And, retag and push it to the Quay.
[root@preserve-olm-env operator-marketplace]# docker tag quay.io/olmqe/marketplace-operator quay.io/olmqe/marketplace-operator:cs
[root@preserve-olm-env operator-marketplace]# docker push quay.io/olmqe/marketplace-operator:cs
The push refers to repository [quay.io/olmqe/marketplace-operator]
8a4d09641909: Pushed 
cc5492997673: Pushed 
302efb89260f: Pushed 
f65c5c3b30fc: Pushed 
4ef6a7eaf5a6: Pushed 
762eded89aa4: Pushed 
e18335b29b2e: Pushed 
4f25b188f48c: Pushed 
cs: digest: sha256:d9fad855cebb41038567fa1b80e5cf9a8abe83997bc6ac3e4c2cc3c1d1ef1bd1 size: 1990

3, Set this image in the marketplace-operator manifest, as follows: 
https://github.com/jianzhangbjz/operator-marketplace/blob/cs/manifests/09_operator.yaml

4, Set the marketplace-operator to `Unmanaged` for the CVO. And, delete the marketplace-operator.

mac:operator-marketplace jianzhang$ oc patch clusterversion version --type=merge -p '{"spec": {"overrides":[{"kind": "Deployment", "name": "marketplace-operator", "namespace": "openshift-marketplace", "unmanaged": true, "group": "apps"}]}}'

mac:operator-marketplace jianzhang$ oc delete deployment marketplace-operator -n openshift-marketplace
deployment.apps "marketplace-operator" deleted

5, Redeploy the marketplace-operator.
mac:operator-marketplace jianzhang$ oc create -f manifests/09_operator.yaml 
deployment.apps/marketplace-operator created

mac:operator-marketplace jianzhang$ oc get deployment
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
certified-operators    1/1     1            1           4h18m
marketplace-operator   1/1     1            1           26s
qe-app-registry        1/1     1            1           4h4m
redhat-marketplace     1/1     1            1           4h18m
redhat-operators       1/1     1            1           4h18m

mac:operator-marketplace jianzhang$ oc get pods
NAME                                   READY   STATUS    RESTARTS   AGE
certified-operators-5c65f7c4c5-lmkdc   1/1     Running   0          2m15s
community-operators-699ps              1/1     Running   0          2m20s
marketplace-operator-5cc9b978d-qkl5m   1/1     Running   0          2m39s
qe-app-registry-589b449c66-b2z6d       1/1     Running   0          2m13s
redhat-marketplace-bb99b89f7-g9h99     1/1     Running   0          2m18s
redhat-operators-5f865d76f6-5cbvb      1/1     Running   0          2m17s

The old community operator resources had been removed. Looks good to me, verify it.
mac:operator-marketplace jianzhang$ oc get operatorsource
NAME                  TYPE          ENDPOINT              REGISTRY                          DISPLAYNAME           PUBLISHER   STATUS      MESSAGE                                       AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators               Certified Operators   Red Hat     Succeeded   The object has been successfully reconciled   4h20m
qe-app-registry       appregistry   https://quay.io/cnr   openshift-qe-optional-operators                                     Succeeded   The object has been successfully reconciled   4h7m
redhat-marketplace    appregistry   https://quay.io/cnr   redhat-marketplace                Red Hat Marketplace   Red Hat     Succeeded   The object has been successfully reconciled   4h20m
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators                  Red Hat Operators     Red Hat     Succeeded   The object has been successfully reconciled   4h20m
mac:operator-marketplace jianzhang$ oc get catalogsource
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
certified-operators   Certified Operators   grpc   Red Hat     4h21m
community-operators   Community Operators   grpc   Red Hat     2m52s
qe-app-registry                             grpc               4h7m
redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     4h20m
redhat-operators      Red Hat Operators     grpc   Red Hat     4h20m

mac:operator-marketplace jianzhang$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-05-18-225907   True        False         6h17m   Cluster version is 4.5.0-0.nightly-2020-05-18-225907
mac:operator-marketplace jianzhang$ oc exec marketplace-operator-5cc9b978d-qkl5m -- marketplace-operator --version
Marketplace source git commit: 
time="2020-05-19T09:37:28Z" level=info msg="Go Version: go1.10.8"
time="2020-05-19T09:37:28Z" level=info msg="Go OS/Arch: linux/amd64"
time="2020-05-19T09:37:28Z" level=info msg="operator-sdk Version: v0.8.0"

Comment 9 errata-xmlrpc 2020-07-13 17:35:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.